Patching the .NET 8 runtime to fix debugging performance on macOS and Linux

The .NET runtime has a bug which makes debugging extremely slow on macOS and linux when working with code that does a lot of async/await. 

One example: Running one of the Integration Tests for the Octopus Deploy server took roughly 30 seconds when run on Windows with a Debugger attached. When running on either Windows or Mac without a debugger it took roughly 25 seconds, but on Mac with a Debugger it took over 3 minutes.
With a patched version of the .NET runtime, the MacOS debugger time drops down to match the non-debugger time.

The issue was tracked here in Debugger.NotifyOfCrossThreadDependency can be slow when a debugger is attached and was fixed in .NET 9. However .NET 9 is a short-term-support release and a lot of codebases can't upgrade to it. 

To fix the issue on .NET 8 or lower, you need to build a patched version of System.Private.CoreLib.dll and put it into your local .NET runtime folder. This needs to be done for every new runtime version (e.g. 8.0.16 and 8.0.17)

At Octopus we were hit by the issue in the days of .NET 6, so we needed to patch the runtime, or put up with debugging performance so bad that it nearly took macOS off the table as a developer option. To solve it at scale, we wrote a GitHub actions script to build the patched version of the runtime, so developers could simply download a patched dll and install it on their macs. Here's how it works.

Note: This is not novel or OctopusDeploy-specific information. We got this information from various comments and discussions on GitHub issues where it is public. I would link to them except it was a few years ago and I can't recall or easily find the exact ones.

Anyway, to resolve the issue you need to turn the NotifyOfCrossThreadDependencySlow function in src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Debugger.cs into a no-op. Our GitHub action does it by applying this patch file:

diff --git a/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Debugger.cs b/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Debugger.cs
index 9a51c47cb02..2bbd33efb74 100644
--- a/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Debugger.cs
+++ b/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Debugger.cs
@@ -32,8 +32,10 @@ private sealed class CrossThreadDependencyNotification : ICustomDebuggerNotifica
 
         // Do not inline the slow path
         [MethodImpl(MethodImplOptions.NoInlining)]
-        private static void NotifyOfCrossThreadDependencySlow() =>
-            CustomNotification(new CrossThreadDependencyNotification());
+        private static void NotifyOfCrossThreadDependencySlow()
+        {
+            // Removed as per the .NET debugger on macOS hack
+        }
 
         // Sends a notification to the debugger to indicate that execution is about to enter a path
         // involving a cross thread dependency. A debugger that has opted into this type of notification

Aside: You might wonder if there are any harmful side-effects of no-opping this function. Presumably it does something useful? At Octopus Deploy, we've had roughly 50 macOS-using developers doing full time .NET development with this patch since 2023 across many versions of .NET 6 and .NET 8, and there was not one single report of it causing any problems. So from that perspective, it seems 100% safe.

You need to compile the .NET Runtime codebase in release mode with this patch applied, which goes like this:

1. You need to do this on an Apple Silicon mac, and you need the Xcode command line developer tools installed if you don't already have them. You can do this by running xcode-select --install in the terminal. MacOS will prompt and ask if you want to install the command line tools.

2. Once you have the command line tools you also need cmake, icu4c and pkg-config. You can get these via homebrew with brew install cmake icu4c pkg-config. Note: If you're allergic to homebrew like I am, there's a chance these may not be needed on later versions of macOS. You could try skipping this step and proceeding with building the patch without them and see if it works for you, then loop back and add them if not.

3. Find the runtime version you want to patch. For example 8.0.17. You can see which runtimes you have by running dotnet --list-runtimes. If you have multiple installed, you probably just need to patch the latest 8.0.x version.

4. Second, check out the dotnet/runtime Git repository and checkout the tag matching that version. Microsoft put 'v' in front of their tags, so you'd check out refs/tags/v8.0.17

5. Third, apply the patch. If you're just doing it as a one-off the easiest way is to open the aforemorentioned Debugger.cs in a text editor and simply edit it to no-op the NotifyOfCrossThreadDependencySlow function body. 

6. Then you can build the runtime by running build.sh clr -arch arm64 -c release inside the checked-out dotnet runtime repository.

7. The build is going to drop a file into runtime/artifacts/bin/coreclr/OSX.arm64.Release/System.Private.CoreLib.dll. You want to copy this into your 'real' dotnet runtime installation by doing something like sudo cp ./runtime/artifacts/bin/coreclr/OSX.arm64.Release/System.Private.CoreLib.dll /usr/local/share/dotnet/shared/Microsoft.NETCore.App/8.0.17/

That's it! Restart your instances of JetBrains Rider / VSCode or whatever other tools you're using to work with .NET applications and you should observe vastly increased Debugging performance.

For completeness, here's our complete GitHub actions yaml workflow file which automates this, if you'd like to do that too. 

name: CI

on:
  workflow_dispatch:
    inputs:
      version:
        description: 'Git tag in dotnet/runtime repository to check out (e.g. v6.0.22)'
        required: true
        type: string

jobs:
  build:
    # Run specifically on Apple Silicon runner: https://github.com/actions/runner-images/issues/8439
    runs-on: macos-latest-xlarge

    steps:
      - name: Install dependencies
        run: brew install cmake icu4c pkg-config
        
      - name: Checkout patch
        uses: actions/checkout@v4
        with:
          path: patch
        
      - name: Checkout .NET runtime
        uses: actions/checkout@v4
        with:
          repository: 'dotnet/runtime'
          ref: refs/tags/${{ inputs.version }}
          fetch-depth: 1
          path: runtime

      - name: Apply patch
        run: patch -N -i patch/Debugger.cs.patch runtime/src/coreclr/System.Private.CoreLib/src/System/Diagnostics/Debugger.cs

      - name: Build runtime
        run: runtime/build.sh clr -arch arm64 -c release
      
      - name: Compute hash
        run: shasum -a 256 runtime/artifacts/bin/coreclr/OSX.arm64.Release/System.Private.CoreLib.dll > dotnet_runtime_${{ inputs.version }}_SHA256SUM.txt

      # The create-release action created by GitHub is no longer maintained: https://github.com/actions/create-release.
      # At the time of writing, ncipollo/release-action is listed on the repo as one of the recommended alternatives
      - uses: ncipollo/release-action@440c8c1cb0ed28b9f43e4d1d670870f059653174 # v1.16.0
        with:
          tag: ${{ inputs.version }}
          artifacts: "runtime/artifacts/bin/coreclr/OSX.arm64.Release/System.Private.CoreLib.dll,dotnet_runtime_${{ inputs.version }}_SHA256SUM.txt"