Kernel Debugging Setup on MacOS

9 min readJun 7, 2024

--

Introduction to Two-Machine Configuration for Kernel Debugging.

INTRO

A short guide on how to start Kernel Debugging of macOS using two Mac machines (MacBook → Mac Mini) and Kernel Debug Kit.

MacBook — a host that runs the debugger and is connected to the router (it does not matter if it is via ethernet or Wi-Fi).
Mac Mini — a target machine for debugging code. It is connected to the MacBook using an ethernet to USB-C adapter.

Connecting devices using ethernet is necessary because the debugging facility does not work through the Wi-Fi. I tried to achieve it with no luck.

Moreover, the setup I used here using a USB-C to Ethernet adapter (Dell) did not work either. It is not perfect, but I found a workaround to this, explained later.

I used here the ReadMe.md found during the second step of the KDK installation. I also uploaded it to Snake&Apple — X. NU/mac/KDK.pdf.

The article is somehow incomplete as na active debugging on physical devices on Apple Silicon is not supported. I wrote this more for myself so I can return to it in the future and not start from zero again. I decided to publish it because, in my opinion, it is not completely useless, as it shows some problems that can be encountered during the process and some solutions.

REMOTE MANAGEMENT

As we are using the same network, we can use the built-in Screen Sharing for communication between devices without complicated configuration.

However, when we run Mac Mini for the first time, we need a keyboard, mouse, and monitor to set up Remote Management:

Alternatively, we can set Screen Sharing + File Sharing.

After this is set, we can reboot the Mac Mini and then, on our MacBook, open the Screen Sharing. We should see a device under the Network tab:

As we are using Gigabit Ethernet, we can use High-Performance option:

This makes managing the Mac Mini much faster, especially when working with two or more devices that are far away from us. Here is also a tip on how to remotely reboot with the PowerButton — by Ivan Kuleshov in this blog post.

KDK

Check the BuildVersion of macOS by running the sw_vers command:

Download the right version of KDK that corresponds to your MacOS. You must be logged in using your Apple ID to be authorized:

It seems like the KDK is not updated in parallel with the macOS version. I could not find the exact build version, so I downloaded the latest available:

In this case, the best option for the researcher is not to update the device for a certain time and wait for the latest KDK to be compatible (probably ^^).

There is also an alternative way to download KDK from the below repository. Surprisingly, I could find the 23F79 version there:

dortania/KdkSupportPkg

Kernel Debug Kit Mirror for macOS Ventura and newer

Install KDK on both machines:

It is located in /Library/Developer/KDKs/KDK_14.5_23F5074a.kdk/:

Now, we need to configure our Host (MacBook) and target (Mac Mini).

HOST DEVICE

On the MacBook, we need to install Xcode and, as the instruction states, some additional Python packaged for kernel debugging:

Downloads and Resources — Xcode — Apple Developer

xcode-select --install
xcrun python3 -m pip install --user --ignore-installed macholib
xcrun python3 -m pip install --user --ignore-installed future

Additionally, we can configure the kdumpd Core Dump Server with:

mkdir /var/tmp/PanicDumps
chmod 1777 /var/tmp/PanicDumps
sudo launchctl load -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist

Verify if com.apple.kdumpd works:

sudo launchctl list | grep com.apple.kdumpd

The default port defined in plist is 1069. We can also check if it is open:

netstat  -an  | grep 1069

The Core Dump Server is especially needed for the Apple chip. For reference on core dumps check: Technical Note TN2118 — Kernel Core Dumps.

TARGET DEVICE

We need to reboot the Mac Mini in Recovery Mode by pressing and holding the power button. For this task, we also need a monitor and keyboard.

Then, in the Recovery Mode, we need to launch the terminal and disable SIP (we need SIP disabled to set boot arguments in the next steps):

csrutil disable
reboot

Identify which network interface is used to connect to our MacBook:

ifconfig

Here is the tricky part. On Intel, we would normally configure boot arguments as follows (kdp_match_name is our network interface):

# 0x1C44 means - wait for debugger to attach on NMI/Panic using network.
sudo nvram boot-args="debug=0xC44 kdp_match_name=en0 wdt=-1"
sudo reboot

All debug flag values can be seen in debug.h. There is also an explanation of some of the flags in the TN2118: Kernel Core Dumps article:

However, we can read during installation that the Apple chip sadly does not support active kernel debugging, so we can only work on core dumps :/

So, the only thing we can achieve right now using physical devices without some tricky hacks is to get the core dump during panic and send it to host:

sudo nvram boot-args="debug=0xCD44 kdp_match_name=en0 _panicd_ip=HOST_IP wdt=-1"
sudo reboot

0x8000 == DB_NMI_BTN_ENA — shortcut to trigger NMI using PowerButton

With 0x4000 we can trigger NMI by clicking PowerButton on Mac Mini, but default functionalities like putting the system to sleep are overwritten.

0x4000 == DB_REBOOT_POST_CORE — reboot after the core dumping is done
0x400 | 0x800 — trigger core dump on panic|NMI
0x40 == DB_ARP— support debugging across the subnet
0x04 == | DB_NMI —activate kernel debugging + support NMI
_panicd_ip — IP address where to send Core Dump on panic|NMI
On Intel with active debugging, we can also use the 0x1000 == DB_DBG_POST_CORE — wait for a debugger to attach after dumping.

To make this work, we cannot use an ethernet to USB-C adapter (as I did).

We would need a Thunderbolt 3 (USB-C) ethernet adapter as described in this brilliant article by Antonio Zekic:

MacOS Two-machine Kernel Debugging

diverto.github.io

There is also another option that might work for Apple Silicon with two FireWire cables described by Min(Spark) Zheng at DEFCON25:

However, there is another way. We can dump the core locally with the automatical reboot set and, after the reboot, send it manually to MacBook:

# On panic|NMI dump core to /var/tmp/kernel_panics and reboot after
sudo nvram boot-args="debug=0xCC44 wdt=-1"
sudo reboot

The image above shows example core dump file set after panic. The most important for us is the kernel.core.gz which stores the state for lldb.

KERNEL PANIC TRIGGER

To get a core dump or, in Intel’s case, to set the kernel debugging facility to wait for the debugger to attach from the host, we must trigger panic|NMI.

The most straightforward option is to trigger NMI manually. Below, we can see the key sequence for that:

However, in the case of Mac Mini, we do not have a keyboard, so it is more handy to set the 0x4000 debug flag and press the power button.

Another way is to trigger panic programmatically using dtrace:

sudo dtrace -w -n "BEGIN{ panic();}"

There is also a way to trigger kernel panic on Intel using an Instant Panic Kernel Extension:

sudo kextload InstantPanic/build/InstantPanic.kext

We can probably use the third option for Apple Silicon. The InstantPanic.kext is just not precompiled for arm, yet I have not tried it.

DEBUGGING

In the case of Intel (and hopefully for macOS on Apple chip soon), we can now attach the debugger from a host to a target using lldb and start debugging:

# Command issued in lldb:
target create /Library/Developer/KDKs/KDK_14.5_23F5074a.kdk/System/Library/Kernels/kernel.development
kdp-remote IP_ADDRESS_OF_TARGET

It would automatically load symbols from the KDK we installed:

In the case of Apple Silicon, we download and unzip kernel.core.gz:

Then we can load it to lldb:

Still, this is an unsimbolicated core dump because lldb could not find the kernel binary on our system. It searches it using UUID, which must be the same as the kernel binary from KDK, but we do not have the exact BuildVersion.

We can check UUIDs of all kernel binaries from KDK using:

dwarfdump --uuid /Library/Developer/KDKs/KDK_14.5_23F5074a.kdk/System/Library/Kernels/ *

We can also load it manually if we have the correct version:

target symbols add /Library/Developer/KDKs/KDK_14.5_23F5074a.kdk/System/Library/Kernels/kernel.release

In case we have the valid version of KDK (BuildVersion must be the same!) we should see an output similar to the one below:

When loaded with the correct KDK version installed on our host system, we will find the symbolicated output in disassembly:

Anyway, we are in a “halted” state here.

EPILOG

To clean the changes we made on our target device, we should first run:

sudo nvram -d boot-args

After that, we should reboot in recovery mode and enable SIP:

sudo csrutil enable

We should also turn off the kdumpd server on our host device:

sudo launchctl unload -w /System/Library/LaunchDaemons/com.apple.kdumpd.plist

REFERENCES

Final Words

It is strange that active kernel debugging on macOS with physical devices is not supported and is not as straightforward as I thought.

Yet, it is not impossible, and there is some guide about how to do it virtually, but this is for another story:

One day, I will for sure go back to this topic.