Using dtrace on MacOS with SIP enabled
The problem
On all current MacOS versions (Catalina 10.15.x, Big Sur 11.x) System Integrity Protection (SIP) is enabled by default and prevents most uses of dtrace and other tools and scripts based on it (i.e. dtruss).
The usual way to make dtrace work on MacOS is to boot into recovery mode and disable some of the SIP protections:
csrutil enable --without dtraceHowever, this only works if you actually can boot into recovery mode — which isn’t the case if your Mac is in a remote place and you can only access it through some sort of software-based Remote Desktop (i.e. VNC, Apple Remote Desktop, Teamviewer, etc).
It’s also not possible (at least not via self-service to the best of my knowledge) to disable SIP on most Mac Cloud providers, like AWS EC2, Flow Swiss or MacStadium. Same for Mac CI runners on services like GitHub Actions. (For AWS, support has confirmed that it’s not possible at all).
In these cases trying to change the SIP mode in the OS (and not recovery mode) fails:
$ csrutil enable --without dtrace
csrutil: This tool needs to be executed from Recovery OS.and without that trying to use dtrace based tools fails in most cases:
$ sudo dtruss ls -R /Applications
dtrace: system integrity protection is on, some features will not be available
dtrace: failed to execute ls: (os/kern) failureBut there’s hope: SIP doesn’t block tracing entirely
SIP only blocks tracing of system executables that ship with MacOS (i.e. executables that live in paths like /bin , /System, etc). Which means dtrace works for all non-system executables (like your own app).
But what about system executables? Let’s say we wanted to trace all syscalls made by the /bin/ls utility, we could make a copy of it and remove the code signing signature.
$ cd ~
$ cp /bin/ls .
$ sudo codesign --remove-signature ./ls
$ codesign -dv ./ls #verify
./ls: code object is not signed at allNow we can trace the syscalls it makes 🎉:
$ sudo dtruss -t open_nocancel ~/ls -R /Applications/
...
open_nocancel(".\0", 0x1000000, 0x0) = 3 0
open_nocancel("/Applications\0", 0x1100004, 0x0) = 4 0
...Going further: Using chroot
I had the case that I wanted to trace a binary tool with its child processes that makes a lot of calls to standard cmdline tools and the paths to these executables were hardcoded (i.e. /bin/ls ).
My starting point for getting better tracing results was to let the tool run in a copy of the MacOS base root filesystem via chroot. All files in that chroot had codesigning signatures removed.
Here’s a quick summary how I created the chroot. (I’m not going into all details, in case of questions feel free to leave a comment):
$ sw_vers | grep ProductVersion
ProductVersion: 10.15.7# download MacOS image for this version
$ softwareupdate --fetch-full-installer --full-installer-version 10.15.7# we'll use BaseSystem.dmg for our chroot fs
$ cd ~
$ cp "/Applications/Install macOS Catalina.app/Contents/SharedSupport/BaseSystem.dmg" .# mount it writeable, so we can remove cs sigs
$ hdiutil attach -owners on BaseSystem.dmg -shadow
...
/dev/disk2s1 Apple_HFS /Volumes/macOS Base System# remove cs sigs
$ sudo find "/Volumes/macOS Base System" -exec codesign --remove-signature {} \; > /dev/null 2>&1# verify that sigs are really removed
$ codesign -dv "/Volumes/macOS Base System/usr/sbin/chroot"
chroot: code object is not signed at all# make symlink as dtruss doesn't like spaces in path to executable
$ sudo ln -s "/Volumes/macOS Base System" /Volumes/myroot# also copy the executable that we want to trace,
# including all required dependencies, config, data dirs, etc
# into /Volumes/myroot
As a simple example, here’s running a trace on ls inside the chroot:
$ sudo dtruss -t open_nocancel /Volumes/myroot/usr/sbin/chroot /Volumes/myroot ls -R /Applications
...
open_nocancel(".\0", 0x1000000, 0x0) = 3 0
open_nocancel("/Applications\0", 0x1100004, 0x0) = 4 0
...It works. 🎉🎉
Note that we’re using dtruss on our real root fs to launch the chroot executable from our fake root fs as it already has the signature removed. We tell it to root into our fake root and execute the ls command in it.
(This very simple example doesn’t really benefit from the chroot — but imagine to trace a tool that calls several other binaries from the chroot like ls , cp , etc — they’re all traceable).
Update 27-Oct-2021: A better alternative: Tools based on the Apple Endpoint Security Framework
I haven’t been fully aware of this when I wrote this blogpost, but there’s an alternative to dtrace for some types of traces: The Apple Endpoint Security Framework.
If a tool is properly signed and notarized (with an additional entitlement that needs to be manually requested from Apple and that seems hard to get approved), it can run even when SIP is enabled.
A few developers have built tracing tools using this framework and have managed to get that entitlement — so they provide binaries of their tools that run on Mac instances of the cloud providers. (So as long as you don’t want to hack/recompile these tools you’re good).
Here are three tools that I don’t want to miss anymore for troubleshooting Mac instances in the cloud:
One note in advance: All of these tools display information about the same events. These events are not syscalls but framework specific events. The reference docs for these events can be found here. Looking at this list of events also shows the limitations of the framework — it mainly “only” covers events around processes and files.
1.) FileMonitor (from objective-see)
A cmdline tool that displays file events (creation, modifications, deletions). Here’s how opening a file file test.txt with vim looks like on an AWS EC2 Mac instance:
2.) ProcessMonitor (from objective-see)
From the same developers also comes a tool that shows events about process creations and terminations. It feels a lot like the dtrace based newproc.d (part of MacOS). Here’s the first event from starting vim on an AWS EC2 Mac:
3.) Crescendo
Crescendo is a GUI tool with the look and feel Windows users may know from Microsoft Sysinternal’s Procmon. It lets you start and stop recording events of all types, filter them by categories (file, process, network, etc) and save the recorded events as json file. Unfortunately searching and filtering beyond event categories isn’t possible in the GUI yet, so the json export comes in handy.
Here’s how saving a file test.txt with vim looks like (again on a Mac on AWS with SIP enabled). Vim writes to a temp file and renames it to the final name — Crescendo shows that nicely:
Wrapping up.
These are just some workarounds that have helped me in specific situations. They’re not meant as a universal solution to any dtrace problem caused by SIP. Especially the chroot solution is only meant as a starting point and might not work in more complex situations.
If you can use the various new tools based on the Apple Endpoint Security Framework, they’re probably the easiest way to get at least some application tracing on Mac cloud instances (though they’re rather limited). If you have any additional ideas please drop a comment.
