I ran git status on a newly created, very small repository, but the command took more than 10 seconds to finish. This was highly unusual, as git status is mostly instantaneous for me. Indeed it was instantaneous the next time I ran it on the same repository. Puzzled, I could only think of one thing out of the ordinary: I had just rebooted my Mac. So I tried rebooting again, and then the issue occurred again!
For the impatient who don't care to hear the story of how I discovered the cause, I'll cut to the chase. The
/usr/bin/git executable on macOS is just a stub that calls the
_xcselect_invoke_xcrun function in the
/usr/lib/libxcselect.dylib library, which you can see by using the
/usr/bin/otool command-line tool — which coincidentally is also just a stub that calls the
_xcselect_invoke_xcrun function. By the way, if you're looking for
/usr/lib/libxcselect.dylib on disk you may not find it, because Big Sur. Is your head spinning yet? The actual tools are inside the Xcode app bundle:
$ xcrun --find git /Applications/Xcode.app/Contents/Developer/usr/bin/git
When you attempt to run one of the developer tools, the
_xcselect_invoke_xcrun function must look up the actual path of the tool. The paths of Xcode and the developer tools are cached on disk in a database file named
xcrun_db located in your
$TMPDIR. (In Terminal, call
echo $TMPDIR to see the temporary directory path.)
Perhaps you already see the problem here: the contents of
$TMPDIR are emptied on every reboot! Thus, the first time you run a developer tool after reboot, the
xcrun_db cache needs to be regenerated. Sigh.
As an experiment, I saved a duplicate of my
xcrun_db file, rebooted, and then put the duplicate in
$TMPDIR. With the cache restored, there was no longer any slowness on the first run of developer tools after reboot. Q.E.D.
Why does it take so long to regenerate the cache? While I was reproducing and diagnosing the issue, I noticed that when I ran a developer tool after reboot, the process
syspolicyd went crazy and used almost 100% CPU until the command finished. I took samples of
syspolicyd when this happened, and the process seemed to be spending a lot of time in the security framework checking code signing. Running
fs_usage showed a lot of calls to
getattrlist on files inside the Xcode bundle. (Not all the files, though; verifying the code signature of the entire gigantic Xcode bundle typically takes much longer, more like 10 minutes than 10 seconds.) So it appears that some limited kind of code signature verification is done before the
xcrun_db file is written. I should mention that neither Xcode nor the files inside its bundle have any
com.apple.quarantine extended attributes. The only xattr are a few
com.apple.lastuseddate#PS on files that I've opened in a text editor. (Again, sigh.)
You may remember
syspolicyd as the culprit in checking notarization of unsigned executables, but this is different. I can reproduce the developer tools issue with my wifi turned off, and I've verified with Little Snitch that
syspolicyd does not even attempt any network connections in this case. Moreover, the tools in
/usr/bin/ and Xcode itself are not unsigned, they're validly signed by Apple. Clearly, the slowness here is caused by checking local files and not by waiting on a network connection.
Why haven't I noticed the issue until now? I'm not sure, but I did recently update from Mojave to Big Sur and from Xcode 11 to Xcode 12. I can reproduce the slowness on Catalina, Big Sur, and Monterey, with Xcode 12 and 13. I haven't tested with Xcode 11, because it takes practically forever to download and install Xcode. It's not just an app anymore, it's an operating system! Actually four operating systems, since Xcode includes simulators for iOS, iPadOS, tvOS, and watchOS.
Do I have a workaround for the slowness? Yes and no. I don't know how to stop
xcrun_db from getting deleted on reboot. However, I discovered a way to accelerate the cache regeneration to around 3 seconds, down from over 10 seconds: disable System Integrity Protection. Seriously, that works.