Six months after the Mac OCSP appocalypse, here we go again, and here I go again. Back then I discovered that the
trustd process was having trouble connecting to
ocsp.apple.com, an issue temporarily solved by preventing connection attempts to that domain (for example in Little Snitch). Yesterday I started noticing another issue with
trustd, but this time the issue was a little different: very high CPU usage. I've heard from several other people who started noticing the same issue yesterday too, one of whom helpfully referred me to this reddit thread with even more reports. On investigation, I found that the
nsurlsessiond process was connecting to the server
valid.apple.com, and immediately afterward
trustd CPU jumped from 0% to 100%. It seems that the issue can be temporarily solved by preventing
nsurlsessiond from connecting to
valid.apple.com. You may have to reboot or force quit
trustd to get its CPU usage back to normal. It's important to note that this is only a temporary workaround to the CPU usage problem;
trustd is an important macOS system process that checks certificate validity and revocation status, so you probably don't want to block
I can reproduce the issue easily on my M1 Mac mini running the latest macOS 11.3.1. I can also reproduce it, though not as reliably, on my Intel MacBook Pro running macOS 10.14.6 patched with the latest security updates. (I can hear the fans running on my MacBook Pro, which is how I first noticed the issue. Whereas I almost never hear the Mac mini fans.) I haven't been able to reproduce it booted into my external disk with macOS 11.2.1 or an unpatched 10.15.7, so this issue may be related somehow to the security updates on April 26. I'm going to try later to update those unpatched volumes, but I haven't had time yet, because the updates could take hours (sigh). In any case, I haven't spent as much time testing the external boot volumes, so I don't want to say definitely that the issue can't be reproduced on older macOS versions.
On my MacBook Pro, I set Little Snitch to ask for
nsurlsessiond connection attempts. Here you see it attempting to connect to
valid.apple.com right after I plug my MacBook Pro into the power adapter (more on that later). The CPU usage of
trustd is 0% at that point.
I click the Allow button, and suddenly
trustd CPU jumps way up.
Here's a screenshot from my Mac mini after I force quit
trustd because of high CPU usage.
trustd restarts, CPU usage jumps up again, and every time I see a message about
com.apple.powerlog.plxpclogger.xpc in the Console log.
This happens about every 5 minutes on my Mac mini, which is of course always plugged in to the AC power. My impression is that this is all caused by some kind of macOS diagnostic logging related to power management, which is why it occurred on my MacBook Pro when I plugged in the power cord. (Note that I have Analytics disabled in the Privacy tab of the Security & Privacy pane of System Preferences on both Macs.)
I took a sample of the
trustd process, and it's spending a lot of time in sqlite functions. Which makes sense, because
trustd stores certificate information in SQLite databases. You can use the
lsof command-line tool to see which files
trustd has open. I haven't had time to analyze the databases yet. My theory is that there was some kind of configuration change on
valid.apple.com, and macOS for some reason is having trouble with the downloaded data. Thus, if you prevent connections to
valid.apple.com, you can avoid the issue.
That's what I know so far. I don't yet have a full technical analysis, because it's been less than 24 hours, I'm scrambling to diagnose the issue, and there are pesky non-computing activities such as sleep that get in the way. Hopefully more details later, and I encourage others to investigate the issue as well. By the way, I originally believed the issue was related to yesterday's Google Chrome update — which I installed on both of my machines right before noticing the issue — but that turned out to be just a coincidence, ruled out by totally uninstalling Chrome and also talking with other people experiencing the issue.
I believe I've discovered a "permanent" solution to the problem. This works on both my M1 Mac mini and my Intel MacBook Pro. I booted into recovery mode and renamed the
valid.sqlite3 files. (I kept the files instead of removing them, just in case.) These files are located in the directory
/private/var/protected/trustd/ on Big Sur and
/Library/Keychains/crls/ on Mojave. When I booted back into macOS, the files were regenerated. And the next time
nsurlsessiond connected to
valid.apple.com everything was fine. The CPU usage of
trustd did temporarily shoot up, but it went back to 0% within a few seconds and stayed there. So far, so good!
For forensic purposes I tried comparing the old and new databases, but they're both over 10 MB in size, and there were enough differences that I didn't feel like spending a lot of time on it. I've already spent too much time on this! My postmortem theory is that at some point Apple had some bad data in their Certificate Revocation List online, everyone downloaded the bad data, and then
trustd got stuck. Anyway, it looks like blocking
valid.apple.com is no longer necessary after removing and regenerating the
valid.sqlite3 database. As always, hack your system at your own risk, make backups, I cannot be held liable, yadda yadda.
If you're booted into macOS Recovery, be aware that the root volume in Terminal will be the recovery volume and not your main macOS volume, so you'll need to look for the latter in the
/Volumes/ directory. Also, Big Sur and Catalina have a read-only System volume, so the writable
valid.sqlite3 files will be found on the read-write "Data" volume. Moreover, if you have FileVault enabled, then your "Data" volume is encrypted at rest, so you'll need to unlock and mount that volume to get to the files. You can use the
diskutil apfs list command to list your Apple File System volumes and
diskutil apfs unlock to mount your "Data" volume. (On Catalina, the files are located in the
/Library/Keychains/crls/ directory on the Data volume.)
By the way, I've heard that the trustd high CPU usage issue has also been occurring on jailbroken iOS devices! I don't think the issue is related to jailbreaking; rather, the issue may occur on non-jailbroken iOS devices too, but you'd never know it, because iOS does not provide users with access to that information. Also there are no fans to rev up and make noise. Ignorance is bliss?