Articles index

Mac trustd high CPU

May 11 2021 by Jeff Johnson
Support this blog: StopTheMadness, Link Unshortener, Underpass, PayPal.Me

Six months after the Mac OCSP appocalypse, here we go again, and here I go again. Back then I discovered that the trustd process was having trouble connecting to ocsp.apple.com, an issue temporarily solved by preventing connection attempts to that domain (for example in Little Snitch). Yesterday I started noticing another issue with trustd, but this time the issue was a little different: very high CPU usage. I've heard from several other people who started noticing the same issue yesterday too, one of whom helpfully referred me to this reddit thread with even more reports. On investigation, I found that the nsurlsessiond process was connecting to the server valid.apple.com, and immediately afterward trustd CPU jumped from 0% to 100%. It seems that the issue can be temporarily solved by preventing nsurlsessiond from connecting to valid.apple.com. You may have to reboot or force quit trustd to get its CPU usage back to normal. It's important to note that this is only a temporary workaround to the CPU usage problem; trustd is an important macOS system process that checks certificate validity and revocation status, so you probably don't want to block valid.apple.com forever.

I can reproduce the issue easily on my M1 Mac mini running the latest macOS 11.3.1. I can also reproduce it, though not as reliably, on my Intel MacBook Pro running macOS 10.14.6 patched with the latest security updates. (I can hear the fans running on my MacBook Pro, which is how I first noticed the issue. Whereas I almost never hear the Mac mini fans.) I haven't been able to reproduce it booted into my external disk with macOS 11.2.1 or an unpatched 10.15.7, so this issue may be related somehow to the security updates on April 26. I'm going to try later to update those unpatched volumes, but I haven't had time yet, because the updates could take hours (sigh). In any case, I haven't spent as much time testing the external boot volumes, so I don't want to say definitely that the issue can't be reproduced on older macOS versions.

On my MacBook Pro, I set Little Snitch to ask for nsurlsessiond connection attempts. Here you see it attempting to connect to valid.apple.com right after I plug my MacBook Pro into the power adapter (more on that later). The CPU usage of trustd is 0% at that point.

trustd CPU 0.0%

I click the Allow button, and suddenly trustd CPU jumps way up.

trustd CPU 99.6%

Here's a screenshot from my Mac mini after I force quit trustd because of high CPU usage.

trustd CPU 0.0%

When trustd restarts, CPU usage jumps up again, and every time I see a message about com.apple.powerlog.plxpclogger.xpc in the Console log.

trustd CPU 93.9%

This happens about every 5 minutes on my Mac mini, which is of course always plugged in to the AC power. My impression is that this is all caused by some kind of macOS diagnostic logging related to power management, which is why it occurred on my MacBook Pro when I plugged in the power cord. (Note that I have Analytics disabled in the Privacy tab of the Security & Privacy pane of System Preferences on both Macs.)

I took a sample of the trustd process, and it's spending a lot of time in sqlite functions. Which makes sense, because trustd stores certificate information in SQLite databases. You can use the lsof command-line tool to see which files trustd has open. I haven't had time to analyze the databases yet. My theory is that there was some kind of configuration change on valid.apple.com, and macOS for some reason is having trouble with the downloaded data. Thus, if you prevent connections to valid.apple.com, you can avoid the issue.

That's what I know so far. I don't yet have a full technical analysis, because it's been less than 24 hours, I'm scrambling to diagnose the issue, and there are pesky non-computing activities such as sleep that get in the way. Hopefully more details later, and I encourage others to investigate the issue as well. By the way, I originally believed the issue was related to yesterday's Google Chrome update — which I installed on both of my machines right before noticing the issue — but that turned out to be just a coincidence, ruled out by totally uninstalling Chrome and also talking with other people experiencing the issue.

Addendum: Solution

I believe I've discovered a "permanent" solution to the problem. This works on both my M1 Mac mini and my Intel MacBook Pro. I booted into recovery mode and renamed the valid.sqlite3 files. (I kept the files instead of removing them, just in case.) These files are located in the directory /private/var/protected/trustd/ on Big Sur and /Library/Keychains/crls/ on Mojave. When I booted back into macOS, the files were regenerated. And the next time nsurlsessiond connected to valid.apple.com everything was fine. The CPU usage of trustd did temporarily shoot up, but it went back to 0% within a few seconds and stayed there. So far, so good!

For forensic purposes I tried comparing the old and new databases, but they're both over 10 MB in size, and there were enough differences that I didn't feel like spending a lot of time on it. I've already spent too much time on this! My postmortem theory is that at some point Apple had some bad data in their Certificate Revocation List online, everyone downloaded the bad data, and then trustd got stuck. Anyway, it looks like blocking valid.apple.com is no longer necessary after removing and regenerating the valid.sqlite3 database. As always, hack your system at your own risk, make backups, I cannot be held liable, yadda yadda.

Addendum: May 19 2021

If you're booted into macOS Recovery, be aware that the root volume in Terminal will be the recovery volume and not your main macOS volume, so you'll need to look for the latter in the /Volumes/ directory. Also, Big Sur and Catalina have a read-only System volume, so the writable valid.sqlite3 files will be found on the read-write "Data" volume. Moreover, if you have FileVault enabled, then your "Data" volume is encrypted at rest, so you'll need to unlock and mount that volume to get to the files. You can use the diskutil apfs list command to list your Apple File System volumes and diskutil apfs unlock to mount your "Data" volume. (On Catalina, the files are located in the /Library/Keychains/crls/ directory on the Data volume.)

By the way, I've heard that the trustd high CPU usage issue has also been occurring on jailbroken iOS devices! I don't think the issue is related to jailbreaking; rather, the issue may occur on non-jailbroken iOS devices too, but you'd never know it, because iOS does not provide users with access to that information. Also there are no fans to rev up and make noise. Ignorance is bliss?

Support this blog: StopTheMadness, Link Unshortener, Underpass, PayPal.Me

Articles index