On Thursday, the day that macOS 11 Big Sur was released, Apple's Developer ID Online Certificate Status Protocol (OCSP) service went down. This seems to have been part of a larger outage affecting a number of Apple services, as indicated by their System Status. When you launch a Mac app, macOS may check with Apple's Developer ID OCSP to see whether the app developer's code signing certificate is revoked. Since 2012, macOS (then known as Mac OS X) has required that all apps downloaded from the web (outside the Mac App Store) be signed with a valid Developer ID certificate, issued by Apple to developers. The purpose of Developer ID, according to Apple, is to prevent the spread of malware; if Apple discovers that a developer has distributed malware, Apple will revoke that developer's code signing cert, and then macOS will prevent any software signed with that cert from launching, thus protecting Mac users. Unfortunately, if there's an internet connection problem involving the Developer ID OCSP, that can also prevent Mac apps from launching. For several hours on Thursday, Mac users around the world experienced extreme slowness when launching their installed apps. It's possible that millions of Macs were affected by this OCSP problem, a major if short-lived computing disaster. Many Mac users, completely unaware of why their apps wouldn't launch, feared that there was a problem with their operating system, or even with their hardware.
This actually wasn't the only Developer ID disaster recently. A few weeks ago I wrote another blog post after Apple temporarily revoked HP's Developer ID cert, which caused a widespread failure of HP printer software. I described Apple's OCSP service in detail in that blog post, so I won't repeat the details here. What I wish to mention now is something I remember from writing that blog post: if I recall correctly, Developer ID OCSP responses were standardly cached by macOS for 5 minutes. This means that if you launch an app more than once within a 5 minute period, macOS won't check OCSP every time, it will just use the last cached response. The reason I mention the cache period is that it appears Apple has greatly increased it, from 5 minutes to half a day, likely in order to mitigate the problems caused by Thursday's outage. I noticed today that macOS seemed to be checking OCSP much less frequently, which led me to investigate. Apple can adjust OCSP cache periods without a macOS update, because they're determined by the OCSP responses.
You can see an OCSP response by taking a packet trace. But how do you trigger an OCSP check if the responses are all cached for a long period of time? Easy: download a Mac app from a developer whose apps you don't already have installed. (To be safe, make sure the developer is well-known. And while you're at it, you might as well buy their app, because indie Mac devs need your support!) You might not even need to launch the app, because Finder usually triggers an OCSP check when you unzip a downloaded app. A good tool for analyzing packet traces is Wireshark, because Wireshark automatically understands OCSP. The OCSP response has timestamps for "producedAt", "thisUpdate", and "nextUpdate", though usually the first two are the same time. The response is valid until the "nextUpdate", and the difference between "producedAt" and "nextUpdate" is how long the response can be cached.
By looking at packet traces, we can see what Apple's OCSP is doing now. But how do we know what Apple's OCSP was doing before Thursday? That's what backups are for, of course! OCSP responses are cached, and you can look at the cache file from your backups to see previous OCSP responses. As I mentioned in my earlier blog post, the OCSP cache is located at com.apple.trustd/ocspcache.sqlite3 inside your DARWIN_USER_CACHE_DIR (use the command getconf DARWIN_USER_CACHE_DIR in Terminal to get that directory). The cache is an SQLite database, so you can use sqlite3 [path] .dump to see the contents. Of the two tables in the db, responses interests us here. You can see the table format:
CREATE TABLE responses(responseId INTEGER PRIMARY KEY,ocspResponse BLOB NOT NULL,responderURI BLOB,expires DOUBLE NOT NULL,lastUsed DOUBLE NOT NULL);The relevant responderURI for our purpose is X'687474703a2f2f6f6373702e6170706c652e636f6d2f6f6373702d64657669643031', the hexadecimal representation of http://ocsp.apple.com/ocsp-devid01, so search for that. (The cache contains responses from multiple OCSP services, not just Developer ID.) The ocspResponse is also a hexadecimal representation, so you can paste it into a hex editor. The timestamps in the response are of the form 20201113232122Z (2020-11-13 23:21:22 UTC).
I didn't have a lot of data, but the data that I found did indicate that macOS was caching OCSP responses for 5 minutes before Thursday and half a day now. I hope that other people will investigate this too. I think it's important for us outside of Apple to know what's happening with our computers. I also hope that Apple itself will make a public statement about yesterday's incident, but sadly Apple's history of "postmortems" is almost nonexistent.
One more thing (heh). Developer ID should not be confused with Mac app notarization. As I said, the former requirement was imposed in 2012, while the latter was not imposed until just last year. Notarization is an addition to and not a replacement for Developer ID. An app distributed outside the Mac App Store needs to be signed with a valid Developer ID cert, and then it needs to be uploaded to Apple for notarization. After the app is notarized, the notarization "ticket" can be "stapled" to the app. (Stapling is optional but recommended.) I explained notarization checks in more detail in yet another blog post. A crucial difference between OCSP and notarization is that the latter is only checked on first launch of the app. The notarization status is cached permanently and has no expiration, unlike OCSP. Thus, notarization only affects your ability to install new apps, it doesn't affect your ability to launch already installed apps.