Last week, Mac users with HP printers were unable to print or run their printer software, because HP's code signing certificate was temporarily revoked. In this blog post I'll talk about how this works from a technical perspective, and clear up some misconceptions about the situation. Software distributed outside the Mac App Store, such as a printer driver from HP, is signed with a Developer ID code signing certificate. This certificate is issued by Apple's Developer ID Certification Authority (CA). There's a different CA, the Apple Worldwide Developer Relations Certification Authority, that's used for Mac App Store development. Information about Apple certificates and CAs can be found on the Apple PKI (Public Key Infrastructure) page. You can use the codesign command-line tool to see the certificate used to sign an app. For example, if you have my own app StopTheNews installed, use this command:
codesign --display --extract-certificates /Applications/StopTheNews.appThis creates the files codesign0, codesign1, and codesign2 in the current working directory. These files are in DER (Distinguished Encoding Rules) format, so you may want to give them the .der file extension, which allows you to preview the files with QuickLook. The 3 files are the 3 code signing certificates that make up the "chain of trust". In the case of StopTheNews, codesign0 is my own Developer ID Application cert, codesign1 is the Developer ID CA cert, and codesign2 is the Apple Root CA cert. The latter 2 certs are already in your System Roots keychain, which you can see in the Keychain Access app.
Apple's Developer ID CA does not have a Certificate Revocation List. The revocation status of a Developer ID cert is only available via Online Certificate Status Protocol. This is explained in the CA's Certification Practice Statement:
4.10. CERTIFICATE STATUS
The Developer ID Sub-CA utilizes Online Certificate Status Protocol (OCSP) to provide information whether a certificate has been revoked.
This is in contrast to the Apple WWDR CA, which use a CRL in addition to OCSP:
4.10. CERTIFICATE STATUS
The WWDR Sub-CAs utilize two methods for certificate validation, Certificate Revocation List (CRL) and Online Certificate Status Protocol (OCSP).
An unfortunate consequence of the lack of a Developer ID CRL is that you can't obtain a list of all revoked Developer ID certs. You can only query the status of known certs one-by-one.
When you launch a Developer ID signed app, it may perform an OCSP check. You can see this if you take a packet trace. The check consists of an HTTP GET request (port 80, unencrypted!) to ocsp.apple.com with a path that is both Base64 encoded and URL encoded. According to the OCSP format, the encoded bytes end with the serial number of the Developer ID cert. The OCSP check is performed by the trustd processes (/usr/libexec/trustd). An OCSP check is different and separate from a notarization check, which is performed by the syspolicyd process. Developer ID certs were always checked for revocation via OCSP, before notarization even existed.
The OCSP result is cached for a period of time, so a packet trace may not show another check the next time you launch an app. At a path of the format /var/folders/[RANDOM]/C/com.apple.trustd/ocspcache.sqlite3 there's an OCSP cache in an SQLite database. Enter the command getconf DARWIN_USER_CACHE_DIR in Terminal to get the directory for caches on your Mac. Since the result is cached, the current revocation status of a cert may not be reflected on your Mac.
As the Certificate Authority, Apple can revoke a Developer ID certificate at any time. This is done when Apple discovers that a cert has been used to sign malware. Unfortunately, we've seen cases where Apple has revoked a Developer ID cert mistakenly, such as with the indie developer Charlie Monroe. Is it possible for a developer to revoke their own Developer ID cert? The answer is no. From the Developer ID CA CPS:
4.8. CERTIFICATE REVOCATION
The Subscriber may initiate a revocation request by sending an email to productsecurity@apple.com. The request for revocation will then be evaluated by Apple.
Again, this is in contrast to the WWDR CA CPS:
4.8. CERTIFICATE REVOCATION
Certificates with Apple as the intended Subscriber are revoked at the request of an authorized Apple employee in accordance with internal business practices.
For Pass Certificates, Enhanced Pass Certificates, Apple Pay Pass Certificates, and Website Push Notification Certificates, the Subscriber may initiate a revocation request by sending an email to product-security@apple.com. The request for revocation will then be evaluated by Apple.
For all other certificates issued to Developers, the certificate revocation process will commence upon receipt of a valid request to revoke the Certificate from the Subscriber via the Apple Developer website or via Xcode. The Subscriber will be required to log in using their Apple ID and password. After authentication, the Subscriber will indicate that they wish to revoke their Certificate. Revocation of a Certificate cannot be undone. After revocation, a new Certificate must be requested.
The reason for this difference in policy is that revoking a Developer ID cert has severe consequences, as we've seen with HP printer software: Mac users will no longer be able to run software signed with the revoked cert. Developers are allowed to revoke their own Mac App Store code signing certificates, because those certs are only used for development purposes. Apps downloaded from the Mac App Store are not actually signed by the app developers, they're signed by Apple. Developers only use their Mac App Store certs to sign an app for upload to App Store Connect, or to test the app locally on their own Mac. Thus, revoking the development cert has no effect on Mac App Store customers. If you're a developer yourself, you can go to your Certificates, Identifiers & Profiles on the Apple Developer site and see the difference. Mac (and iOS) App Store certs have a "Revoke" button you can press, whereas Developer ID certs do not have a "Revoke" button. You have to email Apple Product Security to request that your Developer ID cert be revoked. You might do this if the private key for your cert becomes compromised. This happened for example to Panic. (However, Apple did not revoke Panic's Developer ID cert! Older versions of their apps still run.)
Except in cases of malware and compromised private keys, there's really not ever a good reason to revoke a Developer ID cert. Every code signing cert has an expiration date, but it's crucial to understand what this means. The valid dates of a code signing cert only apply to the use of the cert to sign software. After a code signing cert is expired, a developer can no longer use the cert to sign software. But any software that was signed with the cert before the expiration date is valid forever, unless the cert is revoked. In other words, the expiration date applies to signing software, not to running software. Users should always be able to run their validly signed software, for as long as the software is installed on their Mac.
With respect to the HP printer situation, there has been some dispute about who was at fault. Here's a press statement:
The Register understands from sources familiar with the matter that HP Inc asked Apple to revoke its printer driver code-signing certificates. It appears this request backfired as it left users unable to print. A HP Inc spokesperson told us on Friday night:
We unintentionally revoked credentials on some older versions of Mac drivers. This caused a temporary disruption for those customers and we are working with Apple to restore the drivers. In the meantime, we recommend users experiencing this problem to uninstall the HP driver and use the native AirPrint driver to print to their printer.
As my blog post makes clear, it is not possible for HP to unilaterally revoke their own Developer ID cert. HP had to contact Apple and request for the cert to be revoked. Apparently Apple granted that request. So blame must be apportioned to both companies. There have been no reports of malware or private key compromise. Therefore, no good reason exists for HP to request that their cert be revoked, and no good reason exists for Apple to grant that misguided request.
With great power comes great responsibility. Both HP and Apple failed in their responsibility, and Mac users suffered as a result.