On 2020.06.08, we received a report from Miłosz Kaniewski with Fudo Security that our delegated OCSP signing certificate ‘OCSP Root-X1’ expired 4 days prior on 2020.06.04. This delegated signing certificate was reissued on 2020-06-09, 24 hours after the expiration was reported. During the period that it was expired but not replaced, TLS clients building chains to ISRG Root X1 would experience OCSP validation errors if checking OCSP and validating the signing certificate.
The OCSP responses served by our OCSP responder were signed as valid until December 2020, but because the signing certificate had expired, validation of the response would fail.
This affected the OCSP responses for the 2 non-expired intermediate certificates signed by ISRG Root X1: Let’s Encrypt Authority X3 and Let’s Encrypt Authority X4. All subscriber certificate OCSP responses are signed by the unexpired Let’s Encrypt Authority X3 and were unaffected.
The root cause of this incident is that an internal tool we use to generate OCSP responses for our intermediate signing certificates had a design flaw. The tool was able to generate a response with an expiration date later than the expiration of the certificate doing the signing. Additionally, the delegated OCSP signing certificate was not being monitored for expiration which allowed the expiration to pass without being noticed. We have fixed the internal tool as a remediation item and will be adding better monitoring for the new certificate.
Between 2020-06-04 00:00 UTC and 2020-06-09 00:00 UTC, there were 32,400,723 requests for the OCSP status of the affected intermediates, and 1,090,441,047 requests for the OCSP status of the cross-signed (unaffected) intermediates, leading us to estimate that approximately 2.9% of clients verifying Let’s Encrypt intermediates received an response with the expired OCSP signer during the outage.
Timeline
2020-06-08 17:53 UTC - Security officers received an encrypted email with incident details.
2020-06-08 18:29 UTC - A security officer decrypted the email report and began an investigation.
2020-06-08 18:44 UTC - The reported problem confirmed valid for TLS clients building chains to ISRG Root X1.
2020-06-08 18:58 UTC - Let’s Encrypt Staff coordinated datacenter access and began key ceremony preparations to issue a new delegated OCSP signing certificate.
2020-06-09 19:04 UTC - Completed key ceremony to sign the new certificate.
2020-06-09 22:28 UTC - New OCSP responses generated and served by the OCSP responder. Incident Resolved.
We apologize to our community for any errors they experienced while validating certificate chains to ISRG Root X1, and as always, will strive to do better in the future.