2021.09.07 Delay updating OCSP responses

At approximately 2021-09-06 01:00 UTC, Let’s Encrypt began serving OCSP responses which had not been updated in the previous 3.5 days, in violation of the Microsoft Trusted Root Program Requirements, Section C. At approximately 2021-09-07 05:00 UTC, Let’s Encrypt began serving OCSP responses which had not been updated in the previous 4 days, in violation of the Baseline Requirements, Section 4.9.10.

We became aware of the issue at 2021-09-07 14:00 UTC and began incident response at that time. We have deployed a few technical mitigations, which appear to have allowed our systems to keep up with OCSP updating volume, but not enough to work through the backlog and bring the current staleness back within program requirements. The root cause is not yet clear.

This incident should not affect the availability or reliability of any sites using Let's Encrypt certificates -- this is a policy violation, not an availability incident. OCSP responses are still being updated well before their 7-day expiration, and so are still valid for verification by browsers and other user-agents.

You can follow more detailed updates on this incident at 1729567 - Let’s Encrypt: Delay updating OCSP responses.

11 Likes

This incident was fully resolved as of 2021-09-08 05:58 UTC. A full incident report including timeline, root cause analysis, and remediation items will be posted to the Bugzilla ticket above in the next few days.

7 Likes

The full incident report, including detailed timeline and remediation items, is now available at 1729567 - Let’s Encrypt: Delay updating OCSP responses.

7 Likes