2020.02.29 CAA Rechecking Bug

After learning about and remediating a bug in our CAA checking code [1] on 2020-02-29 UTC (the evening of Friday February 28, U.S. Eastern time), we announced that we would be revoking approximately 2.6% of our active certificates that were potentially affected by the bug, totalling approximately 3 million certificates [2].

We announced the plan to revoke because even though the vast majority of the certificates in question do not pose a security risk, industry rules require that we revoke certificates not issued in full compliance with specific standards. These rules exist for good reasons. We work hard to comply with them and have an excellent track record for doing so.

Since that announcement we have worked with subscribers around the world to replace affected certificates as quickly as possible. More than 1.7 million affected certificates have been replaced in less than 48 hours. We’d like to thank everyone who helped with the effort. Our focus on automation has allowed us, and our subscribers, to make great progress in a short amount of time. We’ve also learned a lot about how we can do even better in the future.

Unfortunately, we believe it’s likely that more than 1 million certificates will not be replaced before the compliance deadline for revocation is upon us at 2020-03-05 03:00 UTC (9pm U.S. ET tonight). Rather than potentially break so many sites and cause concern for their visitors, we have determined that it is in the best interest of the health of the Internet for us to not revoke those certificates by the deadline.

Let’s Encrypt only offers certificates with 90 day lifetimes, so potentially affected certificates that we may not revoke will leave the ecosystem relatively quickly.

The following certificates have been, or will be, revoked prior to the compliance deadline at 2020-03-05 03:00 UTC (9pm U.S. ET tonight):

  • 1,706,505 certificates that we are confident were replaced during the incident period

  • 445 certificates that we treated as highest priority for revocation because, at the time we found the bug, they had CAA records that forbid issuance by Let’s Encrypt.

We plan to revoke more certificates as we become confident that doing so will not be needlessly disruptive to Web users.

I would like to thank the Let’s Encrypt team for tirelessly working to resolve this situation in the best way possible. It involved incredible effort and I couldn’t be more proud of what we have been able to get done in such a short amount of time.

[1] 2020.02.29 CAA Rechecking Bug

[2] Revoking certain certificates on March 4

16 Likes