Revoking certain certificates on March 4

Help
#158

I just read through the incident report and noticed these three lines:

2020-02-29 03:08 UTC: Let’s Encrypt engineer confirms bug.
2020-02-29 03:10 UTC: Let’s Encrypt SRE team disables issuance.
2020-02-29 05:22 UTC: Fixed version of Boulder is deployed (containing https://github.com/letsencrypt/boulder/pull/4690). Issuance is re-enabled.

This is an absolutely amazing response time for both disabling issuance, and fixing/testing/re-enabling.

I know this is a stressful and frustrating situation, and many end-users are expectedly upset, but the emergency performance of the LetsEncrypt staff here is simply amazing and deserves commendation.

14 Likes
#159

They seem to have failed at this. We’re past the deadline now and they still haven’t deployed.

(edit: this was a reply in regards to Akamai-managed LE certificates; apologies for any confusion. (they referred to Akamai in my post))

#160

EDIT: Think, I misread your comment and thought that you said le missed the deadline. Gonna leave this here, tho:

Is that really a bad thing tho? The fact that the deployment wouldn’t be at once was clearly stated. They got to have it deployed in 5.5 hrs tho.
Every minute won from deploying later are potentially also won for end users (especially those that use outlook) from not having that problem - and of course for their respective IT departs for renewing the cert.

#161

There’s an extension in coordination with LE.

#162

sorry for the confusion. I edited my post to reflect the unclear wording. I was referring to Akamai’s handling of their Akamai-managed LE certificates.

1 Like
#163

And Akamai is still not committing to be able to hit that extended deadline, currently.

#164

A post was split to a new topic: Are single-hostname certificates affected?

#165

I received the same email as everyone, claiming a problem due to CAA. However, my domains don’t use CAA records, so there’s no chance of this bug affecting them. Did LE check the suspect certificates to see whether the domains use CAA before invoking? I don’t know what the usage of the CAA record is, but I suspect you could have reduced the revocations if you’d checked DNS beforehand.

1 Like
#166

I understand what you’re trying to say, but your domains were affected by this bug as it’s possible your lack of CAA records was not checked at time of issuance. That’s why they have to revoke the certs. They were issued in a noncompliant way.

6 Likes
#167

Your Akamai account Rep is certainly giving you better updates than mine. My rep did mention just some time ago that they are more “positive” now that we will be able to get our certs deployed prior to the new deadline. Do you mind sharing any other updates that you get from Akamai. Are they looking to do this big bang at 7:59 PM EST :slight_smile:

1 Like
#168

Part of it comes from the messages in Luna. We had a couple of calls during the day (CET).

However this is typically a case of getting notified too late by LE, Akamai having a system in place that’s designed to renew certificates in a slower pace than currently required. I have to be honest, that current slowness is something I haven’t seen from it in the past 4 years. But it’s been rocksolid for us previously as well as support which has approached this matter for us and our customer proactively.

1 Like
#169

Hey, if someone stuck with dozens domains to check on you nginx web server. here is a handy little script. https://gist.github.com/sashareds/98095b8b7b322c8283dc7afbec658690

1 Like
#170

Just incase anyone is having this issue (Azure CDN managed cert), I got this health advisory this (AEST) morning.

Last update (4 h ago)

You have been identified as a customer using Let’s Encrypt certificates managed by Akamai CDN, who may be impacted by a Let’s Encrypt security event. Between 01:00 UTC on 05 Mar 2020 and 03:00 UTC on 05 Mar 2020, Let’s Encrypt will begin revoking 2.6% of their active certificates in-order to mitigate a security bug. See here for more information.

To check if you are using an affected certificate, pleased use this online tool: https://checkhost.unboundtest.com/

Next Steps: Akamai CDN is aware of this issue and are actively working to mitigate the issue. There will be no action needed from you at this time.

3 Likes
#171

We have provided an update on the original incident regarding our revocation efforts:

7 Likes
#174

CA/B OKed this? (Post must be at least 20 characters)

#177

The decision to delay(or to avoid) revocation of certificates beyond the deadline requires an additional compliance report to be detailed according to CA/B Forum requirements.

5 Likes
#180

@jillian can you clarify the current revocation plan? It seems to have gone from:

  • revoking all starting @ 0000 UTC March 5, 2020
    to
  • revoking all starting @ 2000 UTC March 5, 2020
    to
  • revoking all the ones known to have been updated now and will be monitoring and dealing with the others … sometime later?
1 Like
#181

Found that update:

https://letsencrypt.status.io/pages/history/55957a99e800baa4470002da

March 5, 2020 21:42 UTC

[Resolved] Over 1.7 million affected certificates have been replaced and were revoked. We will continue to revoke more as we become confident that doing so will not be needlessly disruptive to Web users. Please read our community forum update for full details community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591/3

6 Likes
#182

We updated our FAQ this morning to reflect yesterday’s work.

I hope this clarifies the current state of revocations.

6 Likes
#183

FWIW https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c7 has a more complete explanation, specifically:

Over the next 83 days we will continue to work with our Subscribers to get certificates replaced, and will continue to revoke certificates as they are replaced. Specifically, we will check at least twice a week for certificates that have been replaced, and revoke those that have. Additionally, when subscribers with large numbers of certificates notify us that their replacement process is complete, we will revoke those certificates.
After 83 days, all affected certificates will have expired, due to our 90-day certificate lifetime.

1 Like