I received the same email as everyone, claiming a problem due to CAA. However, my domains don’t use CAA records, so there’s no chance of this bug affecting them. Did LE check the suspect certificates to see whether the domains use CAA before invoking? I don’t know what the usage of the CAA record is, but I suspect you could have reduced the revocations if you’d checked DNS beforehand.
I understand what you’re trying to say, but your domains were affected by this bug as it’s possible your lack of CAA records was not checked at time of issuance. That’s why they have to revoke the certs. They were issued in a noncompliant way.
Your Akamai account Rep is certainly giving you better updates than mine. My rep did mention just some time ago that they are more “positive” now that we will be able to get our certs deployed prior to the new deadline. Do you mind sharing any other updates that you get from Akamai. Are they looking to do this big bang at 7:59 PM EST
Part of it comes from the messages in Luna. We had a couple of calls during the day (CET).
However this is typically a case of getting notified too late by LE, Akamai having a system in place that’s designed to renew certificates in a slower pace than currently required. I have to be honest, that current slowness is something I haven’t seen from it in the past 4 years. But it’s been rocksolid for us previously as well as support which has approached this matter for us and our customer proactively.
Hey, if someone stuck with dozens domains to check on you nginx web server. here is a handy little script. https://gist.github.com/sashareds/98095b8b7b322c8283dc7afbec658690
Just incase anyone is having this issue (Azure CDN managed cert), I got this health advisory this (AEST) morning.
Last update (4 h ago)
You have been identified as a customer using Let’s Encrypt certificates managed by Akamai CDN, who may be impacted by a Let’s Encrypt security event. Between 01:00 UTC on 05 Mar 2020 and 03:00 UTC on 05 Mar 2020, Let’s Encrypt will begin revoking 2.6% of their active certificates in-order to mitigate a security bug. See here for more information.
To check if you are using an affected certificate, pleased use this online tool: https://checkhost.unboundtest.com/
Next Steps: Akamai CDN is aware of this issue and are actively working to mitigate the issue. There will be no action needed from you at this time.
We have provided an update on the original incident regarding our revocation efforts:
CA/B OKed this? (Post must be at least 20 characters)
The decision to delay(or to avoid) revocation of certificates beyond the deadline requires an additional compliance report to be detailed according to CA/B Forum requirements.
@jillian can you clarify the current revocation plan? It seems to have gone from:
- revoking all starting @ 0000 UTC March 5, 2020
- revoking all starting @ 2000 UTC March 5, 2020
- revoking all the ones known to have been updated now and will be monitoring and dealing with the others … sometime later?
Found that update:
March 5, 2020 21:42 UTC
[Resolved] Over 1.7 million affected certificates have been replaced and were revoked. We will continue to revoke more as we become confident that doing so will not be needlessly disruptive to Web users. Please read our community forum update for full details community.letsencrypt.org/t/2020-02-29-caa-rechecking-bug/114591/3
We updated our FAQ this morning to reflect yesterday’s work.
I hope this clarifies the current state of revocations.
FWIW https://bugzilla.mozilla.org/show_bug.cgi?id=1619179#c7 has a more complete explanation, specifically:
Over the next 83 days we will continue to work with our Subscribers to get certificates replaced, and will continue to revoke certificates as they are replaced. Specifically, we will check at least twice a week for certificates that have been replaced, and revoke those that have. Additionally, when subscribers with large numbers of certificates notify us that their replacement process is complete, we will revoke those certificates.
After 83 days, all affected certificates will have expired, due to our 90-day certificate lifetime.
I got lucky. I renewed my certificate on the 29th of February. I didn’t get an email and only learned of the bug today. Even so, my domain with the certificate is on a private server. WAMP Apache. The online test wouldn’t work for the site. To err on the side of caution, I downloaded the serial-numbers text, which at 315 MB was too large for my text editors to handle. In future I suggest several smaller files.
Good luck to everyone with this bug.
you’re not supposed to check by hand
zgrep something filename.txt.gz
I used a little tool called Search&Replace that didn’t open the file, just searched it. But I had to reformat my serial-number – remove the colons and convert to lowercase. Without being able to see into the file, it was hard to know what string to search for. I take your point though. Ingenuity makes anything possible.
the unix world has powerful tools like
awk that can be really really useful
For next time maybe …
We have reverted the following rate limit overrides back to their pre-CAA Rechecking incident values:
- certificatesPerFQDNSet: 10 => 5
- invalidAuthorizationsPerAccount: 10 => 5
- newOrdersPerAccount: 10000 => 300
If your organization requires a rate limit override, please see the form in this document.