Revoking certain certificates on March 4

Don’t worry too much. Most browsers do not check if certificates are revoked, and your cached ocsp staple will probably be valid for some days.

Just renew the cert, you won’t be kicked offline at midnight UTC.

4 Likes

Hi everybody,

We’ve doubled the “Duplicate Certificate limit” (certificatesPerFQDNSet) limit for the duration of this incident.

6 Likes

@JamesLE @jillian @tdelmas @Phil_LE we have to renew a few thousand Certs on one of our CMS Servers … but we are running against Rate Limits (guess it’s the 300 Orders by 3 Hours per Account Rate Limit?!?) … need a quick Solution on that - as we need to renew thousands of Certs with that Account we haven’t got the Time to only do 300 renewals per 3 Hours … we can’t have thousands of Websites with revoked Certs … !!! We will effectively loose Customers if that happens …
thx
Andreas S.

Example of one of the Rate-Limited Domains: physio-woergl.at

429
{
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

Update 17:00 UTC:

Thank you for saving our A… :wink:

4 Likes

The bug might be considered as a violation to BR by CA/B and other agencies.
I believe Let’s Encrypt is now requesting an exception from all CA programs to not revoke all certificates (Mozilla).

3 Likes

We have to check CAA within 8 hours prior to issuance (per BRs §3.2.2.8). The domains affected did not have all of the necessary CAA checks so we have to revoke them.

We’re glad to hear that your certs were not issued in error, but we didn’t do proper CAA checks at issuance and will have to revoke them. Please renew and replace them before the deadline.

Unfortunately, we don’t have the time or staff to handle making sure everyone checks their certs. More importantly, we have to revoke because of the BRs.

4 Likes

A post was split to a new topic: Renewal redirect help

Great - now all my certs will renew on the same day. Well done!

6 Likes

if your netlify site’s auto-issued cert is affected, check this (manual steps necessary):

Update: Netlify have acknowledged (in the above linked forum thread) that they will take care of the issue for all their users.

3 Likes

Could you tell me please how to renew the certificate within ISPCOnfig? I tried to remove SSL and Letscencrípt and add it again, but the tool says the certification still need to be renewed!

Ok this is complete crap. Fine, there was a bug. I have less than 12 hours to fix this and right now I am 2500 miles from home on vacation, and all I have with me is a cell phone and a bad 3rd world internet connection. It was a complete fluke that the notification email even got to me before this weekend, as I am in Waikiki on a ship that pulls out in a few hours.

I deliberately manually updated certs to avoid this problem while I was gone, and because you guys are giving us literally no time to fix it before you revoke them you are not only screwing up my vacation I don’t even know if I can maintain a connection long enough to get this done. Nor do I know for certain that the one other person with access is available to talk to before you revoke.

4 Likes

In addition to this rate limit override, we just deployed another global rate limit override from 300/3h to 10000/3h for newOrdersPerAccount.

4 Likes

A post was split to a new topic: Baseline Requirements revocation requirement

I would be great if you could mention this in your blog. When I got the mail it looked like some phishing attempt, especially because the Let’s Encrypt homepage doesn’t mention this at all.

4 Likes

2 posts were split to a new topic: Public certificate hostnames

Is LE going to publish their after-incident summary publicly?

I’m excited to hear if there will be any lessons learned for your processes :slight_smile:

2 Likes

2 posts were split to a new topic: Filtering notices for unaffected certificates

We will definitely be conducting an internal post-mortem and will likely share some of it publicly. We have provided an initial Incident Report with more details including some remediation items. here

5 Likes

I had a DNS issue (my secondaries weren’t getting NOTIFY) so I’m getting too many failed authorization requests.

How long till that goes away?

(I fixed the notify issue, and validated that it works with your staging server).

Then they should have sent out the notice 5 days ago, not waited and then give us 12 hours to fix. 4 days ago I was still at home, not on vacation in the middle of the ocean

1 Like

Unfortunately, we have no way to know whether prior certs are still in use after they’ve been renewed. Renewal does not invalidate the old certificate, and some subscribers may use different certificates simultaneously on different endpoints for the same hostname (e.g. CDNs).

3 Likes