Revoking certain certificates on March 4

You are right, thanks!

But if I remove that, how the renewal would work? At the moment the good thing is that I do not need to renew it manually, I just have a line in my crontab that says “/usr/bin/certbot renew”.

If I move this to weekly and I remove the “renew-by-default = True”, would the renewal work unattended?

Assuming the revocation will be applied at 00:00 UTC, you gave us less than 13 hours of warning via email despite holding onto this bug for days prior to the disclosure. Can I ask your reasoning for doing this, especially since you believe the bug may have been introduced many months ago?

The one domain I was notified of seems to not be affected due to a lucky biweekly renewal that coincidentally happened to occur last night. However, why weren’t more domains affected, given that I have many similarly configured multi-hostname certs? Since the issue was related to CAA verifications, why would the reported domain have been listed at all, which has no CAA record and should therefore have been considered valid both when the invalid certs were issued and when the bug was found?

This whole thing seems poorly handled and perhaps a little impulsive, given the short timelines for remedy and the miscalculated audience scope. Please elaborate so that individuals and organizations can trust this service again.

3 Likes

@msmm @nboelter Please the consider the timezone to be UTC. 00:00 UTC on 04 March 2020 is the earliest we will start the process of revocation.

4 Likes

Hi @bburhans,

I’d like to give my sincerest apologies for the short notice. We have been working as fast as we can since we discovered the bug to generate the complete list of affected domains and notify subscribers. The short revocation timeline is what the Baseline Requirements require of CAs who discover certificates that have been issued in error: https://cabforum.org/baseline-requirements-documents/.

11 Likes

Yes: Renewal By default

The bug was discovered the 29/02/2020 and they have 5 days to revoke them. It’s an obligation, they do not control it.

8 Likes

Pardon the potential misunderstanding on my part, but in reading the baseline requirements you linked to my understanding is that the domains being revoked very shortly are not all issued in error - merely affected by a bug that potentially didn’t apply all required validation checks before issuing.

I checked the list that was emailed to me, and none of my certificates (several hundred different applications worth) were actually issued in error - my tools authorized them all. Can large ACME accounts just attest that our domains were NOT in fact issued in error and avoid the revocation?

Perhaps a better way to handle this is inform people of the bug to review, and let them revoke certificates they don’t recognize.

2 Likes

Don’t worry too much. Most browsers do not check if certificates are revoked, and your cached ocsp staple will probably be valid for some days.

Just renew the cert, you won’t be kicked offline at midnight UTC.

4 Likes

Hi everybody,

We’ve doubled the “Duplicate Certificate limit” (certificatesPerFQDNSet) limit for the duration of this incident.

6 Likes

@JamesLE @jillian @tdelmas @Phil_LE we have to renew a few thousand Certs on one of our CMS Servers … but we are running against Rate Limits (guess it’s the 300 Orders by 3 Hours per Account Rate Limit?!?) … need a quick Solution on that - as we need to renew thousands of Certs with that Account we haven’t got the Time to only do 300 renewals per 3 Hours … we can’t have thousands of Websites with revoked Certs … !!! We will effectively loose Customers if that happens …
thx
Andreas S.

Example of one of the Rate-Limited Domains: physio-woergl.at

429
{
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}

Update 17:00 UTC:

Thank you for saving our A… :wink:

4 Likes

The bug might be considered as a violation to BR by CA/B and other agencies.
I believe Let’s Encrypt is now requesting an exception from all CA programs to not revoke all certificates (Mozilla).

3 Likes

We have to check CAA within 8 hours prior to issuance (per BRs §3.2.2.8). The domains affected did not have all of the necessary CAA checks so we have to revoke them.

We’re glad to hear that your certs were not issued in error, but we didn’t do proper CAA checks at issuance and will have to revoke them. Please renew and replace them before the deadline.

Unfortunately, we don’t have the time or staff to handle making sure everyone checks their certs. More importantly, we have to revoke because of the BRs.

4 Likes

A post was split to a new topic: Renewal redirect help

Great - now all my certs will renew on the same day. Well done!

6 Likes

if your netlify site’s auto-issued cert is affected, check this (manual steps necessary):

Update: Netlify have acknowledged (in the above linked forum thread) that they will take care of the issue for all their users.

3 Likes

Could you tell me please how to renew the certificate within ISPCOnfig? I tried to remove SSL and Letscencrípt and add it again, but the tool says the certification still need to be renewed!

Ok this is complete crap. Fine, there was a bug. I have less than 12 hours to fix this and right now I am 2500 miles from home on vacation, and all I have with me is a cell phone and a bad 3rd world internet connection. It was a complete fluke that the notification email even got to me before this weekend, as I am in Waikiki on a ship that pulls out in a few hours.

I deliberately manually updated certs to avoid this problem while I was gone, and because you guys are giving us literally no time to fix it before you revoke them you are not only screwing up my vacation I don’t even know if I can maintain a connection long enough to get this done. Nor do I know for certain that the one other person with access is available to talk to before you revoke.

4 Likes

In addition to this rate limit override, we just deployed another global rate limit override from 300/3h to 10000/3h for newOrdersPerAccount.

4 Likes

A post was split to a new topic: Baseline Requirements revocation requirement

I would be great if you could mention this in your blog. When I got the mail it looked like some phishing attempt, especially because the Let’s Encrypt homepage doesn’t mention this at all.

4 Likes

2 posts were split to a new topic: Public certificate hostnames