Revoking certain certificates on March 4

[Update 2020-03-05: The most up-to-date summary is at 2020.02.29 CAA Rechecking Bug]

Due to the 2020.02.29 CAA Rechecking Bug, we unfortunately need to revoke many Let’s Encrypt TLS/SSL certificates. We’re e-mailing affected subscribers for whom we have contact information.

This post and thread will collect answers to frequently asked questions about this revocation, and how to avoid problems by renewing affected certificates early. If you’re affected, please: thoroughly read this thread, and search the community forum, for an answer to your question. If you don’t find one, please make a new post to the “Help” category, filling in the questions in the template that appears as you compose your post.


Q: How many certificates are affected?
A: Updated 2020-03-05: Of the original 3,048,289 certificates that were affected, over 1.7M have been replaced and revoked. There are more than 1M whose status is either unknown or not replaced. Of the original 3M affected certificates, about 1M were duplicates of other affected certificates, in the sense of covering the same set of domain names.

Because of the way this bug operated, the most commonly affected certificates were those that are reissued very frequently, which is why so many affected certificates are duplicates.


Q: When will the revocations start?
A: Updated 2020-03-05: The following certificates were revoked prior to the compliance deadline at 2020-03-05 03:00 UTC (9pm U.S. ET 2020-03-04):

  • 1,706,505 certificates that we are confident were replaced during the incident period
  • 445 certificates that we treated as highest priority for revocation because, at the time we found the bug, they had CAA records that forbid issuance by Let’s Encrypt.

We plan to revoke more certificates as we become confident that doing so will not be needlessly disruptive to Web users. Please continue to renew and replace affected certificates. If there are any changes to revocation plans, updates will be provided in this thread. Thank you all very much for your patience, understanding, and help as we work through this issue.


Q: How do I know if I’m using an affected certificate?
A: Here is an online tool that will show you: https://checkhost.unboundtest.com/

Or, on a Linux/BSD-like system, this command will show you example.com's current certificate serial number:

openssl s_client -connect example.com:443 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

You can see the list of all affected serial numbers at: https://letsencrypt.org/caaproblem/


Q: I received the email telling me I should renew my certificate, however, the online testing tool isn’t indicating my cert needs replacing.
A: Even if you received an email, it’s possible that the affected certificates have been replaced by newer certs not affected by the bug. (Either due to being issued in the last few days since it was fixed, or simply by not meeting the specific timing criteria necessary for the bug to trigger.) In that case, it’s not necessary to renew them again. You can use the checking tool at https://checkhost.unboundtest.com/ to verify if the certificate you’re currently using needs renewal, or verify that the serial number of the cert you’re currently using is present in the list of affected certs at https://letsencrypt.org/caaproblem/.


Q: I get this error when trying to renew: "urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains"
A: This usually means your client has been renewing every day, which means you likely have a newer, unaffected certificate already. Check your hostname with https://checkhost.unboundtest.com/.

We’ve also increased the duplicate certificate limit so fewer people will get this error.


If you’d like to suggest more questions or corrections for this post, please make a new post to the “Site Feedback” category.

Thank you all very much for your patience, understanding, and help as we work through this issue.

16 Likes

Here’s the text of the notification e-mail:

We recently discovered a bug in the Let’s Encrypt certificate authority code, described here:

2020.02.29 CAA Rechecking Bug

Unfortunately, this means we need to revoke the certificates that were affected by this bug, which includes one or more of your certificates. To avoid disruption, you’ll need to renew and replace your affected certificate(s) by Wednesday, March 4, 2020. We sincerely apologize for the issue.

If you’re not able to renew your certificate by March 4, the date we are required to revoke these certificates, visitors to your site will see security warnings until you do renew the certificate. Your ACME client documentation should explain how to renew.

If you are using Certbot, the command to renew is:

certbot renew --force-renewal

If you need help, please visit our community support forum: Revoking certain certificates on March 4

Please search thoroughly for a solution before you post a new question. Let’s Encrypt staff will help our community try to answer unresolved questions as quickly as possible.

Your affected certificate(s), listed by serial number and domain names:

(certs)

[Or, if the list is very long, the e-mail has this text instead:]

You can find which of your certificates are affected by downloading the list of affected certificates at https://letsencrypt.org/caaproblem/ and searching for lines starting with account IDs:

(account IDs)

6 Likes

The online tool seems only to support checking HTTPS on port 443.

For other services, the suggested openssl s_client command, modified as appropriate, can be used. Here are some examples:

  • openssl s_client -connect example.com:25 -servername example.com -starttls smtp -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

  • openssl s_client -connect example.com:587 -servername example.com -starttls smtp -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

  • openssl s_client -connect example.com:143 -servername example.com -starttls imap -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

  • openssl s_client -connect example.com:993 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :

3 Likes

I appreciate the urgency of the issue BUT at this time the client is refusing to renew the certificate because of rate limits! Can those be lifted (or slightly increased) for the next two days? Otherwise, what should I do exactly?

# certbot renew --force-renewal
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/qa.foobar.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (qa.foobar.com) from /etc/letsencrypt/renewal/qa.foobar.com.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: qa.foobar.com,qa.foobar.io: see https://letsencrypt.org/docs/rate-limits/. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/qa.foobar.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/qa.foobar.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
2 Likes

If you can’t renew because of the duplicate certificate rate limit (“too many certificates already issued for exact set of domains”), you can temporary bypass it by creating a certificate with the domain(s) you want and additional domain.

That way, the new certificate will not be for the “exact set of domains”.

Keep in mind it’s a temporary measure that will complicate your configuration, use it only if absolutely necessary.

Also, it’s abnormal to have reach that limit. When doing testing you should use the staging environment: https://letsencrypt.org/docs/staging-environment/

3 Likes

We’re not testing. We’re urgently trying to renew our certs because of this bug.

When I try, the command “certbot renew --force-renewal -d MYDOMAIN” does not work. It returns something about an “invalid response” and “404 Not Found”. I have no idea why that is, and I’m trying to find out. That means trying several times, and now I can’t do that anymore, and I really don’t have time for this.

Hi,

Could I suggest you make an announcement on Twitter (and any other social media channels you use)? Emails are notoriously unreliable…

Is the short timeframe before revoking the certificates something Let’s Encrypt is in control of? I’m just thinking if this had happened on a weekend, many people may not have picked up on this in time to act.

Thanks

You mention a date of Wednesday 4th of March - that time is upon us in some parts of the world. Can you confirm in which time zone this date is to be considered? For example, is the cutoff exactly 2020-03-04T00:00Z (as in UTC)?

2 Likes

Also, it’s abnormal to have reach that limit.

I believe this is related to the configuration, as explained in this other thread:


My assumption is that because I have a cli.ini file with renew-by-default = True then it’s renewed every day, but again, this was the provided configuration when I installed the servers (also explained in the same thread)
Is that the reason why I am in this situation? If so. I am not alone. This config was the default one, and I assume a lot of other people will suffer from the same problem I am suffering.

When doing testing you should use the staging environment: https://letsencrypt.org/docs/staging-environment/

Yes, but this is not a drill. The deadline is tomorrow. What am I supposed to do? As soon as I received the email. I run the command.

you can temporarily bypass it by creating a certificate with the domain(s) you want and additional domain .

So I am assuming you will not temporarily lift the limits for these two days?

3 Likes

Did you checked your domain with https://unboundtest.com/caaproblem.html or did you only rely on the email? If your hypothesis is correct, maybe the certificate you are currently using is not affected, only an old certificate is.

6 Likes

I received the email about 75 minutes before it was March 4th in New Zealand. Giving a 75 minute heads up via email is absolute inacceptable.

8 Likes

I relied on the email and yup, you are absolutely right. The certificate has been already renewed. Dunno know what to do with the config, given the fact that it saved me…

4 Likes

I suggest you don’t keep it, because if you really do need to renew it, you can’t, if you hadn’t that config, your first renew after the email would have succeed and you wouldn’t have lost time and get stressed :slightly_smiling_face:

2 Likes

Does certbot check if a certificate for revocation, and automatically initiate a renewal before the normal renewal watermark time?

I found the cause of the malfunction (some lingering experiments with IPV6 … perhaps the certbot can’t handle IPV6… Hm…). I removed the IPV6 DNS record and waited a while. Then it worked.

I’m glad only a dormant domain was affected. (That’s why I used if for experimentation with IPV6.)

4 Likes

If I’m not mistaken, no, certbot do not check the revocation status, only the expiration date: https://github.com/certbot/certbot/issues/1028

And, after the revocation, the previous (valid) OCSP response may be cached for a few days. So the revocation status may appear a few days later.

4 Likes

I know the account ID isn’t any sort of secret, and I get that this helps users look up their domains, but the the ACME spec had some privacy (and the subsequent usability) concerns which led to the whole POST-AS-GET method so I have to ask: was it a wise decision to include the account ID in this large dataset?

3 Likes

Why did you have to include the account id as well as all domains in the downloadable list?

  1. The users were notified by email
  2. There is an online check available already

This is probably not a sensitive information, but from an organization that issues certificates I would have expected a bit more sensitiveness

1 Like

Just an FYI, the normal notification emails usually work fine for my account but I haven’t yet gotten the email warning of the pending revocation, despite having affected certs.

1 Like