Here’s the text of the notification e-mail:
We recently discovered a bug in the Let’s Encrypt certificate authority code, described here:
Unfortunately, this means we need to revoke the certificates that were affected by this bug, which includes one or more of your certificates. To avoid disruption, you’ll need to renew and replace your affected certificate(s) by Wednesday, March 4, 2020. We sincerely apologize for the issue.
If you’re not able to renew your certificate by March 4, the date we are required to revoke these certificates, visitors to your site will see security warnings until you do renew the certificate. Your ACME client documentation should explain how to renew.
If you are using Certbot, the command to renew is:
certbot renew --force-renewal
If you need help, please visit our community support forum: Revoking certain certificates on March 4
Please search thoroughly for a solution before you post a new question. Let’s Encrypt staff will help our community try to answer unresolved questions as quickly as possible.
Your affected certificate(s), listed by serial number and domain names:
[Or, if the list is very long, the e-mail has this text instead:]
You can find which of your certificates are affected by downloading the list of affected certificates at https://letsencrypt.org/caaproblem/ and searching for lines starting with account IDs:
The online tool seems only to support checking HTTPS on port 443.
For other services, the suggested
openssl s_client command, modified as appropriate, can be used. Here are some examples:
openssl s_client -connect example.com:25 -servername example.com -starttls smtp -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :
openssl s_client -connect example.com:587 -servername example.com -starttls smtp -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :
openssl s_client -connect example.com:143 -servername example.com -starttls imap -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :
openssl s_client -connect example.com:993 -servername example.com -showcerts </dev/null 2>/dev/null | openssl x509 -text -noout | grep -A 1 Serial\ Number | tr -d :
I appreciate the urgency of the issue BUT at this time the client is refusing to renew the certificate because of rate limits! Can those be lifted (or slightly increased) for the next two days? Otherwise, what should I do exactly?
# certbot renew --force-renewal Saving debug log to /var/log/letsencrypt/letsencrypt.log - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Processing /etc/letsencrypt/renewal/qa.foobar.com.conf - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - Plugins selected: Authenticator apache, Installer apache Renewing an existing certificate Attempting to renew cert (qa.foobar.com) from /etc/letsencrypt/renewal/qa.foobar.com.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: qa.foobar.com,qa.foobar.io: see https://letsencrypt.org/docs/rate-limits/. Skipping. All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/qa.foobar.com/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - All renewal attempts failed. The following certs could not be renewed: /etc/letsencrypt/live/qa.foobar.com/fullchain.pem (failure) - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - 1 renew failure(s), 0 parse failure(s)
If you can’t renew because of the duplicate certificate rate limit (“too many certificates already issued for exact set of domains”), you can temporary bypass it by creating a certificate with the domain(s) you want and additional domain.
That way, the new certificate will not be for the “exact set of domains”.
Keep in mind it’s a temporary measure that will complicate your configuration, use it only if absolutely necessary.
Also, it’s abnormal to have reach that limit. When doing testing you should use the staging environment: https://letsencrypt.org/docs/staging-environment/
We’re not testing. We’re urgently trying to renew our certs because of this bug.
When I try, the command “certbot renew --force-renewal -d MYDOMAIN” does not work. It returns something about an “invalid response” and “404 Not Found”. I have no idea why that is, and I’m trying to find out. That means trying several times, and now I can’t do that anymore, and I really don’t have time for this.
Could I suggest you make an announcement on Twitter (and any other social media channels you use)? Emails are notoriously unreliable…
Is the short timeframe before revoking the certificates something Let’s Encrypt is in control of? I’m just thinking if this had happened on a weekend, many people may not have picked up on this in time to act.
You mention a date of Wednesday 4th of March - that time is upon us in some parts of the world. Can you confirm in which time zone this date is to be considered? For example, is the cutoff exactly 2020-03-04T00:00Z (as in UTC)?
Also, it’s abnormal to have reach that limit.
I believe this is related to the configuration, as explained in this other thread:
My assumption is that because I have a
cli.ini file with
renew-by-default = True then it’s renewed every day, but again, this was the provided configuration when I installed the servers (also explained in the same thread)
Is that the reason why I am in this situation? If so. I am not alone. This config was the default one, and I assume a lot of other people will suffer from the same problem I am suffering.
When doing testing you should use the staging environment: https://letsencrypt.org/docs/staging-environment/
Yes, but this is not a drill. The deadline is tomorrow. What am I supposed to do? As soon as I received the email. I run the command.
you can temporarily bypass it by creating a certificate with the domain(s) you want and additional domain .
So I am assuming you will not temporarily lift the limits for these two days?
Did you checked your domain with https://unboundtest.com/caaproblem.html or did you only rely on the email? If your hypothesis is correct, maybe the certificate you are currently using is not affected, only an old certificate is.
I received the email about 75 minutes before it was March 4th in New Zealand. Giving a 75 minute heads up via email is absolute inacceptable.
I relied on the email and yup, you are absolutely right. The certificate has been already renewed. Dunno know what to do with the config, given the fact that it saved me…
I suggest you don’t keep it, because if you really do need to renew it, you can’t, if you hadn’t that config, your first renew after the email would have succeed and you wouldn’t have lost time and get stressed
Does certbot check if a certificate for revocation, and automatically initiate a renewal before the normal renewal watermark time?
I found the cause of the malfunction (some lingering experiments with IPV6 … perhaps the certbot can’t handle IPV6… Hm…). I removed the IPV6 DNS record and waited a while. Then it worked.
I’m glad only a dormant domain was affected. (That’s why I used if for experimentation with IPV6.)
If I’m not mistaken, no, certbot do not check the revocation status, only the expiration date: https://github.com/certbot/certbot/issues/1028
And, after the revocation, the previous (valid) OCSP response may be cached for a few days. So the revocation status may appear a few days later.
I know the account ID isn’t any sort of secret, and I get that this helps users look up their domains, but the the ACME spec had some privacy (and the subsequent usability) concerns which led to the whole POST-AS-GET method so I have to ask: was it a wise decision to include the account ID in this large dataset?
Why did you have to include the account id as well as all domains in the downloadable list?
- The users were notified by email
- There is an online check available already
This is probably not a sensitive information, but from an organization that issues certificates I would have expected a bit more sensitiveness
Just an FYI, the normal notification emails usually work fine for my account but I haven’t yet gotten the email warning of the pending revocation, despite having affected certs.
You are right, thanks!
But if I remove that, how the renewal would work? At the moment the good thing is that I do not need to renew it manually, I just have a line in my crontab that says “/usr/bin/certbot renew”.
If I move this to weekly and I remove the “renew-by-default = True”, would the renewal work unattended?