Revoking certain certificates on March 4

I appreciate the urgency of the issue BUT at this time the client is refusing to renew the certificate because of rate limits! Can those be lifted (or slightly increased) for the next two days? Otherwise, what should I do exactly?

# certbot renew --force-renewal
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/qa.foobar.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (qa.foobar.com) from /etc/letsencrypt/renewal/qa.foobar.com.conf produced an unexpected error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for exact set of domains: qa.foobar.com,qa.foobar.io: see https://letsencrypt.org/docs/rate-limits/. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/qa.foobar.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/qa.foobar.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
3 Likes

If you can’t renew because of the duplicate certificate rate limit (“too many certificates already issued for exact set of domains”), you can temporary bypass it by creating a certificate with the domain(s) you want and additional domain.

That way, the new certificate will not be for the “exact set of domains”.

Keep in mind it’s a temporary measure that will complicate your configuration, use it only if absolutely necessary.

Also, it’s abnormal to have reach that limit. When doing testing you should use the staging environment: https://letsencrypt.org/docs/staging-environment/

4 Likes

We’re not testing. We’re urgently trying to renew our certs because of this bug.

When I try, the command “certbot renew --force-renewal -d MYDOMAIN” does not work. It returns something about an “invalid response” and “404 Not Found”. I have no idea why that is, and I’m trying to find out. That means trying several times, and now I can’t do that anymore, and I really don’t have time for this.

1 Like

Hi,

Could I suggest you make an announcement on Twitter (and any other social media channels you use)? Emails are notoriously unreliable…

Is the short timeframe before revoking the certificates something Let’s Encrypt is in control of? I’m just thinking if this had happened on a weekend, many people may not have picked up on this in time to act.

Thanks

You mention a date of Wednesday 4th of March - that time is upon us in some parts of the world. Can you confirm in which time zone this date is to be considered? For example, is the cutoff exactly 2020-03-04T00:00Z (as in UTC)?

3 Likes

Also, it’s abnormal to have reach that limit.

I believe this is related to the configuration, as explained in this other thread:

My assumption is that because I have a cli.ini file with renew-by-default = True then it's renewed every day, but again, this was the provided configuration when I installed the servers (also explained in the same thread)
Is that the reason why I am in this situation? If so. I am not alone. This config was the default one, and I assume a lot of other people will suffer from the same problem I am suffering.

When doing testing you should use the staging environment: Staging Environment - Let's Encrypt

Yes, but this is not a drill. The deadline is tomorrow. What am I supposed to do? As soon as I received the email. I run the command.

you can temporarily bypass it by creating a certificate with the domain(s) you want and additional domain .

So I am assuming you will not temporarily lift the limits for these two days?

4 Likes

Did you checked your domain with Check whether a host's certificate needs replacement or did you only rely on the email? If your hypothesis is correct, maybe the certificate you are currently using is not affected, only an old certificate is.

7 Likes

I received the email about 75 minutes before it was March 4th in New Zealand. Giving a 75 minute heads up via email is absolute inacceptable.

8 Likes

I relied on the email and yup, you are absolutely right. The certificate has been already renewed. Dunno know what to do with the config, given the fact that it saved me…

5 Likes

I suggest you don't keep it, because if you really do need to renew it, you can't, if you hadn't that config, your first renew after the email would have succeed and you wouldn't have lost time and get stressed :slightly_smiling_face:

3 Likes

Does certbot check if a certificate for revocation, and automatically initiate a renewal before the normal renewal watermark time?

1 Like

I found the cause of the malfunction (some lingering experiments with IPV6 … perhaps the certbot can’t handle IPV6… Hm…). I removed the IPV6 DNS record and waited a while. Then it worked.

I’m glad only a dormant domain was affected. (That’s why I used if for experimentation with IPV6.)

5 Likes

If I'm not mistaken, no, certbot do not check the revocation status, only the expiration date: Check OCSP as part of determining if the certificate is due for renewal · Issue #1028 · certbot/certbot · GitHub

And, after the revocation, the previous (valid) OCSP response may be cached for a few days. So the revocation status may appear a few days later.

5 Likes

I know the account ID isn't any sort of secret, and I get that this helps users look up their domains, but the the ACME spec had some privacy (and the subsequent usability) concerns which led to the whole POST-AS-GET method so I have to ask: was it a wise decision to include the account ID in this large dataset?

4 Likes

Why did you have to include the account id as well as all domains in the downloadable list?

  1. The users were notified by email
  2. There is an online check available already

This is probably not a sensitive information, but from an organization that issues certificates I would have expected a bit more sensitiveness

2 Likes

Just an FYI, the normal notification emails usually work fine for my account but I haven’t yet gotten the email warning of the pending revocation, despite having affected certs.

2 Likes

You are right, thanks!

But if I remove that, how the renewal would work? At the moment the good thing is that I do not need to renew it manually, I just have a line in my crontab that says “/usr/bin/certbot renew”.

If I move this to weekly and I remove the “renew-by-default = True”, would the renewal work unattended?

1 Like

Assuming the revocation will be applied at 00:00 UTC, you gave us less than 13 hours of warning via email despite holding onto this bug for days prior to the disclosure. Can I ask your reasoning for doing this, especially since you believe the bug may have been introduced many months ago?

The one domain I was notified of seems to not be affected due to a lucky biweekly renewal that coincidentally happened to occur last night. However, why weren’t more domains affected, given that I have many similarly configured multi-hostname certs? Since the issue was related to CAA verifications, why would the reported domain have been listed at all, which has no CAA record and should therefore have been considered valid both when the invalid certs were issued and when the bug was found?

This whole thing seems poorly handled and perhaps a little impulsive, given the short timelines for remedy and the miscalculated audience scope. Please elaborate so that individuals and organizations can trust this service again.

4 Likes

@msmm @nboelter Please the consider the timezone to be UTC. 00:00 UTC on 04 March 2020 is the earliest we will start the process of revocation.

5 Likes

Hi @bburhans,

I'd like to give my sincerest apologies for the short notice. We have been working as fast as we can since we discovered the bug to generate the complete list of affected domains and notify subscribers. The short revocation timeline is what the Baseline Requirements require of CAs who discover certificates that have been issued in error: Baseline Requirements for TLS Server Certificates – CAB Forum.

12 Likes