Cannot renew certificate "too many requests"

I am trying to renew my certificate but I received a too many requests for at least two of my domain names.

tail of the log file:

File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/letsencrypt/client.py”, line 228, in _obtain_certificate
authzr)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 310, in request_issuance
headers={‘Accept’: content_type})
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 634, in post
return self._check_response(response, content_type=content_type)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 550, in _check_response
raise messages.Error.from_json(jobj)
Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: Too many certificates already issued for: pridetechdesign.com

There’s a rate limit of 5 certificates per domain (TLD + 1, so subdomains count too) per 7 days in place.

You will have to wait a week.

Yes. You’ll get that error if you’ll just keep requesting certificates:

01:12:f9:53:70:03:61:ad:ea:26:b1:5d:95:3f:cf:f2:d8:7f on Jan 17 21:49:00 2016 GMT
01:cb:38:a7:5a:9a:6b:a1:cf:f6:ef:6a:7a:bf:b3:1e:46:8d on Jan 17 21:53:00 2016 GMT
01:d4:bf:c6:11:b4:69:57:9a:5e:6b:94:43:44:3c:19:de:39 on Jan 17 21:57:00 2016 GMT
01:a8:df:7c:ce:79:d0:c0:33:c4:89:f1:ee:4d:fe:35:67:66 on Jan 20 21:25:00 2016 GMT
and finally 01:df:a4:aa:0e:d7:e8:f2:48:4f:c8:ed:8a:93:e0:d1:d1:a3 on Jan 20 21:45:00 2016 GMT makes 5 certificates within 7 days…

And 4 out of 5 look quite the same?

If you’re not sure what you’re doing, you should experiment with the --test-cert option, so you can just test around without consequences and when you’re satisfied with everything, just once remove the --test-cert option…

That’s junk, I added new domain names, why didn’t that renew my certificate? And now I cannot renew because I added new domain names? Junk. Junk Junk Junk.

Please fix.

Once a certificate has been issued, it’s issued. Period. It’s signed, it’s submitted to the Certificate Transparency logs. Done deal.

If you “”“change”"" something to the certificate, you’ll get a new one. You just can’t “edit” a previously issued certificate… Which, of course, adds to the issued certificates for that domain and the rate limit. There’s no other way.

That isn’t “junk”, that’s just the way certificate issuing/signing works. And the rate limits are there for a very good reason. (OCSP signing for example.)

There’s nothing to fix, besides some manual/FAQ reading for some of us :slightly_smiling:

Adding a new domain or renewing a certificate is no different from issuing a new certificate. Both will create a new certificate, which is the rate-limited operation (necessary to limit the OCSP signing load).

If you regularly need certificates for new domains (i.e. a few times a week), a better option would be to request a new certificate just for that domain and use SNI. That way, a new domain won’t cause the rate limit for your existing domains to be increased. Note that this doesn’t apply to subdomains: They all count towards the rate limit of your “parent” domain (TLD+1).

Except they did not, the certificate expired jan 19th.

Something is wrong, and my server is now crippled by this. I understand this is beta but I am drawing your attention to this because it is a huuuuuuge issue.

There must be a foolproof way to renew certificates, with zero failures, for you guys to achieve your goals. Otherwise you need to throw out the 90 day renewals.

Yes, your original certificate expired on Jan 20 00:53:00 2016 GMT.

I don’t know how you ended up with all those quadruple identical certificates, but I’m quite sure it’s not the fault of Let’s Encrypt. There’s no way the official client would have done so without user influence and/or misconfigured cronjobs et cetera.

It’s hard to comment on this without knowing how you’re renewing your certificates. crt.sh clearly shows 5 domains requested for your domain in the last 7 days.

Without knowing how you’re invoking the client, we can’t really help you figure this one out. Let’s Encrypt currently doesn’t automatically add any cronjobs or other automation that might renew your certificate and fail in this way (this area is still very much a work in progress).

You might want to look into the --keep-until-expiring client flag, as well as --test-cert for testing your scripts to avoid running into this limit in production.

I was using this command to renew:

letsencrypt-auto certonly -v -t -c /etc/letsencrypt/domain.ini

domain.ini was based on sample configs.

I did add domain names recently, and I ran the command to add those domains and I assumed the new certificates would include new expirations but obviously that isn’t the case.

Perhaps the software failed to replace the old cert with the new certs.

There’s nothing to “update”. That’s not how certificates work. Once issued, always issued.

That’s not an issue with renewal then, but rather how you utilize the client.

When you add domains to your existing set of domains, during the same client execution, Let’s Encrypt will create a SAN certificate that covers all domains and replace the existing certificate with the new one. This will increase the rate limit for all domains included in that configuration, as they’re also included in the certificate.

If you regularly add new domains (i.e. multiple times a week - where the rate limit might come into effect), it might be better to issue separate certificates per domain. For example, you might originally run something like this:

letsencrypt-auto certonly -v -t -c /etc/letsencrypt/example.com.ini

… where example.com.ini contains a couple of subdomains of example.com.

Now let’s say you need to add a new domain example2.com. You would create a new .ini file example2.com.ini with only the new domain (and possibly subdomains) and invoke the client with:

letsencrypt-auto certonly -v -t -c /etc/letsencrypt/example2.com.ini

This would then issue a new certificate only for the domains defined in example2.com.ini, and would not affect the existing certificates for example.com. When it’s time to renew your certificates, you simply put both of those lines in your script and run that. I would suggest adding --keep-until-expiring as well so that your certificates are only renewed when they’re actually close to expiring (the client will just keep them as-is when they’re not).

On the server end, you can use SNI to use more than one certificate per IP address. This is widely supported by all modern browsers.

That’s what I did except it didn’t work today.

Also, the certificates do not appear replaced, at least the expiration dates are the same.

I would suggest testing this setup on a clean box with the staging environment, where you won’t be running into rate limits. If all goes well, you’d end up with two directories in /etc/letsencrypt/live/ and two different certificates/keys.

I’m certain this works, but if you encounter any issues on a clean installation please provide the step-by-step instructions and any errors so we can debug this.

This is a production server that is offline because of the failures in your renewal process. That is all.

It is obscene that you are blaming me. This is software meant to be completely automated. BETA does not mean “unfinished” it means release-candidate. Since renewals are not automated, and in situations like this, barely functional, “BETA” is generous.

I’m the biggest fan of this project. Renewal needs to be a top priority.

So how long do I have to operate without SSL because of this flaw in the renewal process? Is there a way to address the rate limitation manually?

For the record, I’m not affiliated with Let’s Encrypt or the ISRG. My contributions to the client can be counted on one hand.

No one is blaming you. You confirmed on IRC that there were some manual modifications in /etc/letsencrypt (missing symlink). That could lead to undefined behaviour in the client and make it hard to debug the exact problem that has caused this.

To reiterate: renewal is not where things went wrong here. It just happened to be the moment you ran into the rate limit, but that could’ve just as well happened when you were adding new domains.

You added domains to your configuration and ran the client manually. For whatever reason, you didn’t get a new certificate that only included those new domains, but rather replaced the existing one with new ones that included old and new domains. This increased the rate limit for all domains and in turn lead to the rate limit error during renewal.

Whether this happened due to a client bug, a documentation issue or because the wrong commands were used is hard to say after the fact.

I can only suggest you try the setup I recommended above on a clean installation using the staging server and see which certificates you end up with. If you end up with the same problem, please share all configuration files, client invocations and output/log files. Let’s Encrypt is beta software and certainly not perfect, but participating in a beta means you’re willing to invest the time to debug issues when they arise. If that’s not the case, you might be happier with a more traditional CA in the meantime. WoSign would be another free CA you could look into.

2 Likes

My thinking was it would be better to include all the appropriate domains under one certificate that way I wouldn’t run into this exact issue.

The rate limit makes some sense but it should probably be measured in minutes or hours, not days or weeks.

I’ve got 4 servers under one domain, while only the primary server is domain.com and the others are machine.domain.com, would using the client on one of those secondary servers effect the rate limitation for the primary server? I would assume not.

Does revoking an issued certificate allow the issuance of a new one, or will the rate limit still cause rejection?

I’m happy to help with the beta process. I was in a lousy mood earlier and my website was offline because of this so perhaps my frustration came through in my posts. My apologies if so, I will try to remain constructive and I do appreciate all efforts to help.

This comes down to personal preference, I guess. I would group certificates by the service they cover. For example your website would probably include example.com and www.example.com. Your mail server might include imap.example.com, smtp.example.com and webmail.example.com. An argument can be made to rather separate it by either the physical host they're used on, or by the software using it, or any combination of those things.

As you correctly noted, this might not always be possible due to the rate limits if you need to cover more than 5 services that include the same domain. SNI support might also be a concern, if you have to support very old clients.

It's certainly not the easiest thing to get right. The main concern or reason behind the rate limits is scaling the OCSP signing process. Let's Encrypt needs to sign OCSP responses for every non-expired certificate they have issued every (IIRC) few days. The key used for OCSP signing is stored in a Hardware Security Module, which is a rather expensive piece of equipment and supports only a limited amount of sign operations per second - so it's not the easiest thing in the world to scale. Currently there are about 350k unexpired certificates out there, and relaxing those rate limits too much in combination with a lot of first-time users, newly-written automation and not-yet-perfect client software could easily cause this number to spike quite rapidly, overwhelming the HSM, which would eventually lead to OCSP errors and delays for new certificates.

Hopefully as this project matures the client will learn to avoid requesting duplicate certificates (there've already been some improvements here) and provide better renewal support. Documentation could also be improved to avoid issues like this one. Additionally, once the limits are ready to be increased, a two-tiered rate limiting system could avoid issues where you essentially are banned from requesting additional certificates for an entire week. One option would be to have a rate limit of e.g. 3 certificates per 2 hours, 5 per day and 10 per week. Finally, the client (and CA server) could become better at exposing the current rate limit status to users and provide warnings when they are close to running into one. Unfortunately, there are only so many things a relatively small team can do at once.

Unfortunately yes. This specific rate limit is not bound to IP, Let's Encrypt registration or anything else. It simply goes by number of certificates per domain, or specifically TLD+1 (example.com, www.example.com, foo.bar.example.com all count towards the rate limit of example.com). The limit is per certificate, so using multiple subdomains of the same domain on one (SAN) certificate would increase the rate limit only by one.

Let's Encrypt needs to sign OCSP responses for all certificates until they expire, even for certificates that are revoked. Since this wouldn't affect the OCSP signing load, it doesn't decrease the rate limit.

I would recommend looking into WoSign or StartSSL (for non-commercial sites) as a stop-gap until your rate limits reset.

Thank you for your answers, they are very informative.

My interests are commercial. I have a purchased certificate however I am fully committed to this project as the future of encryption online. In addition to my own servers, I will also be utilizing Let’s Encrypt on hosted client sites, once it is out of beta.

So if the earliest issued certificates effecting the rate limit are from Jan 17th then I should be able to issue new certs starting on Jan 24th?

I’ll need to find a solution for the multiple servers… Maybe I’ll stagger my cert renewals with 1 server per week.

3 posts were split to a new topic: How to access issued certificates for a given domain