I seem to have trouble understanding how the renewal exemption applies to the rate limit.
After hitting the “Certificates per Registered Domain (20 per week)” limit, I cannot issue a new certificate for my domain. But still I can renew old ones, “Even if you’ve hit the limit for the week, you can still issue new certificates that count as renewals.”
However, the order of issuing new certificates and renewing the old ones seem to matter. Let’s say that I’d like to create 10 new certificates, and there are 15 certificates to renew, all for the same subdomain (but for different clients).
If I create the 10 new certificates (10× certbot-auto certonly), and then I renew the old ones (1× certbot-auto renew), all requests are successful. I created a total of 25 new certificates, which is more than 20, but 15 of those were renewals.
However if I renew the 15 old certificates (1× certbot-auto renew), then I try to create the 10 new ones (10× certbot-auto certonly), only the first 5 will be successful, because after 15 renewals + 5 new ones, I hit the limit of 20 per week, and I get an error message.
Is this the way how the rate limit is handled? The strange thing is that if I issue the commands in the “new-and-then-old” order, the load for Let’s Encrypt servers is 25 certificates, ie. I am allowed to go above 20. But when issuing the commands in the “old-and-then-new” order, I am limited after 20, even if the number of requested certificates would have been the same, 25 of them.
What you’re describing matches my understanding of the way rate-limiting works right now.
I don’t think forcing you to use a particular order in which you request your certificates if you want to cover new FQDNs was a conscious design decision. Initially, the rate limiting was only implemented on the Certificates per Registered Domain (TLD plus one label) level. At a later point, an exception was added for renewals, and at the same time a separate rate-limit for identical certificates (covering the same set of FQDNs) was added. The order being relevant is probably just a side-effect of this being implemented on top of the existing rate limit and database schema.
There are some ideas floating around on how the rate-limiting might be improved in the future, possibly switching to something like a tocken bucket algorithm to avoid situations where you can’t issue anything for an entire week. That’s probably going to take a while though.
If this is correct, it’s a serious bug IMO. Because this behaviour will probably stop people from auto-renewing often if they have to fear that they cannot issue new certs.
I’ll check it on a test domain these days and post the results here.
I did a series of tests and I can confirm the behaviour described above.
My test setup:
a webserver with nginx + le-scripts installed
an unused domain I never before used with lets encrypt
a bunch of subdomains on this domain pointing to the web server: [t1...t9].mydomain
I started to create one certificate and renewed it a few times (forcing renew). I could renew it 5 times, the 6th time I got:
urn:acme:error:rateLimited Error creating new cert :: Too many certificates already issued for exact set of domains
Then I created a new certificate and so on. This way I created 6 new certs and did 13 renewals within a few minutes.
Action 20 was to create a new certificate which succeeded.
Action 21 was to create a new certificate which failed:
urn:acme:error:rateLimited Error creating new cert :: Too many certificates already issued for: (mydomain)
So renewal also count's up the new cert limit which is really an issue!
Here's the complete test protocol (times in UTC):
6.2. 21:11 t1 new 1
6.2. 21:12 t1 renew 2
6.2. 21:14 t1 renew 3
6.2. 21:15 t1 renew 4
6.2. 21:16 t1 renew 5
6.2. 21:19 t2 new 6
6.2. 21:20 t1 renew 7
6.2. 21:20 t2 renew 8
6.2. 21:21 t1 renew resulted in urn:acme:error:rateLimited
Error creating new cert :: Too many certificates already issued for exact set of domains
6.2. 21:23 t2 renew 9
6.2. 21:26 t3 new 10
6.2. 21:26 t4 new 11
6.2. 21:27 t2 renew 12
6.2. 21:28 t3 renew 13
6.2. 21:28 t4 renew 14
6.2. 21:30 t2 renew 15
6.2. 21:31 t2 renew 16
6.2. 21:21 t2 renew resulted in urn:acme:error:rateLimited
Error creating new cert :: Too many certificates already issued for exact set of domains
6.2. 21:35 t5 new 17
6.2. 21:35 t6 new 18
6.2. 21:36 t6 renew 19
6.2. 21:38 t7 new 20
6.2. 21:35 t8 new resulted in urn:acme:error:rateLimited
Error creating new cert :: Too many certificates already issued for: (mydomain)
6.2. 21:38 t7 renew 21
I’m running into the same thing. Had quite some renewals this week because i bulk-migrated some servers from another CA when i made the switch several months ago. We use customer_name.domain.com. Now that a bunch of these subdomains have been renewed in the past couple of days, we cannot create new certificates.
I think the implementation must me changed. With the current one, it’s not possible to register a large number of domain slowly (right?), that contradict previous statements.
It is actually possible to register any number of domains slowly. Every week you can increase the number of domains by 20, and after registering the new ones, doing the renewal of the old ones.
@czirkoszoltan is correct. The thing to note is that if you have a large number of subdomains, clumping up their renewals around the same time will help to prevent renewals from using up your rate limit for fresh issuance.
We’ve definitely considered changing the implementation, but it’s a bit tricky, and so far the number of people running into trouble with the situation has been low enough that we’ve prioritized other things higher. However, we’ll definitely keep considering it. Thanks!
This process works, but requires coordination between all hosts requesting certs. This is very hard to achieve e.g. in a university setting, where independent departments share a single domain.
I’m active for freifunk.net. We hand out subdomains as A Records or DNS Zones to a lot of freifunk communities.
There are 396 communities at the moment and we run constantly into rate limits. As we are decentrally organized there’s no chance of coordinating a common time to renew all certificates. We also applied twice for increased ratelimits for freifunk.net, but there was no answer.
So we prefer the change in implementation over the change in documentation
In my opinion, changing the implementation is the way to go here.
Even with the renewal of certs clumped up to, let’s say, once a month, that still means seven following days where no new certificate can be issued. This seems like an unnecessary limitation to have, especially considering that there seem to be no negative consequences from not having renewals count against the rate limit at all.
As a non-developer, I would love some insight into the drawback it might have.
Hi, unfortunately I ran into same issue. Please could you change your implementation to not count certificates renewal into new certificates limit of 20 certificates per weak? I will change my own implementation to first create new certificates and then make renewal, but at one point there will be no place for new certificates because we need 700+ certificates.
Can you at least give us a time window for resolution of the problem since in the moment we have to either lower/drop security or start paying for certificates and charge our customers more.
Just another friendly request to change this behavior, so that renewals don’t count against the global limit. We have a few dozen subdomains and we are slowly adding more; it’s not a great number, but there are enough renewals that sometimes we are blocked from adding any new subdomains for a whole week. If I understand right, making renewals not count should not open any loopholes into the system, and would be really helpful for those of us who need to add new subdomain certificates every now and then.
I think that issue should have a higher priority. The current way how the limits are handled actively prevent users of >= 20 subdomains to use a specifically crafted renewal algorithms.
And even then there is the possibility that after a need to create a new subdomain, one has to wait a week (possibly with delaying further renewals) before putting it online.
To me it seems, this behaviour actively contributes to the fact, that wildcard certificates are way more useful (and for this time, not because of the inability to automatize that some people seem to have)
I wanted to update this thread since we recently announced a change that removes the constraint on order of renewals versus new issuance. The new behaviour grants the renewal exemption regardless of the order of operations and should be much easier to work with.
