My team has been scratching our heads on a recent rate-limit that we've been hitting for the 50 Certificates per registered domain per week.
We have been seeing the rate limiting applied despite not requesting > 50 "new" certificates (allegedly.. I could absolutely be wrong here).
The docs on rate limits make it seem as though the only qualification for something to be called a renewal and not a new certificate is the exact set of domains already being requested before**.
(**presumably the immediate-past certificate with that exact set of domains is not lapsing in expiration when you go to request the certificate again, otherwise it doesn't feel like a renewal?—that bit isn't totally clear but renewal isn't exactly the precise word anyway)
Maybe we're doing something wrong in our requests, but it's not jumping out at us here.
We don't seem to be requesting more than 50 new certs per 7 day period,
We are frequently renewing more than 50 certs per 7 day period (around 500+ active certs based on our crt.sh logs)
We renew at the 60 day mark, and a new private + CSR is used to make the request, but with the same exact set of hostnames.
We can try going down the path of requesting a rate-limit increase, but if something is amiss here, that wouldn't necessarily solve a problem.
Any help would be appreciated—thanks!
Kyle
Boilerplate Support Details (domain, commands, output) below:
@khakala I am equally puzzled why you would see that error if it is as you describe.
My understanding is a cert is considered a renewal when it has the identical set of hostnames in it. So, as you note there are not more than 50 new certs weekly.
From the list Bruce showed there were 104 for that apex domain but many of these were likely renewals and should not count against the 50. And, if that wasn't correct something is wrong to allow you 104 when the limit is 50.
I think we will need someone from staff or more experienced with this. I recall there are some subtle nuances involved but don't recall the details.
I am tempted to flag staff but I think someone will respond soon enough just based on the title.
We've been using letsdebug to get some more insights into this, but we're seemingly not creating new certificates, only renewals based on the same domain name set.
We run daily renewals to even out our requests a bit, though not perfectly distributed. This has been a steady accumulation of about 500 active certs over the last 2 years or so.
The staging environment is indeed helpful in hammering out some of our request tidbits, but given the context of this limit, it didn't seem to be a great approach to try replicating this type of error there (and up to the higher limits of staging as well). We could definitely repoint our automation jobs there, but it didn't feel quite right to bring that into the mix until gathering more info here—let me know what you think!
At what time and date did you encounter the rate limit message?
Let's Debug won't accurately represent things because of two reasons: the number of certificates it has to traverse is too large, so it truncates some of the result set, and renewal exemptions require too much load on crt.sh to calculate, so they're not.
By my calculation, you are only at 21 non-renewal certificates for the last 7 day period. You shouldn't be hitting the limit, and indeed at the moment I don't think Let's Encrypt is rate limiting you - I can create new orders against your domain.
Edit: I tried a different data source and method of calculating, get approximately the same thing:
Cert [qa1.nmsm-core.nmdp.org] at Thu Feb 16 04:53:20 2023 was not covered by renewal exemption
Cert [*.app.nmdp.org app.le.nmdp.org] at Thu Feb 16 17:09:53 2023 was not covered by renewal exemption
Cert [dev3.nmsm-core.nmdp.org] at Thu Feb 16 16:28:19 2023 was not covered by renewal exemption
Cert [qa3.nmsm-core.nmdp.org] at Thu Feb 16 16:44:25 2023 was not covered by renewal exemption
Cert [dev3.nmsm-activemq.nmdp.org] at Fri Feb 17 19:27:53 2023 was not covered by renewal exemption
Cert [qa3.coreactivemq.nmdp.org] at Fri Feb 17 16:52:39 2023 was not covered by renewal exemption
Cert [dev3.nmsm-admin.nmdp.org] at Thu Feb 16 16:24:41 2023 was not covered by renewal exemption
Cert [dev1.nmsm-core.nmdp.org] at Wed Feb 15 18:45:04 2023 was not covered by renewal exemption
Cert [qa3.nmsm-activemq.nmdp.org] at Thu Feb 16 16:38:22 2023 was not covered by renewal exemption
Cert [dev1.nmsm-admin.nmdp.org] at Tue Feb 14 13:39:49 2023 was not covered by renewal exemption
Cert [*.testapplication.nmdp.org acme.le.testapplication.nmdp.org] at Thu Feb 16 16:36:03 2023 was not covered by renewal exemption
Cert [dev1.nmsm-activemq.nmdp.org] at Thu Feb 16 05:02:40 2023 was not covered by renewal exemption
Cert [qa3.nmsm-admin.nmdp.org] at Thu Feb 16 16:47:40 2023 was not covered by renewal exemption
Cert [dev1.nmsm-intake.nmdp.org] at Wed Feb 15 18:48:56 2023 was not covered by renewal exemption
Total unexempted certificates: 14
Hey there, apologies on the delay in response here. I appreciate all of your help!
Still not sure where things went wrong here in our automated jobs that was causing us to appear to be rate-limited. It hasn't impacted us since the original post date, though we've also taken some steps to put ourselves below the limits while looking into a potential rate-limit increase request.
I'll consider this "probably resolved" on my end and throw together a new post if something seems fishy again.
