Certbot "too many certificates already issued" error

[craig_1stdibs]

My domain is:
1stdibs.com

I ran this command:
certbot certonly --debug --webroot -w /var/www/redirect/public_html -d local.intranet.1stdibs.com

It produced this output:
Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new cert :: too many certificates already issued for: 1stdibs.com

My web server is (include version):
nginx/1.11.10

The operating system my web server runs on is (include version):
CentOS Linux release 7.3.1611 (Core)

My hosting provider, if applicable, is:
AWS

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no

More info:

We’re using certbot/letsencrypt for around 200 internal-use-only subdomains; I naturally ran into the 20/week limit, as they are all under intranet.1stdibs.com, but since completing that initial task, I have had no issues with renewals or extra subdomains/other domains.

On 2017-09-08, I started receiving the “too many certificates” error, so I tried adding this subdomain to a working cert renewal, and later, attempted to use the python script linked here to clean out pending authorizations; I still receive the “too many certificates” error.

Any idea what is happening here? crt.sh suggests that we aren’t generating a ridiculous number of certificates (we are using certbot, so I assume it is playing nice). I’m not sure if that pending-authorization-clearing script actually works as advertised, or if that is even really my problem. Really been enjoying using LE for our internal certs and hoping we aren’t hitting some fixed system limit!

[jared.m]

@craig_1stdibs

1stdibs.com

I count 280 in the past 4 days, that's a healthy number. While you'll be able to renew beyond this limit, it will prevent you from issuing new certificates. You should consider bundling these with SANs, or requesting a rate limit increase.

[craig_1stdibs]

OK, so I guess it doesn’t look any different on your side, but those are all renewals. I suppose renewals count against the weekly limit, which isn’t really clearly or expected; let’s say I did create 20 new certs per week, for 3 months (after which time, or really before, renewals would start to kick in, and prevent new certificates from being issued). This is an effective limit of 240 subdomains or so?

Therefore, without a rate limit adjustment, from what I can tell, assuming you have the recommended “certbot renew” cron job running before you wake up each day; you basically need to disable the cron job at some point, let the limit clear, and then renew after the new certs have been issued (as the renewals are exempt).

I’d be happy to bundle hostnames with SANs, I was starting down this path… but of course, can’t do this right now, as we can’t issue any new certs at all (I just disabled our renewal cron this morning).

We’re still quite a bit below the suggested 500 subdomain number for a rate limit adjustment, which led to my assumption that something else was wrong; I did actually fill out the rate limit adjustment form back in June or July, prior to beginning this project, and resubmitted now here. This is the correct form? Or is there somewhere else, where I can expect a similar speedy response to what I’ve received from you, Jared! (I appreciate your assistance and Let’s Encrypt, and should be able to convince my employer to kick a donation their way).

(I guess we can just not worry about any of this once wildcard certs are live on LE…)

[schoen]

@craig_1stdibs

OK, so I guess it doesn’t look any different on your side, but those are all renewals. I suppose renewals count against the weekly limit, which isn’t really clearly or expected; let’s say I did create 20 new certs per week, for 3 months (after which time, or really before, renewals would start to kick in, and prevent new certificates from being issued). This is an effective limit of 240 subdomains or so?

Therefore, without a rate limit adjustment, from what I can tell, assuming you have the recommended “certbot renew” cron job running before you wake up each day; you basically need to disable the cron job at some point, let the limit clear, and then renew after the new certs have been issued (as the renewals are exempt).

This is a common point of confusion. Your updated interpretation is right, but this was recognized as a problem or inconvenience for some users and so there was an updated rate limit that worked in the way you originally expected—but I think it ran into a different set of problems and so was rolled back for the time being to the original behavior. So, you have to do new issuances before renewals in a given week because renewals can block new issuances via the ratelimit, but new issuances can't block renewals.

We realize that this is often counterintuitive and inconvenient especially in terms of providing a reason not to perform automated renewals, and I believe there's an ongoing effort to again change the CA behavior so that the order won't matter this way in the future.

[cpu]

Hi @craig_1stdibs,

@craig_1stdibs

I did actually fill out the rate limit adjustment form back in June or July, prior to beginning this project, and resubmitted now here. This is the correct form? Or is there somewhere else

I'm reviewing your application now - that's the correct place. I only see the one submission from June 14th. I'll DM you to follow-up.