Renewals rate limited on large server with 700+ domains

Issuance Tech
1

Hi,

I am a system administrator at a web hosting company trying to debug an issue with renewals on one of our larger setups. Our server is running CentOS 6 and certbot-auto is running version 0.27.0. This particular server has well over 1000 domains hosted on it and about 700+ of them are using Let’s Encrypt certificates.

The initial setup of the certificates went without an issue, but now the certificates are about 20 days away from renewal. Each domain is unique and in their own certificate. We only combined the www. subdomain into each certificate.

The renewal ran normally for the first 300 domains, but after 300 sucessfull renewals we got rate limited. I verified that the number of sucessfull renewals is exactly 300. A search for the number 300 in the rate limiting documentation reveals this:

“You can have a maximum of 300 Pending Authorizations on your account. Hitting this rate limit is rare, and happens most often when developing ACME clients. It usually means that your client is creating authorizations and not fulfilling them.”

We are only using “certbot-auto renew” to perform the renewals. We are not doing anything fancy with the API calls to Let’s Encrypt. It’s just whatever API calls certbot-auto makes. No custom clients are being used. Also, the previous 300 domains were sucessfull, so why should they be counted as “Pending Authorizations”?

Here is the output from certbot-auto during the renewal process from when it began to fail. I’ve redacted the domain names from the output because they are of a pornagraphic nature and I didn’t want to violate any community guidelines here.

Processing /etc/letsencrypt/renewal/www.<site300_redacted>.com.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for <site300_redacted>.com
http-01 challenge for www.<site300_redacted>.com
Waiting for verification…
Cleaning up challenges

new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/www.<site300_redacted>.com/fullchain.pem

Processing /etc/letsencrypt/renewal/www.<site301_redacted>.com.conf

Cert is due for renewal, auto-renewing…
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Attempting to renew cert (www.<site301_redacted>.com) from /etc/letsencrypt/renewal/www.<site301_redacted>.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/. Skipping.

My question is why are the sucessfull renewals getting counted as pending by the rate limiters? I understand I need to wait a week for the block to expire, but I have over 450 domains awaiting renewal. If I get blocked for another week when I try to run ‘certbot-auto renew’ next week, we could be i a situation where we will have certificates expiring. I didn’t see anything related to this in the issue tracker on github. Is this a bug in certbot? Am I doing something wrong that would cause the request to remain pending even though it already succeeded and renewed the cert?

2

That's the New Orders rate limit, not the Pending Validations limit:

For users of the ACME v2 API you can create a maximum of 300 New Orders per account per 3 hours.

You just have to wait a couple hours before you can continue.

You can request a rate limit increase from Let's Encrypt, but it's not entirely necessary. If you can spread the renewals out more it won't be much of an issue.

3

It’s good to know that I will be unblocked after only a couple of hours, but this still seems like something is wrong to me. These were renewals, not new orders. Also, we were not rate limited when I performed the initial Let’s Encrypt deployment on this server. We registered 781 domains at once. Does that mean that the initial deployment was over the v1 API?

4

Hi @Rocco_Webair

300 new orders in 3 hours are 100 new orders per hour.

So I would create only 60 orders per hour, then there is enough time. 700 domains -> 12 - 13 hours.

1 Like
5

"New orders" is an ACME term, all renewals are identical to original issuance on the ACME side.

1 Like
6

Probably, yes. It might have switched the default to the ACME v2 API in Certbot 0.26.0, released July 11. Your certificates are a few days older than that, right?

1 Like
7

It might have switched the default to the ACME v2 API in Certbot 0.26.0, released July 11. Your certificates are a few days older than that, right?

Yep, about a week and a half older. That was it exactly.

I just ran certbot with the following flag to force the v1 API and it's renewing all the domains:
--server https://acme-v01.api.letsencrypt.org/directory

Is there any news about when the v1 API might get deprecated?

8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.