Renewals hitting rate limit after upgrading certbot from 0.19.0 to 0.31.0 (ACME v1 -> v2)

petertoi · July 7, 2020, 1:13pm

I manage a server with approx 9000 Let’s Encrypt SSL-enabled domains. After updating certbot from 0.19.0 to 0.31.0 last week in order to update to the acme-v2 endpoint we are experiencing issues automatically renewing our domains.

All domains are registered under the same account. Each certificate is registered with both the base domain and a www prefixed domain. All domains are unique.

I had thought that renewals were not subject to rate-limits. If indeed they are I would love to hear how others are managing similar volumes of renewals.

One such domain is: zeenarealty.com

I ran this command: certbot -q renew

It produced this output:

[snippet start]

2020-06-30 22:21:24,137:DEBUG:acme.client:Storing nonce: 0102XCyIfp3jDpTzeV6bVE21Def-cbpuF02s3mYR8nWJTHE
2020-06-30 22:21:24,138:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "value": "zeenarealty.com",\n      "type": "dns"\n    },\n    {\n      "value": "www.zeenarealty.com",\n      "type": "dns"\n    }\n  ]\n}'
2020-06-30 22:21:24,139:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJub25jZSI6ICIwMTAyWEN5SWZwM2pEcFR6ZVY2YlZFMjFEZWYtY2JwdUYwMnMzbVlSOG5XSlRIRSIsICJhbGciOiAiUlMyNTYiLCAidXJsIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1vcmRlciIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMS5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvcmVnLzI3NDY0Mzg1In0",
  "signature": "M8ktxoNL7-urMdOgXdku_5V5Fyasqozkd-Iv4APvYq3pWIIKluCMMyShtM6ldyt3DUO_Vc5uizr8xTJqaiEuUTEW_5b9UH_PfvqE5AkOBkl-QSRXsSlKx456Ko77kJ9dFyLdrjDpuK7Frim-BsF3G2w_yVsFpgvjAYLreFBKSH7QOyQOE9FPzzwOiYYmOHXhjmae3z0DTVripfNS00bcOUfgDeq0gomK4UT-jBIMKK4_xnN0S-20Ax3eMgwjnDGnlVD9q8YQTUS7g2V8--iUPbzeKAUwFpKasz8NlSgcK_SZlO9uJR4xdjc_qX9ZCKNJa7gy9XyRib-GRJ0TABpYQw",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInZhbHVlIjogInplZW5hcmVhbHR5LmNvbSIsCiAgICAgICJ0eXBlIjogImRucyIKICAgIH0sCiAgICB7CiAgICAgICJ2YWx1ZSI6ICJ3d3cuemVlbmFyZWFsdHkuY29tIiwKICAgICAgInR5cGUiOiAiZG5zIgogICAgfQogIF0KfQ"
}
2020-06-30 22:21:24,230:DEBUG:requests.packages.urllib3.connectionpool:"POST /acme/new-order HTTP/1.1" 429 190
2020-06-30 22:21:24,231:DEBUG:acme.client:Received response:
HTTP 429
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Content-Type: application/problem+json
Replay-Nonce: 0102LuUj7aB-AADgc3C2S1XJFaTVC-iz0x59Ka7X1lMOmFY
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Boulder-Requester: 27464385
Date: Tue, 30 Jun 2020 22:21:24 GMT
Content-Length: 190
Server: nginx

{
  "type": "urn:ietf:params:acme:error:rateLimited",
  "detail": "Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/",
  "status": 429
}
2020-06-30 22:21:24,231:WARNING:certbot.renewal:Attempting to renew cert (zeenarealty.com) from /home/ixact/www/core/letsencrypt/certs/renewal/zeenarealty.com.conf produced an unexpected error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/. Skipping.
2020-06-30 22:21:24,232:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 452, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1193, in renew_cert
    renewed_lineage = _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 310, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 385, in _get_order_and_authorizations
    orderr = self.acme.new_order(csr_pem)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 889, in new_order
    return self.client.new_order(csr_pem)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 672, in new_order
    response = self._post(self.directory['newOrder'], order)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 96, in _post
    return self.net.post(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1204, in post
    return self._post_once(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1218, in _post_once
    response = self._check_response(response, content_type=content_type)
  File "/usr/lib/python3/dist-packages/acme/client.py", line 1073, in _check_response
    raise messages.Error.from_json(jobj)
acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/

[snippet end]

My web server is (include version): nginx (N/A)

The operating system my web server runs on is (include version): Ubuntu 16.04.6 LTS

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

JuergenAuer · July 7, 2020, 1:27pm

Hi @petertoi

read that link. The ACME-v2 has a 300 orders / 3 hours limit.

--> 100 per hour --> 2400 per day --> some days, 9000 domains are renewed.

petertoi · July 7, 2020, 2:19pm

Thanks for the quick response, @JuergenAuer .

I suppose I was a little confused that the endpoint is called /new-order yet was being used for a renewal. It’s good to have confirmation that things are indeed working the way they should be.

The same article states that we need to wait a week for the rate limit to expire. That leads to back to my earlier question about how others are managing their batches. What’s curious is that I do see some certificates are renewing on each call to certbot -q renew so I’m a little confused if the rate limit penalty is being imposed. Can you provide any insights here?

Many thanks,
Peter

JuergenAuer · July 7, 2020, 2:34pm

No, that's only a three-hour - limit. The one-week is per domain, but if you don't create 5 identical certificates per domain, you will never hit that limit.

PS:

A renewal creates always a new certificate, so /new-order is used.

Switching from v1 -> v2 with "only 9000 domains" you shouldn't have a problem.

Only thing: Max 100 new orders per hour. But if you start your renews early enough (30 days before expiration), you can renew 1200 certificates per day without hitting any limit -> 1200 * 20 = 24000 domains with 10 additional days.

petertoi · July 7, 2020, 2:53pm

Thanks again @JuergenAuer,

To wrap all this up, can I increase the frequency that certbot.timer is running on my system from every 12 hours to every 3-4 hours and be done with it? In this case, all the certificates will be checked against the endpoint on each run. Or do I need to set up my own batch processing routine that will attempt 100 certificates per hour?

If I need to run small batches, do you know of any community-supported packages or scripts that are ready-built and well-tested that I could take advantage of rather than rolling my own? I’m sure I’m not the only person in this situation.

All the best,
Peter

JuergenAuer · July 7, 2020, 3:44pm

You can run Certbot every hour. If no certificate needs a renew, nothing happens (only a little bit CPU consuming on your system).

That's configuration specific, find a working solution, that's all. It's a question of your certificate renews: 9000 with the same date / time -> you will hit the limit. Max. 50 per hour -> you will never hit a limit.

jvanasco · July 7, 2020, 4:41pm

I think this could be handled by certbot with a few tweaks, and paying careful attention to the ratelimit details – some are by Account, Domain, IP Range.

I would do the following:

utilize multiple accounts for renewals and enrollment. cap each account at 900 or so certificates.
run certbot several times a day, offset to 1.5x-2x the ratetlimit expiry that you have issues with. staggering at 1x the ratelimit may not work, because with a large number of orders, you may have tripped the limit minutes/hours into the last invocation of the script
try to stagger renewals so you don't have a stampede. try not to exceed 80% of any given ratelimit in your planning, because failures will be reattempted on the next invocation and that will trigger rate-limits.

I developed an open-sourced a custom client / framework to handle use-cases like yours, but the ACME-v2 port is not done yet. (Well, the ACME-v2 protocol is finished, but automatic renewals are not done yet). That project is: GitHub - aptise/peter_sslers: or how i stopped worrying and learned to love the ssl certificate

system · August 6, 2020, 4:41pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.