Production server with lots of domains

I’ve just tried to install Let’s Encrypt on my production server which has over 100 websites on it, although this equates to 359 domains.

I selected all for the cert install and hit the authorization limit (I didn’t realise there was one!)

It doesn’t appear to have installed any certificates and I can’t run it again on 1 or a few sites.

How do I resolve this? What’s the best practice for using Let’s encrypt for this many domains?

Hi @SebTucknott,

What software are you using to request these certificates?

I used certbot from Centos 6

thank you.

Certbot should usually not produce this particular error unless it crashes. Would you be willing to post the log file from /var/log/letsencrypt somewhere?

It’s a big file 10mb

all looks normal then hits the 300th domain and get this -

HTTP 429
Server: nginx
Content-Type: application/problem+json
Content-Length: 144
Boulder-Requester: 23336945
Replay-Nonce: -36pfhbsnc9XeMmPawj-EtHpC6ooltyAfeVgqq_h6Ec
Expires: Fri, 27 Oct 2017 09:21:39 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 27 Oct 2017 09:21:39 GMT
Connection: close

{
“type”: “urn:acme:error:rateLimited”,
“detail”: “Error creating new authz :: too many currently pending authorizations”,
“status”: 429
}
2017-10-27 09:21:39,732:DEBUG:acme.client:Storing nonce: -36pfhbsnc9XeMmPawj-EtHpC6ooltyAfeVgqq_h6Ec
2017-10-27 09:21:39,733:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/certbot/main.py”, line 861, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/certbot/main.py”, line 698, in run
certname, lineage)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/certbot/main.py”, line 85, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/certbot/client.py”, line 357, in obtain_and_enroll_certificate
certr, chain, key, _ = self.obtain_certificate(domains)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/certbot/client.py”, line 318, in obtain_certificate
self.config.allow_subset_of_names)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/certbot/auth_handler.py”, line 66, in get_authorizations
self.authzr[domain] = self.acme.request_domain_challenges(domain)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/client.py”, line 212, in request_domain_challenges
typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/client.py”, line 191, in request_challenges
response = self.net.post(self.directory.new_authz, new_authz)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/client.py”, line 682, in post
return self._post_once(*args, **kwargs)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/client.py”, line 695, in _post_once
return self._check_response(response, content_type=content_type)
File “/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/client.py”, line 582, in _check_response
raise messages.Error.from_json(jobj)
Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new authz :: too many currently pending authorizations
2017-10-27 09:21:39,734:ERROR:certbot.log:An unexpected error occurred:
2017-10-27 09:21:39,734:ERROR:certbot.log:There were too many requests of a given type :: Error creating new authz :: too many currently pending authorizations

What version of Certbot are you using?

@bmw, I feel like we came across this before; do you remember what the problem might be? Wasn’t there an issue about not canceling some pending authzs when we should have?

I presume it’s the latest version, did a wget and install this morning.

From what I understand I think it requests all the keys in one step. But at I had more than the 300 limit it stopped/crashed.

I presume it could be prevented by either displaying a warning when trying to submit that many domains or complete the other steps in smaller batches?

This doesn’t make sense to me. The Let’s Encrypt CA limit is 100 domain names per certificate, not 300.

Maybe there’s a bug associated with requesting more than 100 domain names in a single certificate (which the CA won’t allow, but maybe Certbot responds badly to the refusal in this case?).

It does seem to be the case.

I just ran “certbot certonly --webroot --staging” (certbot-auto 0.19.0) with 301 names. When it hit the limit, Certbot just bombed without cleaning up.

Horror letsencrypt.log: https://mn0.us/fcFw/letsencrypt.log
Subsequent letsencrypt.log trying 1 new name: https://mn0.us/rbwA/letsencrypt.log

That sounds like a pretty significant problem!

can you include the commands and your domain names so we can double check what you are reporting is in fact accurate (i.e. SANs with 300 domains in them)

also have a look at this tool - it should clear your pending authz if your log files are still there

Andrei

I wrote up a discussion about the problem and ways to fix it in Certbot here, but the root problem is Let’s Encrypt will only let you include 100 names in a single cert. Unfortunately the UI right now is a little awkward here. You should include --domains on the command line with a comma delimited list of the domains you want in the cert, including no more than 100 domains. You could potentially use the interactive UI to select the first 100 domains, the 2nd 100 domains, etc., but unfortunately there is no guarantee the names will be in the same order on each run. After all your certs are issued, you can renew them normally with certbot renew, but the initial issuance will be a little awkward.

I hope this helps and sorry for the trouble!

1 Like

sudo /etc/httpd/certbot-auto --apache

357: www.***.com
358: *.com
359: www.
.com

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter ‘c’ to cancel):

There are 100 domains approx with 359 when you take into account subdomains.

blank to select all options shown (Enter ‘c’ to cancel): 49,60
Obtaining a new certificate
/opt/eff.org/certbot/venv/lib/python2.6/site-packages/acme/jose/jwa.py:110: DeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
signer = key.signer(self.padding, self.hash)
An unexpected error occurred:
There were too many requests of a given type :: Error creating new authz :: too many currently pending authorizations
Please see the logfiles in /var/log/letsencrypt for more details.

An example domain name in the list is bright-site.co.uk

The problem explained by @bmw accounts for what you’re experiencing, but unfortunately going down to a shorter list of domains now won’t remove the existing pending authorizations.

I think you can circumvent this particular limit by backing up and removing your existing account in /etc/letsencrypt/accounts and re-registering with the CA using a new account key.

1 Like

That worked, thank you so much!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.