Letsencryptauto - dealing with pending Authorizations

[Jeromnimo]

Hi,

I am trying to renew my certificates, but I constantly get 429 error

I am not using any cron to do this, I am used to do it manually when I am notified about close end of life of my current certifcates.

Today they are expired, and I am not able to renew them

I have read several post on the forum on this subject, but didn’t find a suitable answer. I don’t know how to check how many challenges are in the pipe for my account, and it should be very low (as I am not regularly renewing them).

I am using this command to renew my certificates, with no issue until now :
(this is an example, I have 4 domains on my server, and biggest subdomain list is 17)

/opt/letsencrypt/letsencrypt-auto certonly --standalone --renew-by-default -d bausse.fr -d www.bausse.fr -d mail.bausse.fr -d imap.bausse.fr -d smtp.bausse.fr -d mysql.bausse.fr

I have already waited for more than a week before retrying.

Let me know if you needs logs or anything else, help would be greatly appreciated.

Thanks.

[schoen]

Hi @Jeromnimo,

Is there some way that you share a server or server image with a large number of other people, or that some part of your infrastructure hosts sites for other people? (Do you have users who are somehow allowed to use your infrastructure to trigger issuance of their own certificates?) How many different certificates do you have on your machine overall?

Is there some reason that the certificate issuance or renewal process could sometimes be interrupted or crash? Is there any evidence that this has been happening frequently? Do you ever use a manual verification process with certbot --manual or a web-based client or anything like that?

[Jeromnimo]

Hi @schoen,

thanks for your answer.

I’m the only one adminstrating the server, and I only have 4 certificates (for 4 domains) overall. No one except form me ever launched this command.

I am launching certifcate renewal manually, and never interrupted it myself.
I don’t get your point about “Is there any evidence that this has been happening frequently?”, what do you mean ?

I have never tried certbot --manual or a web based client, but I can try if it makes sense ?

[mnordhoff]

I’m the only one adminstrating the server, and I only have 4 certificates (for 4 domains) overall. No one except form me ever launched this command.

Is the account key from the /etc/letsencrypt directory shared with some other server, or some other client?

I don’t get your point about “Is there any evidence that this has been happening frequently?”, what do you mean ?

Evidence that the certificate issuance or renewal process has been getting interrupted or crashing frequently. For example, is there a Certbot cron job that’s failing frequently?

To quote the rate limit documentation:

You can have a maximum of 300 Pending Authorizations on your account. Hitting this rate limit is rare, and happens most often when developing ACME clients. It usually means that your client is creating authorizations and not fulfilling them.

You shouldn’t ever hit the pending authorizations rate limit. You should have, at most, 1 pending authorization per hostname in a certificate, for the seconds Certbot is running. By the time Certbot exits, every authorization it just created should be either valid or invalid, not pending.

So, the question is how this situation is happening. Usually, it happens when a buggy client fails to clean up after itself, or when someone is using the same account key to issue multiple certificates covering hundreds of hostnames in parallel. But Certbot isn’t known to be buggy, and you’re not doing that, right?

I have never tried certbot --manual or a web based client, but I can try if it makes sense ?

No, you don’t need to. They only asked because (mis)using clients like that could contribute to this issue.

[schoen]

Another question could be whether certbot renew --force-renew or something similar is being run from cron. (and then crashing frequently?)

[Jeromnimo]

Thanks for your feedbacks @schoen & @mnordhoff.

After grabing my logs today, discovering letsencrypt logs were updated today (while I was at work, and not doing anything on my server), it seems there is a auto renew retry done by certbot… need to figure out how to deactivate that now

2017-05-30 11:48:53,857:DEBUG:parsedatetime:_buildTime: [30 ][][days]
2017-05-30 11:48:53,858:DEBUG:parsedatetime:units days --> realunit days
2017-05-30 11:48:53,858:DEBUG:parsedatetime:return
2017-05-30 11:48:53,858:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2017-05-26 19:33:00 UTC.
2017-05-30 11:48:53,858:INFO:certbot.renewal:Cert is due for renewal, auto-renewing…
2017-05-30 11:48:53,860:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2017-05-30 11:48:53,869:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Automatically use a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f34da17dd90>
Prep: True

[schoen]

Hi @Jeromnimo,

Cool, I’m glad you identified that. It is probably running from crontab or, depending on how you installed Certbot, from an /etc/cron.d script a or systemd timer.

A subsequent thing to figure out will be why the renewal is failing/crashing, which maybe other logs can show (or maybe you can figure it out by looking at the output of certbot renew run from the command line).

[Jeromnimo]

Thanks @schoen, you were right, I have commented script in /etc/cron.d/certbot file

Here is a full log of a fresh try of a renew, if it can helps.

2017-05-30 20:46:22,966:DEBUG:certbot.main:certbot version: 0.14.2
2017-05-30 20:46:22,966:DEBUG:certbot.main:Arguments: [’–standalone’, ‘–renew-by-default’, ‘-d’, ‘bausse.fr’, ‘-d’, ‘www.bausse.fr’, ‘-d’, ‘mail.bausse.fr’, ‘-d’, ‘smtp.bausse.fr’, ‘-d’, ‘imap.bausse.fr’, ‘-d’, ‘mysql.bausse.fr’]
2017-05-30 20:46:22,966:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2017-05-30 20:46:22,995:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer <certbot.cli._Default object at 0x7ff5c25f9a90>
2017-05-30 20:46:22,995:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x7ff5c2612750>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x7ff5c2612510>, apache=<certbot.cli._Default object at 0x7ff5c25f9150>, apache_challenge_location=<certbot.cli._Default object at 0x7ff5c2618d10>, apache_ctl=<certbot.cli._Default object at 0x7ff5c261d150>, apache_dismod=<certbot.cli._Default object at 0x7ff5c2618750>, apache_enmod=<certbot.cli._Default object at 0x7ff5c2618650>, apache_handle_modules=<certbot.cli._Default object at 0x7ff5c2618e90>, apache_handle_sites=<certbot.cli._Default object at 0x7ff5c261d050>, apache_init_script=<certbot.cli._Default object at 0x7ff5c261d250>, apache_le_vhost_ext=<certbot.cli._Default object at 0x7ff5c2618890>, apache_logs_root=<certbot.cli._Default object at 0x7ff5c2618bd0>, apache_server_root=<certbot.cli._Default object at 0x7ff5c26189d0>, apache_vhost_root=<certbot.cli._Default object at 0x7ff5c2618ad0>, authenticator=‘standalone’, break_my_certs=<certbot.cli._Default object at 0x7ff5c2614390>, cert_path=<certbot.cli._Default object at 0x7ff5c2612ad0>, certname=<certbot.cli._Default object at 0x7ff5c25f9910>, chain_path=<certbot.cli._Default object at 0x7ff5c2612490>, checkpoints=<certbot.cli._Default object at 0x7ff5c2614510>, config_dir=<certbot.cli._Default object at 0x7ff5c2612290>, config_file=None, configurator=<certbot.cli._Default object at 0x7ff5c25f9a90>, csr=<certbot.cli._Default object at 0x7ff5c2614a50>, debug=<certbot.cli._Default object at 0x7ff5c2612e50>, debug_challenges=<certbot.cli._Default object at 0x7ff5c2612f50>, dialog=None, domains=‘mysql.bausse.fr’, dry_run=<certbot.cli._Default object at 0x7ff5c25f9a10>, duplicate=<certbot.cli._Default object at 0x7ff5c2612850>, eff_email=<certbot.cli._Default object at 0x7ff5c25f9e10>, email=<certbot.cli._Default object at 0x7ff5c25f9d10>, expand=<certbot.cli._Default object at 0x7ff5c2612150>, force_interactive=<certbot.cli._Default object at 0x7ff5c25f9710>, fullchain_path=<certbot.cli._Default object at 0x7ff5c26126d0>, func=<function certonly at 0x7ff5c284f938>, hsts=<certbot.cli._Default object at 0x7ff5c2614890>, http01_port=<certbot.cli._Default object at 0x7ff5c2614290>, ifaces=<certbot.cli._Default object at 0x7ff5c2612ed0>, init=<certbot.cli._Default object at 0x7ff5c2614310>, installer=<certbot.cli._Default object at 0x7ff5c25f9a90>, key_path=<certbot.cli._Default object at 0x7ff5c26128d0>, logs_dir=<certbot.cli._Default object at 0x7ff5c25f9e90>, manual=<certbot.cli._Default object at 0x7ff5c2618250>, manual_auth_hook=<certbot.cli._Default object at 0x7ff5c2618610>, manual_cleanup_hook=<certbot.cli._Default object at 0x7ff5c261d490>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x7ff5c261d590>, must_staple=<certbot.cli._Default object at 0x7ff5c2614590>, nginx=<certbot.cli._Default object at 0x7ff5c2661d10>, nginx_ctl=<certbot.cli._Default object at 0x7ff5c261d7d0>, nginx_server_root=<certbot.cli._Default object at 0x7ff5c261d350>, no_bootstrap=<certbot.cli._Default object at 0x7ff5c2612b50>, no_self_upgrade=<certbot.cli._Default object at 0x7ff5c2612a50>, no_verify_ssl=<certbot.cli._Default object at 0x7ff5c2614090>, noninteractive_mode=<certbot.cli._Default object at 0x7ff5c25f92d0>, num=<certbot.cli._Default object at 0x7ff5c2614d10>, os_packages_only=<certbot.cli._Default object at 0x7ff5c2612950>, post_hook=<certbot.cli._Default object at 0x7ff5c26181d0>, pre_hook=<certbot.cli._Default object at 0x7ff5c26180d0>, pref_challs=<certbot.cli._Default object at 0x7ff5c2614f90>, prepare=<certbot.cli._Default object at 0x7ff5c2614110>, quiet=<certbot.cli._Default object at 0x7ff5c2612c50>, reason=<certbot.cli._Default object at 0x7ff5c2614710>, redirect=<certbot.cli._Default object at 0x7ff5c2614690>, register_unsafely_without_email=<certbot.cli._Default object at 0x7ff5c25f9b10>, reinstall=<certbot.cli._Default object at 0x7ff5c2612050>, renew_by_default=True, renew_hook=<certbot.cli._Default object at 0x7ff5c26182d0>, renew_with_new_domains=<certbot.cli._Default object at 0x7ff5c2612410>, rsa_key_size=<certbot.cli._Default object at 0x7ff5c2614490>, server=<certbot.cli._Default object at 0x7ff5c25f9c90>, staging=<certbot.cli._Default object at 0x7ff5c2612d50>, standalone=True, standalone_supported_challenges=<certbot.cli._Default object at 0x7ff5c261d8d0>, staple=<certbot.cli._Default object at 0x7ff5c2614c90>, strict_permissions=<certbot.cli._Default object at 0x7ff5c2614e90>, text_mode=<certbot.cli._Default object at 0x7ff5c25f91d0>, tls_sni_01_port=<certbot.cli._Default object at 0x7ff5c2614190>, tos=<certbot.cli._Default object at 0x7ff5c2612650>, uir=<certbot.cli._Default object at 0x7ff5c2614a90>, update_registration=<certbot.cli._Default object at 0x7ff5c25f9c10>, user_agent=<certbot.cli._Default object at 0x7ff5c2614b10>, validate_hooks=<certbot.cli._Default object at 0x7ff5c26183d0>, verb=‘certonly’, verbose_count=<certbot.cli._Default object at 0x7ff5c2661e90>, webroot=<certbot.cli._Default object at 0x7ff5c2618050>, webroot_map=<certbot.cli._Default object at 0x7ff5c261dad0>, webroot_path=<certbot.cli._Default object at 0x7ff5c261d6d0>, work_dir=<certbot.cli._Default object at 0x7ff5c26120d0>)
2017-05-30 20:46:23,002:DEBUG:certbot.log:Root logging level set at 20
2017-05-30 20:46:23,002:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-05-30 20:46:23,003:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2017-05-30 20:46:23,065:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7ff5c265f310>
Prep: True
2017-05-30 20:46:23,066:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7ff5c265f310> and installer None
2017-05-30 20:46:23,068:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, contact=(u’mailto:jerome@bausse.fr’,), agreement=u’https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf’, key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7ff5c4f3b7d0>)>)), uri=u’https://acme-v01.api.letsencrypt.org/acme/reg/724420’, new_authzr_uri=u’https://acme-v01.api.letsencrypt.org/acme/new-authz’, terms_of_service=u’https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf’), 9b903cc2c1731e562ac43b493a5e353f, Meta(creation_host=u’ns3002977.ip-151-80-42.eu’, creation_dt=datetime.datetime(2016, 3, 5, 21, 8, 16, tzinfo=)))>
2017-05-30 20:46:23,069:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2017-05-30 20:46:23,071:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-05-30 20:46:23,461:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “GET /directory HTTP/1.1” 200 352
2017-05-30 20:46:23,462:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 352
Boulder-Request-Id: amDRNz_UvRNAqipeHu-0iQBdBI5lFZzcG0hfnPc_K4I
Replay-Nonce: 9u_VEtL3G3SrWoic2QCSVXf0UKWJvWIAAZJM5SGkqkI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 30 May 2017 20:46:28 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 30 May 2017 20:46:28 GMT
Connection: keep-alive

{
“key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,
“new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,
“new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,
“new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,
“revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert”
}
2017-05-30 20:46:23,470:DEBUG:certbot.renewal:Auto-renewal forced with --force-renewal…
2017-05-30 20:46:23,471:INFO:certbot.main:Renewing an existing certificate
2017-05-30 20:46:23,471:DEBUG:acme.client:Requesting fresh nonce
2017-05-30 20:46:23,471:DEBUG:acme.client:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz.
2017-05-30 20:46:23,656:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “HEAD /acme/new-authz HTTP/1.1” 405 0
2017-05-30 20:46:23,657:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 91
Allow: POST
Boulder-Request-Id: 7tO9OtwTUlotnghR1j3vyHMrEIb-mJ1NUC755iJ9czk
Replay-Nonce: V8rKmY3NSYJMg7qBlR26adV9TvRRxJAIN3tm8_D5OBA
Expires: Tue, 30 May 2017 20:46:28 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 30 May 2017 20:46:28 GMT
Connection: keep-alive

2017-05-30 20:46:23,657:DEBUG:acme.client:Storing nonce: V8rKmY3NSYJMg7qBlR26adV9TvRRxJAIN3tm8_D5OBA
2017-05-30 20:46:23,658:DEBUG:acme.client:JWS payload:
{
“identifier”: {
“type”: “dns”,
“value”: “bausse.fr”
},
“resource”: “new-authz”
}
2017-05-30 20:46:23,663:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
“header”: {
“alg”: “RS256”,
“jwk”: {
“e”: “AQAB”,
“kty”: “RSA”,
“n”: “qVyImOsTcj8mvlld6AJyzw1lZyXYcD3VpObND218ee7dwoJ3Grwk-FeSDI-yx-JzGlcutNyUxDzbA4hiiU-cuF2FVCqo1o9PC4hDymcJwVPrHr2UKtFO_VtzkoUP0itYn3G85hBwtx_pEC_QP10JCr5N3l_0Puu8aCf3AZuAxG0WwuLx9nWCKU-hrlTgxHEuMcWB4B1R2sPP7IP_9uNbfQTeHyvUvSo4RuTN-GCbjxUs8_OJXPvhsc5cgYyYDDOPMnEy6AnGrsC-g1WKYkfPXhkx-xe8rsTn-LcRymGClXlmn35Mg21XalaESADDNT_fkonwM5guXOXtC4ydL1NwMw”
}
},
“protected”: “eyJub25jZSI6ICJWOHJLbVkzTlNZSk1nN3FCbFIyNmFkVjlUdlJSeEpBSU4zdG04X0Q1T0JBIn0”,
“payload”: “ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAiYmF1c3NlLmZyIgogIH0sIAogICJyZXNvdXJjZSI6ICJuZXctYXV0aHoiCn0”,
“signature”: “GzigtDq6Xic1SiqUveCbHM0pPO4fq4fs1cVyV_jQRiwHHnS2S_g3ByQv7DGFkzdCrDJEF5mFD9W2FVYMXqw-HpwAmSddSEMolWzvg13iLtrYndebRp95LWcFT3nA0Y54kb9icR9DX80E1g7fAVf6nzQL8kwMmVoqQwbYjTN9YzqwPg25NbM1-9NOSux1dgoQ6tHHF0nzZNlOwZ0Iagem39zjOYU8pmvMEhtS8AcL-b8Vf2ce1SRIOIEp_wGZFqGrG82RMge2fgO91fPsXqjX5wldx6TsfKETSLPnnmok2haktDLOJ0877MkKn7K7He_mHnongP3ySn_J9TIO1dEqtA”
}
2017-05-30 20:46:23,992:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “POST /acme/new-authz HTTP/1.1” 429 144
2017-05-30 20:46:23,993:DEBUG:acme.client:Received response:
HTTP 429
Server: nginx
Content-Type: application/problem+json
Content-Length: 144
Boulder-Request-Id: m0xQ_WhtWaf54PwqJhFbD15tcjJujpFa18t-Wbn7HRM
Boulder-Requester: 724420
Replay-Nonce: ZSmuqUIjaAUP8bCGeMXkk-qOs6w5ASpXj9uyl-36Zlw
Expires: Tue, 30 May 2017 20:46:28 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 30 May 2017 20:46:28 GMT
Connection: close

{
“type”: “urn:acme:error:rateLimited”,
“detail”: “Error creating new authz :: too many currently pending authorizations”,
“status”: 429
}
2017-05-30 20:46:23,993:DEBUG:acme.client:Storing nonce: ZSmuqUIjaAUP8bCGeMXkk-qOs6w5ASpXj9uyl-36Zlw
2017-05-30 20:46:23,993:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/root/.local/share/letsencrypt/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 742, in main
return config.func(config, plugins)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 682, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 77, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/renewal.py”, line 296, in renew_cert
new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/client.py”, line 313, in obtain_certificate
self.config.allow_subset_of_names)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 66, in get_authorizations
self.authzr[domain] = self.acme.request_domain_challenges(domain)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 212, in request_domain_challenges
typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 191, in request_challenges
response = self.net.post(self.directory.new_authz, new_authz)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 674, in post
return self._post_once(*args, **kwargs)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 687, in _post_once
return self._check_response(response, content_type=content_type)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 574, in _check_response
raise messages.Error.from_json(jobj)
Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new authz :: too many currently pending authorizations

[schoen]

Heh, that’s still the rate limit that you originally encountered.

If you disable the auto-renewal process and wait for one hour and then try again (even re-enabling it), you should get a different error message, which will be the true underlying error which is making this renewal process fail. If you can post that, we can try to understand why it’s happening.

[mnordhoff]

If you do “ls -lrt /var/log/letsencrypt”, are there files that aren’t within a few bytes of the size of the one you pasted above? Do any recent ones have a different error message?

[Jeromnimo]

Here is the result after 1 hour wait… still the same as far as I see

2017-05-30 22:03:33,197:DEBUG:certbot.main:certbot version: 0.14.2
2017-05-30 22:03:33,198:DEBUG:certbot.main:Arguments: [’–standalone’, ‘–renew-by-default’, ‘-d’, ‘bausse.fr’, ‘-d’, ‘www.bausse.fr’, ‘-d’, ‘mail.bausse.fr’, ‘-d’, ‘smtp.bausse.fr’, ‘-d’, ‘imap.bausse.fr’, ‘-d’, ‘mysql.bausse.fr’]
2017-05-30 22:03:33,198:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2017-05-30 22:03:33,225:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer <certbot.cli._Default object at 0x7fb458efaa90>
2017-05-30 22:03:33,225:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x7fb458f13750>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x7fb458f13510>, apache=<certbot.cli._Default object at 0x7fb458efa150>, apache_challenge_location=<certbot.cli._Default object at 0x7fb458f19d10>, apache_ctl=<certbot.cli._Default object at 0x7fb458f1e150>, apache_dismod=<certbot.cli._Default object at 0x7fb458f19750>, apache_enmod=<certbot.cli._Default object at 0x7fb458f19650>, apache_handle_modules=<certbot.cli._Default object at 0x7fb458f19e90>, apache_handle_sites=<certbot.cli._Default object at 0x7fb458f1e050>, apache_init_script=<certbot.cli._Default object at 0x7fb458f1e250>, apache_le_vhost_ext=<certbot.cli._Default object at 0x7fb458f19890>, apache_logs_root=<certbot.cli._Default object at 0x7fb458f19bd0>, apache_server_root=<certbot.cli._Default object at 0x7fb458f199d0>, apache_vhost_root=<certbot.cli._Default object at 0x7fb458f19ad0>, authenticator=‘standalone’, break_my_certs=<certbot.cli._Default object at 0x7fb458f15390>, cert_path=<certbot.cli._Default object at 0x7fb458f13ad0>, certname=<certbot.cli._Default object at 0x7fb458efa910>, chain_path=<certbot.cli._Default object at 0x7fb458f13490>, checkpoints=<certbot.cli._Default object at 0x7fb458f15510>, config_dir=<certbot.cli._Default object at 0x7fb458f13290>, config_file=None, configurator=<certbot.cli._Default object at 0x7fb458efaa90>, csr=<certbot.cli._Default object at 0x7fb458f15a50>, debug=<certbot.cli._Default object at 0x7fb458f13e50>, debug_challenges=<certbot.cli._Default object at 0x7fb458f13f50>, dialog=None, domains=‘mysql.bausse.fr’, dry_run=<certbot.cli._Default object at 0x7fb458efaa10>, duplicate=<certbot.cli._Default object at 0x7fb458f13850>, eff_email=<certbot.cli._Default object at 0x7fb458efae10>, email=<certbot.cli._Default object at 0x7fb458efad10>, expand=<certbot.cli._Default object at 0x7fb458f13150>, force_interactive=<certbot.cli._Default object at 0x7fb458efa710>, fullchain_path=<certbot.cli._Default object at 0x7fb458f136d0>, func=<function certonly at 0x7fb459150938>, hsts=<certbot.cli._Default object at 0x7fb458f15890>, http01_port=<certbot.cli._Default object at 0x7fb458f15290>, ifaces=<certbot.cli._Default object at 0x7fb458f13ed0>, init=<certbot.cli._Default object at 0x7fb458f15310>, installer=<certbot.cli._Default object at 0x7fb458efaa90>, key_path=<certbot.cli._Default object at 0x7fb458f138d0>, logs_dir=<certbot.cli._Default object at 0x7fb458efae90>, manual=<certbot.cli._Default object at 0x7fb458f19250>, manual_auth_hook=<certbot.cli._Default object at 0x7fb458f19610>, manual_cleanup_hook=<certbot.cli._Default object at 0x7fb458f1e490>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x7fb458f1e590>, must_staple=<certbot.cli._Default object at 0x7fb458f15590>, nginx=<certbot.cli._Default object at 0x7fb458f62d10>, nginx_ctl=<certbot.cli._Default object at 0x7fb458f1e7d0>, nginx_server_root=<certbot.cli._Default object at 0x7fb458f1e350>, no_bootstrap=<certbot.cli._Default object at 0x7fb458f13b50>, no_self_upgrade=<certbot.cli._Default object at 0x7fb458f13a50>, no_verify_ssl=<certbot.cli._Default object at 0x7fb458f15090>, noninteractive_mode=<certbot.cli._Default object at 0x7fb458efa2d0>, num=<certbot.cli._Default object at 0x7fb458f15d10>, os_packages_only=<certbot.cli._Default object at 0x7fb458f13950>, post_hook=<certbot.cli._Default object at 0x7fb458f191d0>, pre_hook=<certbot.cli._Default object at 0x7fb458f190d0>, pref_challs=<certbot.cli._Default object at 0x7fb458f15f90>, prepare=<certbot.cli._Default object at 0x7fb458f15110>, quiet=<certbot.cli._Default object at 0x7fb458f13c50>, reason=<certbot.cli._Default object at 0x7fb458f15710>, redirect=<certbot.cli._Default object at 0x7fb458f15690>, register_unsafely_without_email=<certbot.cli._Default object at 0x7fb458efab10>, reinstall=<certbot.cli._Default object at 0x7fb458f13050>, renew_by_default=True, renew_hook=<certbot.cli._Default object at 0x7fb458f192d0>, renew_with_new_domains=<certbot.cli._Default object at 0x7fb458f13410>, rsa_key_size=<certbot.cli._Default object at 0x7fb458f15490>, server=<certbot.cli._Default object at 0x7fb458efac90>, staging=<certbot.cli._Default object at 0x7fb458f13d50>, standalone=True, standalone_supported_challenges=<certbot.cli._Default object at 0x7fb458f1e8d0>, staple=<certbot.cli._Default object at 0x7fb458f15c90>, strict_permissions=<certbot.cli._Default object at 0x7fb458f15e90>, text_mode=<certbot.cli._Default object at 0x7fb458efa1d0>, tls_sni_01_port=<certbot.cli._Default object at 0x7fb458f15190>, tos=<certbot.cli._Default object at 0x7fb458f13650>, uir=<certbot.cli._Default object at 0x7fb458f15a90>, update_registration=<certbot.cli._Default object at 0x7fb458efac10>, user_agent=<certbot.cli._Default object at 0x7fb458f15b10>, validate_hooks=<certbot.cli._Default object at 0x7fb458f193d0>, verb=‘certonly’, verbose_count=<certbot.cli._Default object at 0x7fb458f62e90>, webroot=<certbot.cli._Default object at 0x7fb458f19050>, webroot_map=<certbot.cli._Default object at 0x7fb458f1ead0>, webroot_path=<certbot.cli._Default object at 0x7fb458f1e6d0>, work_dir=<certbot.cli._Default object at 0x7fb458f130d0>)
2017-05-30 22:03:33,232:DEBUG:certbot.log:Root logging level set at 20
2017-05-30 22:03:33,232:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2017-05-30 22:03:33,233:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer None
2017-05-30 22:03:33,351:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7fb458f60310>
Prep: True
2017-05-30 22:03:33,351:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7fb458f60310> and installer None
2017-05-30 22:03:33,354:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, contact=(u’mailto:jerome@bausse.fr’,), agreement=u’https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf’, key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7fb45b83c7d0>)>)), uri=u’https://acme-v01.api.letsencrypt.org/acme/reg/724420’, new_authzr_uri=u’https://acme-v01.api.letsencrypt.org/acme/new-authz’, terms_of_service=u’https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf’), 9b903cc2c1731e562ac43b493a5e353f, Meta(creation_host=u’ns3002977.ip-151-80-42.eu’, creation_dt=datetime.datetime(2016, 3, 5, 21, 8, 16, tzinfo=)))>
2017-05-30 22:03:33,355:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2017-05-30 22:03:33,357:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2017-05-30 22:03:33,631:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “GET /directory HTTP/1.1” 200 352
2017-05-30 22:03:33,632:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 352
Boulder-Request-Id: KLs5gTKOqRgchXAjAhrJ_Ec9ApgIK8cAsN28JK0HKG4
Replay-Nonce: 9VxJ0cd00BwlmmP7plmQHKgujLKYpyrkJOcW5VEVeII
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 30 May 2017 22:03:38 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 30 May 2017 22:03:38 GMT
Connection: keep-alive

{
“key-change”: “https://acme-v01.api.letsencrypt.org/acme/key-change”,
“new-authz”: “https://acme-v01.api.letsencrypt.org/acme/new-authz”,
“new-cert”: “https://acme-v01.api.letsencrypt.org/acme/new-cert”,
“new-reg”: “https://acme-v01.api.letsencrypt.org/acme/new-reg”,
“revoke-cert”: “https://acme-v01.api.letsencrypt.org/acme/revoke-cert”
}
2017-05-30 22:03:33,641:DEBUG:certbot.renewal:Auto-renewal forced with --force-renewal…
2017-05-30 22:03:33,641:INFO:certbot.main:Renewing an existing certificate
2017-05-30 22:03:33,641:DEBUG:acme.client:Requesting fresh nonce
2017-05-30 22:03:33,641:DEBUG:acme.client:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz.
2017-05-30 22:03:33,828:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “HEAD /acme/new-authz HTTP/1.1” 405 0
2017-05-30 22:03:33,829:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 91
Allow: POST
Boulder-Request-Id: h2YET0D5HFXt-wjEKLZqKTFONSHpg1qgEzWcezxSeEY
Replay-Nonce: dGJe4kj2iaxA6juCRrovh3sjNep1g9wAiAMUKXJEbeo
Expires: Tue, 30 May 2017 22:03:38 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 30 May 2017 22:03:38 GMT
Connection: keep-alive

2017-05-30 22:03:33,829:DEBUG:acme.client:Storing nonce: dGJe4kj2iaxA6juCRrovh3sjNep1g9wAiAMUKXJEbeo
2017-05-30 22:03:33,829:DEBUG:acme.client:JWS payload:
{
“identifier”: {
“type”: “dns”,
“value”: “bausse.fr”
},
“resource”: “new-authz”
}
2017-05-30 22:03:33,835:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
{
“header”: {
“alg”: “RS256”,
“jwk”: {
“e”: “AQAB”,
“kty”: “RSA”,
“n”: “qVyImOsTcj8mvlld6AJyzw1lZyXYcD3VpObND218ee7dwoJ3Grwk-FeSDI-yx-JzGlcutNyUxDzbA4hiiU-cuF2FVCqo1o9PC4hDymcJwVPrHr2UKtFO_VtzkoUP0itYn3G85hBwtx_pEC_QP10JCr5N3l_0Puu8aCf3AZuAxG0WwuLx9nWCKU-hrlTgxHEuMcWB4B1R2sPP7IP_9uNbfQTeHyvUvSo4RuTN-GCbjxUs8_OJXPvhsc5cgYyYDDOPMnEy6AnGrsC-g1WKYkfPXhkx-xe8rsTn-LcRymGClXlmn35Mg21XalaESADDNT_fkonwM5guXOXtC4ydL1NwMw”
}
},
“protected”: “eyJub25jZSI6ICJkR0plNGtqMmlheEE2anVDUnJvdmgzc2pOZXAxZzl3QWlBTVVLWEpFYmVvIn0”,
“payload”: “ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAiYmF1c3NlLmZyIgogIH0sIAogICJyZXNvdXJjZSI6ICJuZXctYXV0aHoiCn0”,
“signature”: “F31O6o6lzz6N-F3AqXfccUWKcSFw5rRSDSPO3QJaRC7gSLVDWTfnNsTvM16aIszBk2jSxvrxrZqkvB2yjVFH33wCpHkP0InvdcMginffSPyF1HRQ-qchvEdspRHWnkrJH-W1Rckb4iMExp1N3pI1Z1KpytPOiA371g2CIrmORM4JPEk8bfe7suVv14boaYJQJs5qCrnbQopF3DlHOMtcImiElMr4eRHUN9ZdabexglPEkM4GuJjqLjFyWh6OWIwS2yiCxxLKY72GQus_bwDyfS2pycavs7ieetyUDqg2iFuazjgiFR8vYQ2UHQEzl59Qk-ndL9NmIGZ8CuDmmMAstw”
}
2017-05-30 22:03:34,270:DEBUG:requests.packages.urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 “POST /acme/new-authz HTTP/1.1” 429 144
2017-05-30 22:03:34,271:DEBUG:acme.client:Received response:
HTTP 429
Server: nginx
Content-Type: application/problem+json
Content-Length: 144
Boulder-Request-Id: JPu7cFVTOXsJG29cgOqv_XQgoWSE8lrobCGH4kEqYP4
Boulder-Requester: 724420
Replay-Nonce: pTX57z5om2X2MpHaQoAxHL23R2AbrWl-mi9Ddl3s0go
Expires: Tue, 30 May 2017 22:03:39 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 30 May 2017 22:03:39 GMT
Connection: close

{
“type”: “urn:acme:error:rateLimited”,
“detail”: “Error creating new authz :: too many currently pending authorizations”,
“status”: 429
}
2017-05-30 22:03:34,271:DEBUG:acme.client:Storing nonce: pTX57z5om2X2MpHaQoAxHL23R2AbrWl-mi9Ddl3s0go
2017-05-30 22:03:34,271:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/root/.local/share/letsencrypt/bin/letsencrypt”, line 11, in
sys.exit(main())
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 742, in main
return config.func(config, plugins)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 682, in certonly
lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/main.py”, line 77, in _get_and_save_cert
renewal.renew_cert(config, domains, le_client, lineage)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/renewal.py”, line 296, in renew_cert
new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/client.py”, line 313, in obtain_certificate
self.config.allow_subset_of_names)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/certbot/auth_handler.py”, line 66, in get_authorizations
self.authzr[domain] = self.acme.request_domain_challenges(domain)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 212, in request_domain_challenges
typ=messages.IDENTIFIER_FQDN, value=domain), new_authzr_uri)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 191, in request_challenges
response = self.net.post(self.directory.new_authz, new_authz)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 674, in post
return self._post_once(*args, **kwargs)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 687, in _post_once
return self._check_response(response, content_type=content_type)
File “/root/.local/share/letsencrypt/local/lib/python2.7/site-packages/acme/client.py”, line 574, in _check_response
raise messages.Error.from_json(jobj)
Error: urn:acme:error:rateLimited :: There were too many requests of a given type :: Error creating new authz :: too many currently pending authorizations

[Jeromnimo]

Hi @mnordhoff, no sounds good:
The little one is a mistake from me, I ran certbot alone, with no arguments. Otherwise, always getting a 429 answer error

-rw-r–r-- 1 root root 48080 May 30 13:48 letsencrypt.log.3
-rw-r–r-- 1 root root 1898 May 30 22:43 letsencrypt.log.2
-rw-r–r-- 1 root root 14232 May 30 22:46 letsencrypt.log.1
-rw-r–r-- 1 root root 14232 May 31 00:03 letsencrypt.log

[schoen]

Oh, I’m sorry, I got confused between the error that you were having and the error that someone else was having. I was wrong to say that one hour would be enough. You’ll have to wait longer, depending on when the problem first started.

There is a way to make this happen sooner in this particular case, but it requires some programming—I don’t think that someone has yet written an application to do this. It is described in “Clearing Pending Authorizations” at

https://letsencrypt.org/docs/rate-limits/