Error 429 Too many pending authorizations

Help
1

Please fill out the fields below so we can help you better.

My domain is: .jewlr. (approx. 12 domains)

I ran this command: ??

It produced this output:

2/6/2017 1:07:52 PMlevel=info msg="Starting Let’s Encrypt Certificate Manager v0.4.0 3c41d73"
2/6/2017 1:07:53 PMlevel=info msg="Using locally stored Let’s Encrypt account for schuyler@jewlr.com"
2/6/2017 1:07:53 PMlevel=info msg="Using Let’s Encrypt Production API"
2/6/2017 1:07:53 PMlevel=info msg=“Found existing certificate ‘jewlr’ in Rancher"
2/6/2017 1:07:53 PMlevel=info msg=“Trying to obtain SSL certificate (dev.jewlr.com,dev.jewlr.ca,dev.jewlr.co.uk,dev.jewlr.com.au,dev.jewlr.fr,dev.jewlr.de,dev.jewlr.es,dev.jewlr.nl,dev.jewlr.ie,staging.jewlr.com,staging.jewlr.ca,staging.jewlr.co.uk,staging.jewlr.com.au,staging.jewlr.fr,staging.jewlr.de,staging.jewlr.es,staging.jewlr.nl,staging.jewlr.ie) from Let’s Encrypt Production CA"
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.es] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.ie] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.nl] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.co.uk] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.de] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.es] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.com] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.ca] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.ie] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.com.au] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.nl] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.fr] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.co.uk] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.ca] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.de] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[dev.jewlr.com] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.com.au] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.“
2/6/2017 1:07:54 PMlevel=error msg=”[staging.jewlr.fr] Error obtaining certificate: acme: Error 429 - urn:acme:error:rateLimited - Error creating new authz :: Too many currently pending authorizations.”

My operating system is (include version):

This Docker image https://github.com/janeczku/rancher-letsencrypt

My web server is (include version): Nginx

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

My cert is now expired after this error has been happening to me for apparently longer than a week. All of my dev/staging environments use the Let’sEncrypt cert.

2

Hi, can you tell us more about the software that this is using? I’m not sure that anyone here is familiar with the Docker image that you’re using or what software is used inside of it.

This error would generally be a result of a bug that lets a client prove control of a domain name but then fail to go on to request the associated certificate. The solution would be to fix the bug so that proofs of control over domain names are always followed by corresponding certificate requests. But I’m not even immediately sure what client software you’re using here (from the Docker image).

3

I’ve left an issue open with the developer here: https://github.com/janeczku/rancher-letsencrypt/issues/63

I do not have any awareness of the internals of the library, it is a black box to me.

However, I don’t have any earlier logs from before the 429 errors so I can’t determine root cause until the rate limit is removed from my account. The client normally proves control using Route53 DNS entries created through AWS in my scenario.

4

Oh, I see. That tool has its own client written in Go (maybe based on another Go client or maybe not?).

So I think the likeliest interpretation is a bug in the rancher-letsencrypt Go client that makes it sometimes request an authorization and then not use it (either requesting an authorization when not requesting a certificate, or requesting an authorization and then crashing or exiting before the corresponding certificate can be requested). This could, for example, be a renewal-related bug if one part of the code says “this certificate should be renewed now” but another part of the code says “this certificate is not yet due for renewal”.

This could probably be pinned down if the client could be set to a higher logging level so that you could go back and see what actions it thought it was trying to take a why.

5

For now I’ve started with a fresh Let’s Encrypt account and certificates since I don’t have the time to broaden my scope on the issue, hopefully the developer will pick up on this bug. It seems to be working with a new account.

6

I’ve figured out the problem, and it’s not on Let’s Encrypt’s side:

[rancher@ip-172-16-0-216 ~]$ docker logs 16e9e872da5f
level=info msg="Starting Let's Encrypt Certificate Manager v0.4.0 3c41d73" 
level=info msg="Generating private key (2048) for ssl@jewlr.com." 
level=info msg="Creating Let's Encrypt account for ssl@jewlr.com" 
level=info msg="Using Let's Encrypt Production API" 
level=info msg="Trying to obtain SSL certificate (dev.jewlr.com,dev.jewlr.ca,dev.jewlr.co.uk,dev.jewlr.com.au,dev.jewlr.fr,dev.jewlr.de,dev.jewlr.es,dev.jewlr.nl,dev.jewlr.ie,staging.jewlr.com,staging.jewlr.ca,staging.jewlr.co.uk,staging.jewlr.com.au,staging.jewlr.fr,staging.jewlr.de,staging.jewlr.es,staging.jewlr.nl,staging.jewlr.ie) from Let's Encrypt Production CA" 
level=error msg="[dev.jewlr.ie] Error obtaining certificate: Error presenting token: Failed to determine Route 53 hosted zone ID: Could not find the start of authority" 
level=error msg="[staging.jewlr.ie] Error obtaining certificate: Error presenting token: Failed to determine Route 53 hosted zone ID: Could not find the start of authority"

There was some old configuration persisted somehow that was using a domain that we don’t own nor have a Route53 domain for. The image did not handle things correctly and the health check killed the container repeatedly, going into an endless loop, which eventually caused these 429 errors.

Thanks for the help!

1 Like
7

Maybe this does lead to some useful guidance for client developers: if you get an authz for one requested domain but fail to get it for another, make sure you proactively destroy the first authz before giving up. (If your error was based on repeated failed attempts to get a certificate for a mixture of names you do and don’t control, that might be the underlying problem here.)

2 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.