Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
My domain is: www.navy.mil
We are hitting the rate limit for www.navy.mil certificate. As per the link https://crt.sh/?q=ipv6.navy.mil , the certificate is issued but our API is still not receiving the certificate.
The last API response I see from LE is below
2019-03-01 09:27 GMT Let’s Encrypt: Error creating new cert :: too many certificates already issued for exact set of domains: ipv6.navy.mil,outreach.navy.mil,www.ah.mil,www.amphib7flt.navy.mil,www.c3f.navy.mil,www.c7f.navy.mil,www.ccsg15.navy.mil,www.clwp.navy.mil,www.cprw1.navy.mil,www.csp.navy.mil,www.cusnc.navy.mil,www.ewtgpac.navy.mil,www.navsea.navy.mil,www.navy.mil,www.oni.navy.mil,www.outreach.navy.mil,www.seaport.navy.mil,www.srf.navy.mil,www.ttgp.navy.mil: see https://letsencrypt.org/docs/rate-limits/
Is there a way to relax this rate limit? Any suggestions here will be very helpful.
No. This rate limit indicates that you are issuing the same certificate over and over. The solution is to issue the certificate once and distribute it as needed.
Do you know more about why this is? What ACME client/provider are you using? It sounds like a bug with their integration.
@cpu Daniel, thanks for your response. The issue here is that as per LE the certificate is issued https://crt.sh/?q=www.navy.mil but we are not seeing it in response from LE. I have shared the logs with you if that is fine.
I looked at our logs and they don't match up with yours. In your logs I saw what looked like a logged response from a POST to the ACME v1 new-cert endpoint that appeared to indicate a 500 Internal Server Error was returned from the ACME server:
That led me to believe there might have been an issue on our side.
However, looking at our logs for the past 7d I can see that there were ~275 POSTs to new-cert from the ACME account in question that contained a .mil domain. None of those requests received a 500 response. I see the following breakdown across the 7d window:
HTTP 403 responses - 196 (~71%)
HTTP 201 responses - 45 (~16%)
HTTP 429 responses - 34 (~12%)
The vast majority of the 403 responses are errors indicating to the ACME client that there were problems re-checking CAA for the zones in question. Here's one example status 403 error message that would have been returned to your client:
"Error":"403 :: caa :: Error creating new cert :: Rechecking CAA: While processing CAA for www.c7f.navy.mil: DNS problem: query timed out looking up CAA for c7f.navy.mil, While processing CAA for www.clwp.navy.mil: DNS problem: query timed out looking up CAA for www.clwp.navy.mil, While processing CAA for www.ttgp.navy.mil: DNS problem: query timed out looking up CAA for www.ttgp.navy.mil, While processing CAA for www.csp.navy.mil: DNS problem: query timed out looking up CAA for www.csp.navy.mil, While processing CAA for www.seaport.navy.mil: DNS problem: query timed out looking up CAA for www.seaport.navy.mil, While processing CAA for www.ccsg15.navy.mil: DNS problem: query timed out looking up CAA for www.ccsg15.navy.mil"
This indicates a problem with your authoritative DNS servers. See our CAA documentation for more information.
The 201 responses were successful certificate issuances. There's nothing to indicate we failed to deliver the certificate to your client and no indication of 500 errors. I still believe there is a problem with your ACME client/integration. Have you engaged with Akamai's engineers about this?
The 429 responses are rate limiting errors like the one that prompted opening this thread. They're caused by re-issuing duplicates of previously issued certificates (e.g. the ones that got a 201 response).
Can you try and find out why your logs contain status responseCode: '500' lines without any HTTP body response body or headers? Our systems should never generate a 500 error in response to a new-cert POST that has no response headers or body.
Before edit:
–In old days when LE used intermediate X1, it had name constraints that deny .mil addresses.
While it’s removed in X3 intermediate cert, something may left in codebase that still denying making such certs. Or even possible that still LE should not sign .mil because of contract between identrust.–
Thanks for checking the logs. The logs on your end shows errors related to CAA queries time out but our logs show “Error creating new cert :: too many certificates already issued for exact set of domains: ipv6.navy.mil,outreach.navy.mil,www.ah.mil,www.amphib7flt.navy.mil,www.c3f.navy.mil,www.c7f.navy.mil,www.ccsg15.navy.mil,www.clwp.navy.mil,www.cprw1.navy.mil,www.csp.navy.mil,www.cusnc.navy.mil,www.ewtgpac.navy.mil,www.navsea.navy.mil,www.navy.mil,www.oni.navy.mil,www.outreach.navy.mil,www.seaport.navy.mil,www.srf.navy.mil,www.ttgp.navy.mil: see https://letsencrypt.org/docs/rate-limits/”
I wanted check, once the certificate is issued, is the rate limits checked after that or it would be checked even before issuing the certificate?
I don't think you correctly interpreted what I shared. The 196 instances of a HTTP 403 response I mentioned would have had the CAA recheck error from the timeout. The 34 instances of a HTTP 429 response I mentioned would have had the "too many certificates already issued" error. It isn't the case that my logs don't contain the rate limit errors you're receiving. If your logs don't have both the HTTP 429 rate limit errors and the HTTP 403 CAA errors then I suspect your logs are incomplete or the ACME client isn't logging all errors.
The duplicate certificate rate limit (which corresponds to the "Too many certificates already issued for exact set of domains" error) is check before issuing new certificates.
Yeah, I work for Akamai. Thanks, it makes sense now. If the check is done before issuing the certificate and certificates are getting issued, I will check this again on our end, as to why we are not receiving the certificate.
We were finally able to download the certificate! I wanted to check if LE has seen timeouts/client aborts from when trying to send this certificate? We suspect this is related to difference in timeout values at our end vs LE.
I think you are chasing a different problem than the two I was able to identify from the evidence at hand. To restate: your system is getting 403's for new-cert because of CAA timeouts and 429's from rate limiting because you're reissuing the same certificates over and over. What was done to address these problems and the way your integration responds to them?
Ignoring the problems identified from the evidence to talk about your certificate fetching timeout, what is the cause of your suspicion here? Do you have timestamped logs that show your ACME integration POSTing new-cert and not receiving a 429, a 403 or a 200 response but instead some kind of HTTP timeout error?
