It produced this output: This client reached the rate limit of the server: Error creat \n ing new order :: too many certificates (5) already issued for this exact se \n t of domains in the last 168 hours: emperorrealestateinvestmentllc.com
My web server is (include version): nginx/1.21.1
The operating system my web server runs on is (include version): CentOS Linux 7 Core
My hosting provider, if applicable, is: AWS
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Notes:
The rate limit was accidentally hit, however we have waited over a week to get a new certificate for this domain and it is still reporting too many request.
Can some help us clear this up, as we should be well past the rate limit by this point.
The first cert on the 4th or April has a notBefore date of "Apr 4 17:55:27 2022 GMT"
That cert has been back-dated one hour, so you have to keep that in mind too. And it's a sliding window of 168 hours, on the minute/seconds exact, so you need to wait a few more hours.
Please use the staging environment for testing in the future.
I waited another day and tried again and am still receiving the same error message.
This issue originated due to an error within our UI allowing our support team to click a button more than once and was not due to testing. This issue has since been fixed.
Is there any way to clear this issue up as we have waited well over the 168 hour threshold?
No, the link is to the actual certs that were created. It is the Certificate Transparency Log for all major CA's to log public activity.
Someone must be making requests. Did you setup some other way to get certs earlier and perhaps it is still active? Often we I see someone get 5 certs in a day like you did on Mar28 and Apr4 they have setup a docker or virtual server that requests a cert at each startup. These systems should store the cert in persistent store to avoid that. Could that be happening?
One clue is in the history the "Common Name" is different. That tells me there may be more than one kind of acme client making requests. Or, at least different command lines or configs for requesting the cert.
We have a single php file that makes the request for a certificate. This script exists on a web server that does not have a restart cycle.
At the beginning of that php files code is a log of every request made to that file. Our log shows one request to that file today, which was the call I made this morning because I was told yesterday to wait.
I'm going to review the code again, but we have no loops in the code that would be causing the request for a certificate to be made more than once, so I'm unsure how it's being made multiple times.
Having a different "Common Name" may be explainable. The acmephp package allows you to pass an alternative-name value and we were passing www. version of the name as alternative.
Perhaps the request was being made twice while using that argument and it can be removed, however that still does not explain how we received an error saying we were over threshold, but the logs say we received a certificate.
I waited until today to make the request specifically because we wouldn't be in the threshold anymore, I expected a certificate but instead received an error.
{"code":429,"status":false,"error":"Too Many Requests","debug":{"message":"Reqeust authorization failure.","output":"Loading account key pair... \nRequesting an authorization token for domains emperorrealestateinvestmentllc.com, www.emperorrealestateinvestmentllc.com ... In ServerErrorHandler.php line 107: [rateLimited] This client reached the rate limit of the server: Error creat \n ing new order :: too many certificates (5) already issued for this exact se \n t of domains in the last 168 hours: emperorrealestateinvestmentllc.com,www. emperorrealestateinvestmentllc.com: see https://letsencrypt.org/docs/rate-l \n imits/ (on request "POST https://acme-v02.api.letsencrypt.org/acme/new-orde \n r") \n \n\nIn RequestException.php line 113:\n \n Client error: POST https:\/\/acme-v02.api.letsencrypt.org\/acme\/new-order re \n sulted in a 429 Too Many Requests esponse: \n { \n "type": "urn:ietf:params:acme:error:rateLimited", \n "detail": "Error creating new order :: too many certificates (5) (truncat \n ed...) \n \n\nauthorize [-s|--solver SOLVER] [--] ()...\n\n"}}
Above is the exact error we logged. Normally a local certificate is saved, in this case no certificate is being saved.
This is the only domain out of serveral hundred domains that is having this issue. We have not run into this problem elsewhere.
Which authentication method is being used?
If DNS-01 (or if that was ever previously used for this FQDN), then check any and all servers that ever made such requests [one might still be out there wreaking havok].
This would prevent Let's Encrypt from issuing any certs.
When you try again with acmephp in 6 days first change the CAA record to: 0 issue "letsencrypt.org"
This is a valid CAA for LE and will allow cert issuance.
Then, promptly change it back to XXXletsencrypt.org or any other wrong value to block this roque requester.
This can buy time until you find where they are getting requested from.
So this whole letsencrypt setup is an entirely new setup, recent within the last month.
We did some more digging a little further into our server setup and we just now realized that the AutoSSL service in cpanel is active and it set to use Letsencrypt. I looked through the logs and found that domain listed several times. I think that may be the bandit that is requesting certificates.
Going get that shut off as we don't need it and weren't even aware it was active until now.
Hopefully this solves are issue, I apologize for the time waste if it is.
