My domain is: *.ci-cd-9.ping-oasis.com
I ran this command: Kubernetes cert-manager with a staging lets encrypt request.
It produced this output:
E1104 16:42:00.955330 1 controller.go:157] "re-queuing item due to error processing" err="error listing alternate certificate URLs: 503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later." logger="cert-manager.controller"
E1104 16:42:11.073238 1 controller.go:157] "re-queuing item due to error processing" err="503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later." logger="cert-manager.controller"
E1104 16:42:31.188212 1 controller.go:157] "re-queuing item due to error processing" err="503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later." logger="cert-manager.controller"
E1104 16:43:11.307913 1 controller.go:157] "re-queuing item due to error processing" err="503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later." logger="cert-manager.controller"
E1104 16:44:31.478007 1 controller.go:157] "re-queuing item due to error processing" err="error listing alternate certificate URLs: 503 urn:ietf:params:acme:error:rateLimited: Service busy; retry later." logger="cert-manager.controller"
My web server is (include version): nginx-ingress-controller (various)
The operating system my web server runs on is (include version): linux (various)
My hosting provider, if applicable, is: AWS Route 53
I can login to a root shell on my machine (yes or no, or I don't know): Yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): cert-manager version v1.16.1, certbot version 2.11.0
Basically we started seeing rate limiting last week, and this is our CI/CD cluster, which only uses the staging LetsEncrypt servers. We have also seen this error when creating orders in cert-manager: Failed to create Order: 400 urn:ietf:params:acme:error:malformed: No Key ID in JWS header - which is strange because we believe that cert manager is setting the Key ID...
Basically comes down to we were having no issues at all and then suddenly saw what appeared to be rate limiting from LE, even though I don't believe we are near the LE staging limits set forth in Staging Environment - Let's Encrypt
Now we are in the process of upgrading cert-manager from version v1.12.1 so this could also be related as we don't see this issue consistently with v1.12.1 (though still occasionally). I know this isn't the cert-manager community but I was able to replicate the certbot issues by running it within a cluster we were seeing issues in. Whereas on my laptop with a different IP we didn't see the rate limiting.
In summary, any advice to point us towards how/why we are being rate limited would be super helpful. Thank you!!