In the last few days we've been seeing a lot of "urn:ietf:params:acme:error:rateLimited: Service busy; retry later." ACME failures. Is there some kind of resource issue happening now? We've never seen these messages before and it's quite disruptive. I didn't see anything on the status page about it.
I'm running into the same issue.
Certainly sometimes it happens, but I haven't heard of it happening more often recently. The staff generally keep a close eye on their metrics. Can you quantify how many requests you're sending and how often you're seeing "Service busy" in response? Do you send a lot of requests at some particular time (especially if it's near 0-minutes-after-an-hour) or are they spread out throughout the day?
I wouldn't expect occasional errors to be "quite disruptive"; most clients will just try again later like it says (and so many users probably wouldn't even notice unless they happen to be looking at their logs). Can you clarify what client you're using and how needing to retry later is so disruptive?
We've had 6 certificates fail to renew over the last 15 minutes. We do retry of course, but for some of our customers they are trying to onboard and get moving and this delays them.
As certificate lifetimes get shorter and more software like Caddy is using on-demand TLS cert issuance, this is indeed disruptive.
Out of how many? You said you saw an increase "in the last few days"; what percentage of time are you getting this?
I thought Caddy specifically tried multiple CAs, so that in the case of any one CA having an issuance outage it could still get a certificate from another CA.
I just want to know if there's a problem happening currently, or if this is some kind of new normal I need to account for.
Out of how many? You said you saw an increase "in the last few days"; what percentage of time are you getting this?
6 for 6 failed to renew just now. So getting it 100% of the time in the last 15 minutes. With two retried 5 minutes later and both of those failed as well. A third got through.
They do have an open "status" for the last week and a half that they're operating with "with reduced redundancy", so it may be that there some load challenges that they're trying to improve.
Yes, I saw that, and I was assuming that might be part of the problem, just looking for confirmation.
I run certbot like:
certbot certonly --cert-name <name> --standalone --expand -d www.example.com
With about 20 subdomains. Every time I run that command I get the same error:
acme.messages.Error: urn:ietf:params:acme:error:rateLimited :: There were too many requests of a given type :: Service busy; retry later.
An unexpected error occurred:
Service busy; retry later.
I've now tried to run that command for about ~5 times w/o any luck. Any advice on how to proceed?
How long of a time period between your first and last attempt?
You will get rate limited for 5 errors in an hour for the production LE system. But, I am not certain that particular Server Busy counts for that. If it's actually a server overload problem probably not. Just something to be wary of: Rate Limits - Let's Encrypt
For now there isn't much you can do other than retry periodically. Avoid repeating frequently in case there is too much load everyone retrying frequently makes it worse. Maybe retry every 12 minutes and then less often if persists. We'll have to see if it gets reported as an issue on the Status page: https://letsencrypt.status.io/ or if a member of staff comments here.
You could also try that against the LE staging system to ensure your cert request is valid and not suffering from an authorization problem.
Thanks!
I think I tested the first 3 just directly after each other the 4th and 5th about 15min in between. I just did another test ~30 minutes later which gave me a lot more output (I added -vv) and the script seemed to be doing a lot more, however also failed with the same error.
When I use --dry-run everything works fine.
The "Service busy" message also in theory comes along with a Retry-After header to say when would be a good time to try again, though I don't know if there are any clients out there that actually use it.
The only Retry-After header I could find (after adding -vv) was Retry-After: 3.
Just for clarity, there isn't any chance either of you might be hitting the general Overall Requests Limit per IP, maybe though like some kind of NAT putting a lot of requests from a lot of systems all through one source IP from Let's Encrypt's perspective? Just brainstorming, I doubt it's actually your problem.
Hmm maybe, but how would I be able to verify that?
Well, how many systems requesting how many certificates per minute are you using? Are you using your own IP space or is your provider somehow sharing the outbound IP you're using with others?
Also, I'm wondering if you are getting the error always at the same step in the process (like, always on the new-nonce, or the new-order, or the finalize endpoint) or if it's a different endpoint being requested that fails each time. Though I'm not sure what to do with that information one way or the other.
We're definitely not at the IP limits. This is something we haven't seen before - but only in the last week is it happening. Our traffic and issuance patterns have not changed dramatically. It's fine if this is expected behavior, we can work around it if it continues - but it felt more like an infrastructure problem.
After several attempts I could update my certificates.
There was a brief disruption at 16:10UTC which paged us and was remediated. That is likely what was experienced today. Please be aware, we have an ongoing incident. While our API is operational, until we have redundancy restored, there may be some increased rate limiting during peak request times. Please keep an eye on the incident status page for any changes.
Thanks!