"detail": "Rate limit for '/acme' reached"

I get a very limited error message:
{"type": "urn:ietf:params:acme:error:rateLimited", "detail": "Rate limit for '/acme' reached"}

My domain is: equip.run

My web server is (include version): Mako Server using a specialized acme client: Automatic SSL Certificate Management

The domain equip.run is a test domain and could have many clients, but I do not see much going on when inspecting https://crt.sh/?q=equip.run

I tried the following debug tool, but it seems to expect special formats:
https://tools.letsdebug.net/clear-authz
I could not complete step two.

The rate limit is not related to authorizations. It's the following rate limit you're hitting:

The “/directory” endpoint and the “/acme” directory & subdirectories have an Overall Requests limit of 40 requests per second.

(Source: Rate Limits - Let's Encrypt)

It's probably 40 requests per second per IP address, but I'm not sure.

This can either be due to a buggy ACME client or you have like, hundreds or thousands of certificates which are e.g. renewing at the same time. If it's the latter, please throttle your ACME clients outbound connection to the ACME API so it sends less than 40 requests to the server per second. If it's the former, please fix your ACME client.

3 Likes

crt.sh is under stress today and not always showing all certs. I just tried request below and see there are nearly 200 active certs for that apex with many subdomains. I agree with advice from @Osiris

https://crt.sh/?Identity=equip.run&exclude=expired&deduplicate=Y

3 Likes

So perhaps the ACME client of @SoCalDude is renewing all those certs? Maybe due to the recent incident with the tls-alpn-01 challenge?

3 Likes

Thanks for the details, but I do not think any of the above is applicable. The engine controlling the ACME client activates one time every 24 hours. It will then check all certs and initiate an update for any expired certificate. This is done is series, not parallel. Each update takes two minutes due to its exclusive use of ACME dns-01 challenge settings. In other words, the server can at maximum send a request every two minute.

1 Like

This is the first I have ever seen SharkTrust. Do you run your own SharkTrust server or do you have a client that connects to that service which in turn obtains your certs? If the latter perhaps the common SharkTrust server is getting that error due to activity from other customers.

I think you need to ask SharkTrust support to provide more details of that error.

I see this from the site you linked. It will be hard for us to know that level of detail. Well, me anyway :slight_smile:

Note that SharkTrust is an advanced tool that typically requires initial consultation provided by Real Time Logic.

2 Likes

I am the guy who designed SharkTrust :slight_smile:

3 Likes

The plot thickens :slight_smile: I am probably not much more help so will step aside. Just for clarity, is it the SharkTrust server that is the sole client to Let's Encrypt? Are there logs which would confirm the low volume of requests you describe?

@osiris Seems like this belongs in Client Development section? What do you think?

2 Likes

Not sure, it's more like a client implementation issue perhaps, if at all. I think the #client-dev section is more fore people who want to write/develop an ACME client, so more the inner workings. Here it seems the client itself does function properly, but its implementation might be off. I'd rather keep it here in #help for lack of a better place.

@SoCalDude Could you perhaps elaborate a little bit more about the structure of SharkTrust? Do all clients reach out to a central server, which in turn connects to Let's Encrypt or something like that?

3 Likes

Just to clarify, regardless of flaws in the client, it should be infeasible for the client to issue more than one ACME request per two minutes. Also, this is the first time I have seen the rate error.

Yes, all clients reach out to a central server, which in turn connects to Let's Encrypt. The domain equip.run is a test domain, enabling anyone to test the service. However, the two minute rate still applies.

The server itself is also exclusively used for test purposes and anyone can register their own domain: https://sharktrust.realtimelogic.com/

The following user manual explains in detail how it works:
https://github.com/RealTimeLogic/SharkTrust/blob/master/doc/SharkTrust-User-Manual.pdf

Is that rate also true if the requests fail? Such as temp comms error, malformed requests, Let's Encrypt server outage, ... Do you have log / alert that would identify such cases?

Have you written your own ACME client from scratch or have you integrated an existing one? I searched this forum history and only saw faulty clients as cause of this error.

Are you still getting same rate limited error?

3 Likes

Yes, the stack is documented here:
https://realtimelogic.com/ba/doc/en/lua/acme.html
The engine is part of SharkTrust. The engine logs errors, but I'll add more logs and I'll let you know.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.