I’ve written a custom ACME client integration to request certificates for a large number of domains (8,000+). I have read and understand the limit of 300 new orders every 3 hours for an ACMEv2 client. However, I seem to be hitting this limit in situations where it is not expected.
For instance, I created exactly 100 new orders earlier today before reaching the limit and getting a 429 error. I had not made any orders within the 3 hours prior to that time.
My questions are:
Is the 3 hour limit rolling or based on UTC?
Once the limit is reached, is there a “Retry-After” of exactly 3 hours?
Is there an undocumented sub-limit such as 100 per hour and 300 per 3 hours, whichever is hit first?
The rate limit docs just don’t seem to match the behavior I’m seeing, and I would love to understand how it works and document it clearly for someone trying this in the future.
This client reached the rate limit of the server: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/ (on request "POST https://acme-v02.api.letsencrypt.org/acme/new-order")
I’m having some trouble locating the header on the POST response with my account id. Would a crt.sh ID for one of my certs suffice?
Ok! That rules out the overall requests rate limit or the certificates per domain ratelimit @rg305 mentioned.
If you share a domain name you've been issuing for I can work backwards to the account ID. Or if you're interacting with the API from a single IP address I can work with that too.
Great, thank you. I was able to find your ACME account ID (it was 45707749 for your records :-)). Looking at the past 14 days I see that:
on 19/11/2018 you successfully created 301 new orders.
on 20/11/2018 you successfully created 499 new orders.
Looking at the 19th I see the first successful new-order response was at 19/11/2018 20:34:11. The first 429 was at 19/11/2018 20:48:31, exactly 300 orders later. That matches my expectation of how the rate limit is implemented since the requests were all within the 3 hour rolling window started at 20:34. I didn't check the 20th but I bet it would match up too.
Overall you sent a lot more new-order requests, the difference would be the # of 429's received.
on 19/11/2018 there were 557 new-order requests
on 20/11/2018 there were 646 new-order requests
When I quickly looked at the new order requests it seems like your system is submitting duplicate orders for every domain name. E.g. on 20/11/2018 18:02:36 your system POSTed new-order with one identifier client.davywhitener.com. Less than one second at 20/11/2018 18:02:37 your system POSTed new-order with the same identifier. That shouldn't affect your new-order rate limit because our system is smart enough to reuse an already existing pending order for the duplicate order request. That said: it does indicate a potential bug on your side and it will affect your overall request rate limit negatively.
One other point of feedback: Your ACME integration is sending the user-agent "GuzzleHttp/6.2.1 curl/7.53.1 PHP/7.2.8" It would be nice if you could change that to something that also includes your organization as an identifier. It makes it a lot easier for us to separate your ACME client from other PHP clients if you can do that.
One more thing! If you haven't already fiddled with it you might find installing Pebble and configuring your ACME integration to use it as the ACME server might help you debug this. You won't have to worry about rate limits that way and can interrogate both the client and server side.
Could it perhaps be that all the checks the code does, before adding it to the NewOrder-database at the bottom of the function, are too slow? That, when two orders are requested fast enough, the first isn't added to the database quickly enough, so both NewOrder()-functions are running simultaneously?
@cpu I studied the call stack carefully and discovered that the duplicate calls to new-order are somewhat intentional in the client lib I am using (https://github.com/acmephp/core). Since I am using http-01 authorization, I must explicitly request this authorization type prior to requesting a new order. But requesting authorization creates a new-order as well.
I am going to test a different method of requesting the authorization which will result in only one call to new-order. I mention it here in case anyone encounters this issue in the future.
I'm sorry, that doesn't make much sense to me. The ACME protocol doesn't have the client choose a challenge type by explicitly requesting it. You always just follow the same process:
The client asks the server to create a new order for a set of identifiers
The server responds with an order. The order has URLs for an authorization for each identifier. Each authorization has some challenges.
The client picks one challenge for each identifier authorization and proceeds.
Let's Encrypt will always return a HTTP-01 challenge as part of step 2 unless the identifier is for a wildcard name. You don't need to create two orders to pre-select HTTP-01 or anything of the sort.
It still feels like a bug to me but perhaps I'm misunderstanding why ACMEPHP is doing this.
I would classify it more as an unintended side-effect of using the API client a certain way. It is possible to memoize the resulting order and use a call to finalize it. In that case, I just have to fetch the authorizations manually and find the one I want (http-01). In looking at the methods of the client alone, it seemed like less work and it wasn’t initially apparent that it would result in duplicate calls to new-order.
I suppose it could be an item for documentation clarification since there isn’t a single way to consume the LetsEncrypt ACME API with this client. It allows for some greater flexibility, and the ability to shoot yourself in the foot, as I have done here.
