This problem was raised a bunch of times already, but I haven’t found the answers to these questions:
Is there a way to get a list of pending authorizations from the LE API? Or at least a metric that shows the number of pending authorizations for particular account? We are creating/renewing ~80 certificates per minute on a regular basis and we are probably leaking some authorizations, and when we reach the limits - it’s not possible to easily find out which authorizations were leaked, and checking ALL of them usually takes close to a day.
Is there a way to increase the number of pending authorizations for particular account?
There isn't an ACME endpoint to retrieve a total list of pending authorizations for an account, and Boulder/LE don't provide one above-and-beyond what ACME specifies. There also isn't a user-visible metric for how many pending authorizations have accrued, though that may be something we could think about.
You can request a rate limit adjustment for this limit. Please be sure to include all of the requested information - folks frequently miss something and based on the volume of requests & our staff size we often can't follow-up if the request is incomplete.
How frequently do you rebase against the latest code from Lego master? In particular I know this commit has supposedly helped with some pending authorization leaks experienced by this client. Does your fork include that commit?
In general it should be possible to carefully log all of the authorizations created to do the bookkeeping required to count pending authorizations and to know whether any have been leaked. If you're regularly leaking authorizations using Lego I would encourage you to try and work with the Lego upstream to resolve the root cause since it is indicative of a client bug or a corner case.
@fxposter One additional note: We recently landed a new feature to automatically reuse existing pending authorizations when a client asks for a new pending authorization be created for the same identifier. This isn’t enabled in production or staging at this time but should be enabled in the coming weeks (along with an announcement in the API announcements subforum). This should help mitigate this particular rate limit from the server side as well.
We are using master + some commits on top of it (which are pull-requests into xenolf/lego).
And by inspecting all the logs today - I think I’ve found the root cause - xenolf/lego uses an http client with 10-second timeouts and we do actually observe a bunch of timeouts when doing a new-authz call, which explains those random authorizations that are in pending queue, but we never actually see them, because we timeout-ed (distributed systems are hard :().
As we issue a lot of certificates - that pending queue is not behaving predictably, so any additional solutions to work with this problem would be quite useful. In the meanwhile, I will adjust the timeout to 30 seconds for all calls in xenolf/lego (as it’s not possible to increase it only for new-authz call).
We will try request an additional authorizations anyway, thanks.
@cpu By the way - can you share a histogram and a max duration of a new-authz call on the LE servers? This will allow us to configure timeout a bit more precisely, cause 30sec is just a random guess (which actually still timeout, but less often).