Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.
I ran this command: Service integrating with LE via Lego
It produced this output: Hit rate limit, too many certificates already issued
The version of my client is lego at 286c44337e7fb0b3aceaf71c42448e887a6db218
We have built a service for creating certificates for our department. The service started receiving rate limits for all creations and renewals within the last couple of hours. We have an increased rate limit for account umbrella-auto-ssl-certs@cisco.com.
If you could provide any details on why we’re being rate limited, that would be greatly appreciated. From what we see, we should be well below the increased limit.
but if I try create one for 4971dfd3-7a3c-435b-82e9-92c141d468c9.vinztest.com, it creates the order just fine and tries to go through domain validation.
What’s the difference between them? Both one label. Hmm.
Edit: nevermind. I think what’s happening is that their server is on the threshold of the rate limit, and as time progresses, individual orders are going through and hitting the rate limit again.
@lestaff probably worth pinging them to check your rate limit for you, anyhow.
Looking at our logs and rate limit config, you do have a rate limit override in place for “Certificates per Registered Domain.” However, you don’t current have a rate limit in place for “New Orders” (300 per account per 3 hours). The New Orders rate limit is new in ACMEv2 and may be newly hitting you if you recently migrated. Would you mind filling out an additional rate limit increase for the New Orders limit at https://docs.google.com/forms/d/e/1FAIpQLSetFLqcyPrnnrom2Kw802ZjukDVex67dOM2g4O8jEbfWFs3dA/viewform? If it’s causing you urgent problems I can escalate and make sure a teammate handles it rapidly.
Also, I wanted to mention: In your output above, you have “Hit rate limit, too many certificates already issued.” I can’t find that message in lego, so I assume it’s an aspect of your service integration. The error messages we send have both a type (rateLimited), and a detail, which in this case was 429 :: rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/. I’d recommend tweaking your service integration so that it shows the error detail on any problems. That will make cases like this much faster to diagnose.
@jsha thanks very much for the prompt response. I have just filled out the form, please let me know if you have any questions or concerns. We are currently unable to create new certificates, which is more impactful than renewal rate limiting. If you are able to expedite the increase, that would be appreciated.
Regarding the logging, we do log the lego error messages, along with our own error messages. Unfortunately I posted our error message rather than the lego message at the top of this thread. Going forward, I’ll be sure to use the lego error message.
@mnordhoff you are correct, after that post we did see some go through, though some were still getting limited.
Incidentally, we scheduled renewals of about half of our certificates after the announcement of impending revocations a couple of weeks ago due the CAA security bug incident. What we didn’t consider was that put half of our certificates on a schedule of renewing on the same day, which is why we hit this limit today.
We’ll be addressing this tomorrow to spread out the renewal schedules to avoid spikes like this in the future. Thanks again for everyone’s input and for the quick response from the LE staff.
