Help with rate limiting

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: shupp-ratelimit-3182020.vinztest.com (and all other domains our service creates)

I ran this command: Service integrating with LE via Lego

It produced this output: Hit rate limit, too many certificates already issued

The version of my client is lego at 286c44337e7fb0b3aceaf71c42448e887a6db218

We have built a service for creating certificates for our department. The service started receiving rate limits for all creations and renewals within the last couple of hours. We have an increased rate limit for account umbrella-auto-ssl-certs@cisco.com.

If you could provide any details on why we’re being rate limited, that would be greatly appreciated. From what we see, we should be well below the increased limit.

Thank you!

Bill Shupp

2 Likes

https://tools.letsdebug.net/cert-search?m=domain&q=shupp-ratelimit-3182020.vinztest.com&d=744

2 Likes

The search there assumes you don’t have any rate limit exemption, which OP does.

Something is a little weird though, check this out:

If I try create a certificate for xxx-not-real.vinztest.com:

>There were too many requests of a given type :: Error creating new order :: too many certificates already issued for: vinztest.com: see https://letsencrypt.org/docs/rate-limits/

but if I try create one for 4971dfd3-7a3c-435b-82e9-92c141d468c9.vinztest.com, it creates the order just fine and tries to go through domain validation.

What’s the difference between them? Both one label. Hmm.

Edit: nevermind. I think what’s happening is that their server is on the threshold of the rate limit, and as time progresses, individual orders are going through and hitting the rate limit again.

@lestaff probably worth pinging them to check your rate limit for you, anyhow.

2 Likes

I assumed without exemption it would have triggered 429s a lot earlier than 173.

this error message is explicitly referring to the “Certificates per Registered Domain” limit, I think.

1 Like

Thanks @_az for the response. @lestaff, could you confirm our increased rate limit is still in place? Again, the acme email being used is umbrella-auto-ssl-certs@cisco.com.

Thank you!

1 Like

@9peppe thanks for the response and info. This does not reflect our rate limit increase.

1 Like

Looking at our logs and rate limit config, you do have a rate limit override in place for “Certificates per Registered Domain.” However, you don’t current have a rate limit in place for “New Orders” (300 per account per 3 hours). The New Orders rate limit is new in ACMEv2 and may be newly hitting you if you recently migrated. Would you mind filling out an additional rate limit increase for the New Orders limit at https://docs.google.com/forms/d/e/1FAIpQLSetFLqcyPrnnrom2Kw802ZjukDVex67dOM2g4O8jEbfWFs3dA/viewform? If it’s causing you urgent problems I can escalate and make sure a teammate handles it rapidly.

Also, I wanted to mention: In your output above, you have “Hit rate limit, too many certificates already issued.” I can’t find that message in lego, so I assume it’s an aspect of your service integration. The error messages we send have both a type (rateLimited), and a detail, which in this case was 429 :: rateLimited :: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/. I’d recommend tweaking your service integration so that it shows the error detail on any problems. That will make cases like this much faster to diagnose.

7 Likes

@jsha thanks very much for the prompt response. I have just filled out the form, please let me know if you have any questions or concerns. We are currently unable to create new certificates, which is more impactful than renewal rate limiting. If you are able to expedite the increase, that would be appreciated.

Regarding the logging, we do log the lego error messages, along with our own error messages. Unfortunately I posted our error message rather than the lego message at the top of this thread. Going forward, I’ll be sure to use the lego error message.

Thanks again for your help @jsha!

2 Likes

The rate limits are rolling, and that one is only over a time period of 3 hours. You shouldn’t be totally unable to issue certificates for long.

4 Likes

The adjustment is now live. Let me know if you have any more problems. By the way, welcome to the forum!

5 Likes

Thanks very much @jsha!

1 Like

@mnordhoff you are correct, after that post we did see some go through, though some were still getting limited.

Incidentally, we scheduled renewals of about half of our certificates after the announcement of impending revocations a couple of weeks ago due the CAA security bug incident. What we didn’t consider was that put half of our certificates on a schedule of renewing on the same day, which is why we hit this limit today.

We’ll be addressing this tomorrow to spread out the renewal schedules to avoid spikes like this in the future. Thanks again for everyone’s input and for the quick response from the LE staff.

3 Likes

4 posts were split to a new topic: Rate limit increase