Failed to create Order: 429

Hi LetsEncrypt team,

We are facing a rate limiting issue when requesting a certificate for the ondemand.com domain. We are using cert-manager deployed on Kubernetes to generate the certificate. The error looks like this:

Failed to wait for order resource
        "<placeholdername>-external-certificate-m8vv2-1825712214" to
        become ready: order is in "errored" state: Failed to create Order: 429
        urn:ietf:params:acme:error:rateLimited: Error creating new order :: too
        many certificates already issued for "ondemand.com". Retry after
        2022-12-06T00:00:00Z: see  https://letsencrypt.org/docs/rate-limits/
  • We recently increased the Certificates Per Registered domain limit to 300 by completing the rate limit form and we are not completely sure why we are hitting these limits because when we look at crt.sh, and search for ondemand.com, we don’t see any new certificates created recently. Can you please help us investigate this by providing potential causes and tips to troubleshoot this further?

  • Secondly, what does it mean when we increased the Certificates Per Registered domain limit to 300? We were under the impression that this limit applies to all LetsEncrypt accounts. But since we just successfully requested an increase to the limit by 300, does this still apply across all accounts or is it just for account that we requested an increase for? If you can provide some more details about this, that will be great. Thanks!

Kind Regards,
Fawaz

The rate limit exception will depend on how you filled out the form: it will be specific to a single ACME registration (account) if you provided a regID, or it will be available to all registrations if you provided the domain name.

7 Likes

I think that query limits on crt.sh prevent them from showing up, due to the sheer number of certificates that match.

If you search for certificates issued in the last day or so on Censys, we see ~297 certificates.

7 Likes

All instances of your ACME client waiting to Retry after?

3 Likes

Hi James, thanks for your reply.

So, when you say "it will be specific to a single ACME registration", does the rate limit apply to certificates only created by this account? or does the creation of an certificate(same registered domain) from another account add to the limit too?

For example lets say you have two accounts Account A and Account B and lets say the Account A has a limit of 300 and Account B has a limit of 1000.

If Account B creates 400 certificates for a specific registered domain, it can still create more because it hasn't reached it's limit yet. But now, Account A cannot create more because Account B already created >300 certificates.

So, I guess, my question is, when you say an account has a limit of 300 Certificates Per Registered domain, does it mean 300 Certificates Per Registered domain CREATED by this account or created any account?

I'm guessing but I'd expect it's cumulative across any account. Note that you can share accounts across machines by using the same account key in your config.

4 Likes

To me, the answer should be obvious:

  • if Account A didn't issue the certs, then they don't count against it.
  • if Account B didn't issue the certs, then they don't count against it.
3 Likes

You'd think, but that means you could just create multiple accounts to work around rate limits, so in practice it doesn't matter if you use multiple accounts, the overall rate limit will still apply for the per-domain limit. However I don't know what logic applies for accounts that have [rate limit] exceptions vs those that don't.

3 Likes

That's the whole point of this thread :slight_smile:

4 Likes

It would mean 300 certificates per registered domain (eTLD+1) within the past week, created by any account.

If an ACME account's adjustment allows it to issue more than (the default) 50 certificates per domain per week, and it has exceeded 50, then other accounts without an adjustment will be rate limited.

There's one more important detail: only "new" certificates count towards this rate limit. A new certificate for the same FQDN won't count.

(I'm very sorry this took so long to answer. I had this bookmarked, but it got buried in my pile of tasks during the holiday rush, and just resurfaced.)

4 Likes

Just to be clear:

Is that "created by all accounts combined"?

will be rate limited "sooner"?
Say one such adjusted account issues more than 50 certs this week, does that still leave 50 for all the other accounts OR zero?

3 Likes

Yes, exactly.

It would leave zero for the other accounts.

4 Likes