Rate limit reached but it seems we only activated 9 certificats this week

Issuance Policy
1

Hello,

We hit the rate limit (urn:ietf:params:acme:error:rateLimited: “too many new orders recently”) but we have created only 9 certificates this week https://tools.letsdebug.net/cert-search?m=domain&q=lifen.fr&d=168

Is there somthing else to check to understand what is happening ?

Is there a way to donate and be allowed to get a higher rate limit ?

Thanks,

Matthieu

2

Hi @matthieulifen

there is a new order limit: 300 new order per Account per 3 hours.

A lot of orders!

This list

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:true;domain:lifen.fr;issuer_uid:4428624498008853827&lu=cert_search

looks buggy. 4 active certificates with CN=grafana.internal.aks-dev.lifen.fr

https://transparencyreport.google.com/https/certificates?cert_search_auth=&cert_search_cert=&cert_search=include_expired:false;include_subdomains:false;domain:grafana.internal.aks-dev.lifen.fr&lu=cert_search

Normally, you should max. have only two active, one older (expires in the next 30 days) and one new certificate.

Which client do you use?

3

Hi Juerden,

Thanks for the links.

If I understand, there is still no explanation of the rate limit we hit.

I’m using cert-manager in a k8s cluster.

4

@jsha, how does this work in ACMEv2? Is there a possibility of hitting a new order rate limit due to a buggy client while only successfully issuing a very small number of certs?

5

Are there other domains? So that this cert-manager creates much more certificates?

4 active certificates - looks like the certificates are not saved (60 - 90 days), instead recreated.

6

Since yesterday, https://tools.letsdebug.net/cert-search?m=domain&q=lifen.fr&d=168 reports now 14 certificates. (perhaps there are more so)

@JuergenAuer for the 4 active certificates for CN=grafana.internal.aks-dev.lifen.fr, it is caused by manual destruction of the previous obtained certificates because recreating the whole k8s cluster without backup. (This should not happened normally)

@schoen about donating to be able to get a higher rate limit, is it something possible or is it out of scope ?

7

You are welcome to donate to this worthy cause, but rate limit increases (including for >300 orders per 3 hours) are granted to anybody with a legitimate need. You can find the application link near the bottom of Rate Limits - Let's Encrypt .

One thing to note is that CT logs do not report on the number of orders, because orders do not necessarily result in a certificate being issued. You should try and figure out why your environment is generating so many ACME v2 orders.

If this is the cause for the spike in orders, then those orders should be spread out over time, and that is probably not a legitimate reason for a rate limit exemption.

8

For users of the ACME v2 API you can create a maximum of 300 New Orders per account per 3 hours.

Are you creating that many orders on purpose?

If you're securing a million domains, you should ask for a rate limit exemption.

If there's a broken client trying every 30 seconds to issue a certificate for a misspelled domain name, you should just fix it...

9

I found the root issue.

We have an old k8s (which is going the be replaced) with kube-logo (version 0.1.4) and I think the rate limit was reached because of this client.

We stopped it (kube-logo) yesterday and today, we can again successfully obtained certificates in our new k8s (using cert-manager).

Thanks all for your time!

1 Like
10

:slightly_smiling_face:

That makes sense – I’m not sure if you’ve seen this, but older versions of kube-lego could retry extremely aggressively.

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.