Rate limit reached based on the error message, but logs says different

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: a465-9q4k.cloudera.site

I ran this command:

We use golang lego lib (GitHub - go-acme/lego: Let's Encrypt client and ACME library written in Go) to request certificate. And basically we use similar code that can be found in the module example part. We use dns01 challenge.

It produced this output:

urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for: a465-9q4k.cloudera.site: see Rate Limits - Let's Encrypt

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): We are using golang lego lib.

We want to request a certificate for domain: *.dw-vslaadobetest.a465-9q4k.cloudera.site
And we got a response about "acme: error: 429 :: POST :: https://acme-v02.api.letsencrypt.org/acme/new-order :: urn:ietf:params:acme:error:rateLimited :: Error creating new order :: too many certificates already issued for: a465-9q4k.cloudera.site: see Rate Limits - Let's Encrypt,
Our problem is that based on the logs on our side, we didn't request that many certificate and we checked crt.sh and it says the same.
So based on all of our information we didn't hit the limit.

Can you help us figuring out how many cert did we request? When can we request new certificate? How can we going forward?

2

Update: See below for better answers

I think you are getting the:
"The main limit is Certificates per Registered Domain (50 per week). "
described here:

I received errors from crt.sh trying to lookup cloudera.site. I once got an error from crt.sh even with a465-9q4k.cloudera.site but one succeeded and it showed very many subdomain names of that were issued recently.

You will need to take this up with cloudera people. They may need to request an exception to this rate limit from Lets Encrypt since they are a shared system.

3 Likes
3

It's quite hard to determine this, because your site has so many certificates issued, crt.sh and similar sites have trouble showing all of the certs..

I think what is happening here is that you're reaching the 50 certificates per domain per week limit. The domain cloudera.site is currently listed in the public suffix list as *.cloudera.site, which means any subdomain of cloudera.site is viewed as if it was a public suffix. However, before 1 september 2021 it was listed just as cloudera.site, i.e., without the wildcard!

This recent update has just been implemented into Boulder, the software for Let's Encrypt, yesterday in this update: Upgrade dependency weppos/publicsuffix-go (#5769) · letsencrypt/boulder@9d07942 · GitHub

So until this change is live in production, all subdomains of cloudera.site will be limited by the 50 certs per week per domain rate limit. When this update is actually live, this rate limit will be shifted towards sub-subdomains. I.e., you can have 50 certs per week for foo.cloudera.site and also 50 certs per week for bar.cloudera.site. I think Boulder updates go live once every two weeks? Not sure about that tho.. In any case, the update is already processed, so the only thing you can do now is wait..

3 Likes
4

Nice catch on the dates of the PSL updates, @Osiris .

I had checked and only saw the *.cloudera.site entry there, which made me wonder if *.*.cloudera.site was somehow not covered. The dates of inclusion you found are more likely the culprit.

2 Likes
5

Thank you for the detailed explanation and fast help, I really appreciate it, and totally makes sense.
Do you know by any chance if I can track somewhere when it goes live? Or do you know who/where can I ask about Boulder release state?

Thank you again.

3 Likes
6

I also saw that initially and thought: hmm, does Boulder even process those wildcards in a proper way? So I made a little test Go Playground - The Go Programming Language :rofl:

I was really confused when I saw this test gave the correct output.. But the playground obviously uses a recent version of the publicsuffix library. And unfortunately I can't use github.com/weppos/publicsuffix-go/publicsuffix@b19572c4b3c21f315a45ce50d6f15a83d4b2ed12 (the older commit before the change) in Go Playground..

3 Likes
7

You could "poll" (i.e.: regularly visit) https://acme-v02.api.letsencrypt.org/build which gives the build date/time and git commit of the currently live Boulder. You can also check the history at https://letsencrypt.status.io/ which includes Boulder release updates. (Seems to be weekly by the way, looking at the dates of the releases..)

4 Likes
8

Many thanks.
Will check the pages and "poll" :slight_smile:

2 Likes
9

Hello,

I am from Cloudera. Do you know how can we request an exception if the 50 cert / subdomain is not enough?

10

There is a link to the Rate Limiting Form in the Overrides section:

@Osiris and @jvanasco know more of this than I do so they may have more to say

2 Likes
11

THAT Rate Limit does not apply to domains in the Public Suffix List. The issue should be solved when LetsEncrypt deploys the current build to the cloud, as this PR against the PSL should fix your issue: Updating Cloudera domains by kedar031 · Pull Request #1400 · publicsuffix/list · GitHub

4 Likes
12

They might want to issue more than 50 certificates per week per sub-sub-domain.

4 Likes
13

Yeah as @Osiris mentioned, we might need more than 50 cert / sub-sub domain. But first let's see how many certs we request / sub-sub domain and we will request if needs.

Thanks both of you

1 Like
14

Note that it may take multiple weeks before your rate limit exception request will be processed. It's a manual process and can take a long time, as Let's Encrypt employees don't always have the time for it.

3 Likes
15

I'm confused about that statement.
Do you mean that PSL rate limit exceptions won't apply to requests on sub-sub-domains of PSL listed domains?

As an example of my query:
PSL = z.example.com
And new cert requests are to:
{MANY}.x.y.z.example.com

1 Like
16

Uch, now I'm confused too. Especially because the PSL is now including a wildcard cert. Let's call the current included PSL entry *.example.com.

Then every subdomain would be a "hit" for the PSL and every subdomain would be an "eTLD" (effective TLD):

  • foo.example.com would be an eTLD.

So:

  • bar.foo.example.com would be seen as a single domain with regards to the LE rate limits.

Therefore you can have:

  • 1.bar.foo.example.com
  • 2.bar.foo.example.com
  • ...
  • 50.bar.foo.example.com

But not:

  • 51.bar.foo.example.com

(within 7 days)

1 Like
17

Hm... I also a little confused.

Is that mean we can have 50 subdomain (like appletree.cloudera.site) cert request a week and 50 cert request per subdomain per week (like lemontree1.appletree.cloudera.site, or lemontree2.appletree.cludera.site)?

18

That sounds very limiting.
I would think that the leading "*." would be implied and just ignored in the rate limit implementation; and treated more like an "ends with".
Time to dig through the code...

1 Like
19

No, add an extra subdomain and then you're correct.

1 Like
20

No, the wildcard adds another subdomain to the picture.

So not just 50 certs for foo.example.com, but 50 certs per bar.foo.example.com

1 Like