Not understanding which rate limit is being hit

I manage an internal network that has a dns-01 based validation “helper server” for internal users to leverage to obtain LE certs.

In past few days, I’ve stood up several new internal servers individually, but am suddenly unable to obtain any more certs due to this error:

{
"type": "urn:acme:error:rateLimited",
"detail": "Error creating new cert :: Too many certificates already issued for: spirenteng.com",
"status": 429
}

Here’s the problem - I’m not sure which limit I am hitting - cause I haven’t gotten 20 new certificates in past week.

Looking at my local server, I am only seeing 6 new certs - and the rest should all be renewals. Can you help clarify what limit I am hitting and why or if I am misinterpreting the rate limit docs?

root@ub-netmgr:/local/letsencrypt/dehydrated/certs# find . -name "cert-1*.pem" -mtime -10 -ls | sort -r -nk 9 | nl
     1	   926064      0 -rw-------   1 netdb    netdb           0 Dec 17 19:42 ./velocity-ilo-core-ci-v6-1.spirenteng.com/cert-1482003760.pem
     2	   926060      0 -rw-------   1 netdb    netdb           0 Dec 17 19:42 ./velocity-ilo-api-ci-v6-1.spirenteng.com/cert-1482003740.pem
     3	   926057      0 -rw-------   1 netdb    netdb           0 Dec 17 19:39 ./velocity-ilo-api-ci-v6-1.spirenteng.com/cert-1482003548.pem
     4	   926029      0 -rw-------   1 netdb    netdb           0 Dec 16 23:29 ./vel-v60ga-sjqa-ite.spirenteng.com/cert-1481930943.pem
     5	   925978      4 -rw-------   1 netdb    netdb        1850 Dec 16 04:25 ./vel-ilo-core-ci-v6-1.spirenteng.com/cert-1481862336.pem
     6	   925967      4 -rw-------   1 netdb    netdb        1850 Dec 16 04:24 ./ite-ilo-core-ci-v6-1.spirenteng.com/cert-1481862255.pem
     7	   922998      4 -rw-------   1 netdb    netdb        1854 Dec 15 02:08 ./velocity-exec-ci-v6-1.spirenteng.com/cert-1481767712.pem
     8	   919428      4 -rw-------   1 netdb    netdb        1842 Dec 15 02:04 ./ite-exec-ci-v6-1.spirenteng.com/cert-1481767438.pem
     9	   919050      4 -rw-------   1 netdb    netdb        1846 Dec 15 01:57 ./ite-herc-test-v6-1.spirenteng.com/cert-1481767017.pem
    10	   912345      4 -rw-------   1 netdb    netdb        1858 Dec 15 01:56 ./velocity-herc-test-v6-1.spirenteng.com/cert-1481767001.pem
    11	   791335      4 -rw-------   1 netdb    netdb        1826 Dec 15 14:17 ./ite-tctest5.spirenteng.com/cert-1481811464.pem
    12	   912251      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:07 ./ite-tctest.spirenteng.com/cert-1481728022.pem
    13	   791396      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:10 ./ite-tctest4.spirenteng.com/cert-1481728215.pem
    14	   791391      4 -rw-------   1 netdb    netdb        1854 Dec 14 15:08 ./velocity-exec-ci-v6-0.spirenteng.com/cert-1481728075.pem
    15	   791386      4 -rw-------   1 netdb    netdb        1854 Dec 14 15:07 ./velocity-exec-ci-v5-3.spirenteng.com/cert-1481728047.pem
    16	   791381      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:06 ./ite-exec-ci.spirenteng.com/cert-1481728001.pem
    17	   791376      4 -rw-------   1 netdb    netdb        1842 Dec 14 15:05 ./ite-exec-ci-v6-0.spirenteng.com/cert-1481727937.pem
    18	   791371      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:05 ./ilo-tctest5.spirenteng.com/cert-1481727920.pem
    19	   791366      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:05 ./ilo-tctest4.spirenteng.com/cert-1481727907.pem
    20	   791361      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:04 ./ilo-tctest1.spirenteng.com/cert-1481727887.pem
    21	   791356      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:04 ./ilo-tctest2.spirenteng.com/cert-1481727864.pem
    22	   791351      4 -rw-------   1 netdb    netdb        1826 Dec 14 15:00 ./ilo-tctest3.spirenteng.com/cert-1481727611.pem
    23	   791342      0 -rw-------   1 netdb    netdb           0 Dec 14 14:51 ./ilo-tctest2.spirenteng.com/cert-1481727073.pem
    24	   791339      0 -rw-------   1 netdb    netdb           0 Dec 14 14:51 ./ilo-tctest1.spirenteng.com/cert-1481727062.pem
    25	   912217      4 -rw-------   1 netdb    netdb        1850 Dec 12 23:45 ./velocity60a-cal-lab.spirenteng.com/cert-1481586193.pem
    26	   912175      4 -rw-------   1 netdb    netdb        1834 Dec 12 23:40 ./ite60a-cal-lab.spirenteng.com/cert-1481586003.pem
    27	   926013      4 -rw-------   1 netdb    netdb        1826 Dec  8 22:55 ./qaut-vel-04.spirenteng.com/cert-1481237697.pem
    28	   926002      4 -rw-------   1 netdb    netdb        1826 Dec  8 22:54 ./qaut-ite-04.spirenteng.com/cert-1481237596.pem

Of those, these should be new and not renewals - you can see the top four I wasn’t able to obtain:

 1	   926064      0 -rw-------   1 netdb    netdb           0 Dec 17 19:42 ./velocity-ilo-core-ci-v6-1.spirenteng.com/cert-1482003760.pem
 2	   926060      0 -rw-------   1 netdb    netdb           0 Dec 17 19:42 ./velocity-ilo-api-ci-v6-1.spirenteng.com/cert-1482003740.pem
 3	   926057      0 -rw-------   1 netdb    netdb           0 Dec 17 19:39 ./velocity-ilo-api-ci-v6-1.spirenteng.com/cert-1482003548.pem
 4	   926029      0 -rw-------   1 netdb    netdb           0 Dec 16 23:29 ./vel-v60ga-sjqa-ite.spirenteng.com/cert-1481930943.pem
 5	   925978      4 -rw-------   1 netdb    netdb        1850 Dec 16 04:25 ./vel-ilo-core-ci-v6-1.spirenteng.com/cert-1481862336.pem
 6	   925967      4 -rw-------   1 netdb    netdb        1850 Dec 16 04:24 ./ite-ilo-core-ci-v6-1.spirenteng.com/cert-1481862255.pem
 7	   922998      4 -rw-------   1 netdb    netdb        1854 Dec 15 02:08 ./velocity-exec-ci-v6-1.spirenteng.com/cert-1481767712.pem
 8	   919428      4 -rw-------   1 netdb    netdb        1842 Dec 15 02:04 ./ite-exec-ci-v6-1.spirenteng.com/cert-1481767438.pem
 9	   919050      4 -rw-------   1 netdb    netdb        1846 Dec 15 01:57 ./ite-herc-test-v6-1.spirenteng.com/cert-1481767017.pem
10	   912345      4 -rw-------   1 netdb    netdb        1858 Dec 15 01:56 ./velocity-herc-test-v6-1.spirenteng.com/cert-1481767001.pem

From a quick check of google transparency logs … https://www.google.com/transparencyreport

You recently obtained …

Dec 16th
vel-ilo-core-ci-v6-1.spirenteng.com
ite-ilo-core-ci-v6-1.spirenteng.com
Dec 15th
ite-tctest5.spirenteng.com
velocity-exec-ci-v6-1.spirenteng.com
ite-exec-ci-v6-1.spirenteng.com
ite-herc-test-v6-1.spirenteng.com
velocity-herc-test-v6-1.spirenteng.com
Dec 14th
ite-tctest4.spirenteng.com
velocity-exec-ci-v5-3.spirenteng.com
velocity-exec-ci-v6-0.spirenteng.com
ite-tctest.spirenteng.com
ite-exec-ci.spirenteng.com
ite-exec-ci-v6-0.spirenteng.com
ilo-tctest2.spirenteng.com
ilo-tctest4.spirenteng.com
ilo-tctest5.spirenteng.com
ilo-tctest1.spirenteng.com
ilo-tctest3.spirenteng.com
Dec 12th
velocity60a-cal-lab.spirenteng.com
ite60a-cal-lab.spirenteng.com

so looks close on 20 to me

In answer to your questions of limits and “renewals” or “new” … they all count - but renewals aren’t blocked ( if that makes sense )

1 Like

We found out previously that the limits are set up such that instead of renewals not counting against the domain limit they do count, but are only limited by the separate renewal limit.

So disappointingly it matters which order you do things in, you can issue more up until you hit the per-suffix limit (which it seems you have, that’s what the message says Too many certificates already issued for: spirenteng.com and then you can continue to issue “renewals” (ie issuances of certificates exactly matching a set of FQDNs already issued) but you can’t issue any “new” certificates (ie not “renewals”) even if most of the 20 issued already this week were “renewals”.

I find this a bit frustrating too, but I guess it was easier to make it work that way.

2 Likes

Ah, ok. That makes a lot more sense then… That is not at all clear from the Rate Limit documentation.

Thank you.

Sure would be nice if the limits scaled according to the number of certificates already in use for a given domain, though I can see how that could be abused.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.