Too many certificates already issued, but Let's Debug is green

My domain is: *.k8s.sec.ibm.com

I ran this command: certbot --preferred-challenges dns --manual -d "*.k8s.sec.ibm.com"

It produced this output:

Plugins selected: Authenticator manual, Installer None
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Obtaining a new certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for: ibm.com: see https://letsencrypt.org/docs/rate-limits/

The operating system my web server runs on is: CentOS 7

I can login to a root shell on my machine: yes

The version of my client is: certbot 0.31.0
________

I've already checked both letsdebug.net and check-your-website.server-daten.de . The former shows a big green "All OK" banner (though details shows "Failed to query certwatch database to check rate limits: pq: canceling statement due to user request"); and the later shows no recent certificates. What's going on here?

Queries that take longer than 10 seconds get cancelled so that we don’t get in trouble with the crt.sh operator. The query for ibm.com takes a very long time due to the relatively large number of certificates it has to scan.

Does IBM have a rate limit exemption with Let’s Encrypt?

Censys says you have 378-ish certificates in the last week, but you have to subtract which ones were renewals of previous certificates in order to figure out the rate limit, which is a bit tricky for everybody except Let’s Encrypt.

Hi @aabraham

I don't see a check. Not with k8s.sec.ibm.com, not with sec.ibm.com, a ibm.com check is running.

PS: Now the check is ready.

337 Letsencrypt certificates in the last 7 days, 2893 active, 4059 complete.

Ah, I didn’t realize there were multiple “Source” sections in that report; I now see that the second source (crt.sh) is returning them. I forgot to mention that I’d searched crt.sh manually and also came up dry. I guess even manual crt.sh queries hit that timeout. When will the rate limit expire? The “next LE” column is completely blank as far as I can see, but given that the most recent cert was issued 2019-09-29 16:41:43, I’m guessing the limit will expire on the 6th at about 16:41:44?

Given our size, it wouldn’t surprise me if we have a limit exemption - but, given our size, it wouldn’t surprise me if everyone says “Oh, I’m sure we have one already.” That being said, it looks like some automated process has gone nuts requesting all those certs, so I don’t know if any limit would be enough …

It's currently hard to tell because of the renewal exemption (https://letsencrypt.org/docs/rate-limits/):

Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week. Note: renewals used to count against your Certificate per Registered Domain limit until March 2019, but they don’t anymore.

Unless you perfectly know the historical lineage of a certificate (i.e. by looking in the Let's Encrypt database which has an optimized index for it), finding the true state of the rate limit state takes a lot of expensive queries against a traditional log aggregator, probably to the point where it's infeasible to do casually. Especially if you have 300+ certificates being renewed per week.

:man_shrugging:

I wonder if there’s someone at IBM who handles coordination of things like this (like a core DNS operations or DNS policy group, perhaps?) and who might be willing and able to keep track of all of the divisions, offices, or contractors that may be communicating separately with Let’s Encrypt about possible rate limit exemptions.

Then "next LE" shows only a result if the "identical list of domain names" limit is hitted.

I've added something new. Now new Letsencrypt certificates (first combination of the list of domain names) are marked, there is a new info:

last 7 days
264 /187 new

And the list is really curious. One sample:

|2019-09-29 16:41:16|2019-12-28 17:41:16|2992019312.au-syd.e2e.certificate-manager.test.cloud.ibm.com

1 entries duplicate nr. 3

|2019-09-29 16:37:02|2019-12-28 17:37:02|2992019312.au-syd.e2e.certificate-manager.test.cloud.ibm.com

1 entries duplicate nr. 2
2019-09-29 16:33:43 2019-12-28 17:33:43
1 entries duplicate nr. 1
--- ---

The same certificate, first seen 16:33:43, next 16:37:02, next 16:41:16.

Looks like there are some tests with random certificate names, so the limit exemption is used.

I figured out how to use the api at certspotter to download all the transparency records, and it looks like this certificate-manager is creating/renewing roughly 70 records every day. I’m suspecting that whatever LE account it’s using has an exemption that doesn’t apply to the whole domain … is that possible?

I’m still trying to track down whoever runs this certificate-manager thing.

Well, I believe you can have exemptions that only apply to issuance by a particular account, so it should be possible that someone else who successfully got an exemption is performing issuance that’s causing trouble for you.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.