The operating system my web server runs on is: CentOS 7
I can login to a root shell on my machine: yes
The version of my client is: certbot 0.31.0
I’ve already checked both letsdebug.net and check-your-website.server-daten.de . The former shows a big green “All OK” banner (though details shows “Failed to query certwatch database to check rate limits: pq: canceling statement due to user request”); and the later shows no recent certificates. What’s going on here?
Queries that take longer than 10 seconds get cancelled so that we don’t get in trouble with the crt.sh operator. The query for ibm.com takes a very long time due to the relatively large number of certificates it has to scan.
Does IBM have a rate limit exemption with Let’s Encrypt?
Censys says you have 378-ish certificates in the last week, but you have to subtract which ones were renewals of previous certificates in order to figure out the rate limit, which is a bit tricky for everybody except Let’s Encrypt.
Ah, I didn’t realize there were multiple “Source” sections in that report; I now see that the second source (crt.sh) is returning them. I forgot to mention that I’d searched crt.sh manually and also came up dry. I guess even manual crt.sh queries hit that timeout. When will the rate limit expire? The “next LE” column is completely blank as far as I can see, but given that the most recent cert was issued 2019-09-29 16:41:43, I’m guessing the limit will expire on the 6th at about 16:41:44?
Given our size, it wouldn’t surprise me if we have a limit exemption - but, given our size, it wouldn’t surprise me if everyone says “Oh, I’m sure we have one already.” That being said, it looks like some automated process has gone nuts requesting all those certs, so I don’t know if any limit would be enough …
Renewals are treated specially: they don’t count against your Certificates per Registered Domain limit, but they are subject to a Duplicate Certificate limit of 5 per week. Note: renewals used to count against your Certificate per Registered Domain limit until March 2019, but they don’t anymore.
Unless you perfectly know the historical lineage of a certificate (i.e. by looking in the Let’s Encrypt database which has an optimized index for it), finding the true state of the rate limit state takes a lot of expensive queries against a traditional log aggregator, probably to the point where it’s infeasible to do casually. Especially if you have 300+ certificates being renewed per week.
I wonder if there’s someone at IBM who handles coordination of things like this (like a core DNS operations or DNS policy group, perhaps?) and who might be willing and able to keep track of all of the divisions, offices, or contractors that may be communicating separately with Let’s Encrypt about possible rate limit exemptions.
I figured out how to use the api at certspotter to download all the transparency records, and it looks like this certificate-manager is creating/renewing roughly 70 records every day. I’m suspecting that whatever LE account it’s using has an exemption that doesn’t apply to the whole domain … is that possible?
I’m still trying to track down whoever runs this certificate-manager thing.
Well, I believe you can have exemptions that only apply to issuance by a particular account, so it should be possible that someone else who successfully got an exemption is performing issuance that’s causing trouble for you.