Basically, we want to set up some alerts that tell us when we’re approaching any Let’s Encrypt limit. We have a bunch of domains, and a bunch more subdomains, and we’ve hit some limits in the past, and want to have advance notice if we’re about to hit limits in the future.
Is there a general article about this? How to hook it up to Prometheus or DataDog or some other third party service would be ideal, but just solid technical detail of an API or something is enough for us to do that final hook up.
There is this old thread here: Better Notifications of when Users Approach Rate Limits which seems close to what I’m talking about, but it seems to’ve closed out without any resolution, and I’m not sure this is completely relevant.
Hi, we have put this on back-burner as there was not much of a follow-up.
Having said that - KeyChest now has a pure TCP API, which can be called from Linux shells without any dependencies. This could be a suitable vehicle to implement counters that could be used to count calls and check against limits.
KeyChest is free for up to 500 domains so it should be ok-ish for most users.
If interested, we can put together an API spec and do it. Looking at limits - https://keychest.net/content/letsencrypt_numbers_to_know, we’d need to log LE calls per account and separately measure validations and number of issued certs. The API would need to:
measure - increment use counters
answer usage requests - this needs some input from you - percentage, number left, and/or time when new requests are allowed, etc
Is this a product advertisement? Doesn’t seem relevant to the discussion.
I will answer assuming you’re a legitimate technical professional.
We already have a method of adding counters of our own, but are concerned someone might interact with LE outside of the library of counters, and therefore we’ll hit a limit we weren’t expecting.
Even if it might be every 24hr, it would be nice to reconcile our numbers with LE’s numbers, so we won’t be (as) surprised when that happens.
I think your best bet is probably to create a Prometheus exporter that occasionally polls something like the CertSpotter API (https://sslmate.com/certspotter/api/pricing)[1].
It’s hard to tell if the exact limit is approaching because you need to exclude renewals, meaning you have to look potentially at 180+ days of history.
1. You could also use free stuff like crt.sh, but a commercial one like CertSpotter or Censys might be a bit more reliable and faster. You can look at my (pretty old) approach of calculating rate limits using crt.sh’s database as well. It predates the renewal exemption so it’s not entirely accurate, but it’s good enough for a Prometheus exporter. I do think that CertSpotter would be superior though.
Hi @philovivero, you tagged me in your post so I responded the best I could. If you want to discuss more offline, just drop me a line. I’d be interested how much time you spent developing your solution.
Looking at your own response. What you ask for is an API on the CA side. I can’t talk directly for LE folks but as far as I understand, that is pretty unrealistic ask. Implementing it would mean building a new API that would access databases deep inside the LE CA/RA systems … with queries involving large numbers of data points. So there are many risks there.
If you already have your own counters, the best you can do is to create a fork on your favourite client, add there a wrapper to enforce updates of your counters.
To check that no-one is bypassing your policy, you could wrapped your account secret with a named pipe and correlated number of reads with the number of your counter updates.
PS: looking at other responses - we have own CT lookup database, which we use to find new certs for all subdomains
