It produced this output: "too many certificates already issued"
My web server is (include version): nginx/1.19.6
The operating system my web server runs on is (include version): Ubuntu 16.04
My hosting provider, if applicable, is: Azure
I can login to a root shell on my machine (yes or no, or I don't know): Yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Cert manager installed using helm on Kubernetes
I believe I have issued SAN certificates correctly, see crt.sh | 4157895221 , and the Let's encrypt documentation mentions 50 requests per week for a registered domain, which I believe is beelivingsensor.eu. Therefore, I should be capable of requesting 50 certificates, each for 100 subdomains each, e.g. static.beelivingsensor.eu. However, if you search https://crt.sh/ for beelivingsensor.eu and find the time frame of 2021-01-09 to 2021-02-16 (including those days), you can see that I have issued 10 certificates, each for 5 subdomains within one week. The problem is, when I tried to issue new certificates on the next day 2021-02-17, I was met with "too many certificates already issued" error in the logs. So my question is, did I issue the SAN certificates correctly? If yes, how come I was not able to issue any more of them? Right now, I almost have 10 subdomains, does that mean I can ask for valid certificates "only" (oh how spoiled I am ) 5 times a week?
You are probably hitting the "Duplicate Certificate Limit". You can only request 5 of the same exact certificates per week.
You should not need to request more than 1 of these same exact certificates a week, a month, or every two months. You possibly have an issue with your deployment/integration in which you are not keeping and re-using valid certificates.
Yeah that could be possible, although it confused me why it allowed 10 certificates per week, but now that I think about it, they are "Precertificate" whatever that means, so you are probably right.
Yes, I have to say I have bad practices, saving the certificates and then restoring them is way more complicated then creating them with a single command again, plus the first 4 requests I forgot that I am requesting them automatically.
This was an incredible help in such a short time, I will be supporting Let's encrypt community for sure. Many thanks @jvanasco !
Please don't think so lightly with regard to your free certificates. With behaviour like that you're abusing the Let's Encrypt service in my opinion. Even though the certificates might be free for you, it costs Let's Encrypt considerable amounts of money to make that happen. A great part of those costs are in the issuing of the certificates and the corresponding signing of the OCSP statusses (even if you delete a certificate, Let's Encrypt is required to keep signing OCSP responses until the certificate has expired
3 months later). And more issued certificates costs more investments in the issuing infrastructure.
Please store the issued certificates on persistent storage. Thank you.
Precertificates are used by LetsEncrypt to embed the recent Certificate Transparency browser requirements directly into the certificate. There are better descriptions on this site, but basically LE generates a "precertificate", submits it to the transparency log, then embeds the proof in the final certificate that you are issued. So you should see two log entries for every actual certificate that is processed.
@Osiris I am sincerely not sure if you are serious. If you are, then it surely is no laughing matter, and I will look into saving those certificates. I was of the assumption that the only cost would be infrastructure generating the certificates, and since you can create a self signed certificate yourself with a few commands, what would be the cost? Especially if they allow you to request 50(!) per week. That is a huge number, like who would ever use that many from a "free" service? I suppose I will have to read up on what "OCSP statusses" are though.
Let's Encrypt doesn't sign one certificate per week or day or hour, as you would with a simple OpenSSL command. They are signing almost 2 million certificates a day! And the signing isn't done by a simple computer, but expensive Hardware Security Modules. And all the systems around it of course.
Let's Encrypt has relatively low marginal cost for issuing each certificate, which is one reason that they can be issued free of charge to the subscriber. But it's also true that each issued certificate uses some of the capacity of Let's Encrypt's HSM hardware
—mostly because of OCSP signatures (confirming that certificates are still valid) rather than initial issuance.
The Let's Encrypt community's attitude is, approximately, that people are welcome to as many distinct certificates as they need, within Let's Encrypt's rate limits
but that they should only request certificates that they are really expecting to use for a network service. Let's Encrypt's services are only sustainable at a given level of HSM capacity if most people follow this guideline. (If many people routinely issued certificates they didn't use, the rate limits would most likely eventually have to be made more restrictive, which would inconvenience other users.)
Thanks for your willingness to learn about the impacts of different certificate issuance strategies!
A few months ago I suggested that maybe we should invent an HTTP header that communicates that an API service had a non-trivial cost to provide
and then tools that run inside of ephemeral containers, for example, could generate a warning when they saw that header. I'm not sure how well this would work, but it might be worth experimenting with.
The ratelimit of 5 duplicates per week is generous enough to keep a system online when users have errors with initial deployment/integration or a critical error that requires a server to reformat (or has been lost). It is not nearly large enough to handle container and auto-scaling style deployments, and a lot of people run into issues as they scale projects and hit rate-limits.
A good rule of thumb is that your systems should stay under 50% of any given ratelimits, as that will let you recover during disasters. If you are routinely using 50-100% of a ratelimit, a disaster is guaranteed to cause you extended downtime.
I am not highly experienced, but so far as I was running our Kubernetes cluster, I don't remember looking at the request headers or logs too much. As far as I can remember, I installed cert-manager through Helm, then spent quite some time figuring out "why are those certificates not being issued?!" and reading the whole "Rate limits" web page at least 3 times. So if my two cents could help, I would most likely notice the "cost warning" somewhere in the "Rate limits" documentation. Although, I know that the larger the documentation, the less likely is it that people will read it, still two sentences shouldn't hurt.
Experimentation is surely a good idea, and better than nothing.
I can confirm it works. I cannot afford to run automatic deployment with real certificates, users are easily scared of the "Self-signed certificate" warning. So I am using the staging certificates at all times, and only when I know everything is working, I manually change the certificates.
Look into using a gateway/proxy that can keep the certificates persistent. There are also gateways/proxies that will cache a certificate into persistent storage or onto a cloud based bucket.
) 5 times a week?
(If many people routinely issued certificates they didn't use, the rate limits would most likely eventually have to be made more restrictive, which would inconvenience other users.)
and reading the whole "Rate limits" web page at least 3 times. So if my two cents could help, I would most likely notice the "cost warning" somewhere in the "Rate limits" documentation. Although, I know that the larger the documentation, the less likely is it that people will read it, still two sentences shouldn't hurt.