We’re using certbot on a centralized server to issue and reissue certificates for all the domains of our customers.
After a few weeks the performance on this server became worse and worse and now it takes 20 minutes to issue a single certificate. What we can see is that certbot loops through ALL the certificates on the server whenever it is issuing a new certificate or reissuing an existing one, which also explains why it takes longer and longer the more certificates we have.
Right now there are about 170’000 certificates in the /etc/letsencrypt/live directory. After seeing certbot processes using up a lot of load we followed them with strace, which is how we found out that it’s looping through all the other certificates on the server.
I did not find any information about why this is happening and how to stop it. It could very well be a feature which is supposed to prevent duplicates or something like that as I can’t imagine any other reason why it would search through everything.
The parameters we’re using for certbot usually look as follows:
certonly --webroot -w /srv/www/challenges/example.com -d example.com --non-interactive --email admin@somewhere.com --agree-tos --force-renewal
We’re using an Ubuntu 16.04 system and we’re redirecting http-traffic from our production webservers to this server for validation. This worked without problems for more than 170’000 domains, but if certbot insists on reading through all existing certificates every time we want to get a new one then it won’t even be possible to renew all those domains without adding new ones because that would still take much much longer than 3 months.
Hi @Hoststar! Thanks for posting. Tagging @bmw@erica, who work on Certbot, so they can take a look.
To clarify: Are you hosting for 170,000 certificates, or does that number include past certificates that have been renewed? Also, can you confirm that your certbot certonly commands are the ones that iterate through all 170k, or is it only your certbot renew commands?
If the former, I think this is probably a bug in Certbot, and can probably be fixed. Certbot treats the certificates in /etc/letsencrypt/ as the canonical source for some data, like the list of hostnames you want certs for, and the expiration dates of existing certificates. That means when you ask to renew, Certbot does need to read and parse all your currently active certificates. However, if you are asking for a new certificate, in theory this shouldn’t be necessary, since all Certbot needs to do is check the /etc/letsencrypt/live directories for the specific hostnames you’re asking for.
You also asked separately about rate limiting: The best way to check if you are being rate limited is to look in /var/log/letsencrypt/letsencrypt.log* for messages containing 429 (the status code for rate limited responses).
We are hosting for 170,000 certificates but we still have more to come. We’re not using the certbot renew command at all as far as I know. Renewals as well as new certificates both get created with certbot certonly in our case.
that my guess is right and specifying --duplicate will entirely avoid calling cert_manager.find_duplicative_certs().
@Hoststar, it might not be the best solution, but could you try adding --duplicate to your Certbot command line for a new cert issuance and see if it happens much faster? (The only drawback in this case is that if you do request a certificate that duplicates one that you already have, it will save a duplicative cert instead of updating the old one. --duplicate effectively means "I know that I (might) already have such a cert, but I still want a new one no matter what". So you would probably want to check and take care of that case in some other way.)
For the current version, to renew all certs (as necessary) in O(# certs), use certbot renew. To create a new cert without running through all existing certs, --duplicate will work. Hoping to get a more general fix in for the next release, thanks for bringing this up!
Problem solved! Thanks a lot to everyone involved!
The --duplicate option does seem to bypass this check. Now it takes a few seconds instead of 20 minutes.
It would be nice to have a solution for renewals without duplicating as well, so thanks for including that as an issue on GitHub. Our system already manages renewals, which is why we’re not currently using certbot renew, but I don’t see much of a problem in generating duplicates if it is done at the end of the lifetime of the previous certificate anyway.
But of course we’ll change that as soon as there is a fix for it available.
For what it’s worth, as of Certbot 0.10, there’s an easy command to delete a certificate lineage:
certbot delete --cert-name example.com
certbot delete --cert-name example.com-0001
and so forth. After renewing and duplicating a certificate, and reconfiguring your software, you can use that command to get rid of the old one. Carefully, of course.
But i don’t know if the delete command is fast, and it may not be worth automating when the whole issue might be solved soon anyway.
With https://github.com/certbot/certbot/pull/4128 (slated for 0.11.0), running certbot certonly --webroot -w /srv/www/challenges/example.com --cert-name example.com --non-interactive --email admin@somewhere.com --agree-tos --force-renewal will not iterate through all certificates. Note that selecting a certificate with --cert-name accesses it directly, but -d will still run through all of the certs to find the correct one. Since 0.10.0, you can run certbot certificates to see the cert name of each existing certificate.
This is a nice feature that should help people in the original poster’s situation a lot.
For more casual users I would like to point out that, like --duplicate, this form won’t check whether you already have another certificate (under a distinct cert name) with identical or overlapping name coverage. So if you aren’t keeping track of that yourself in some way, you might get unexpectedly duplicative certificates when using a command like this.
