We’re using certbot on a centralized server to issue and reissue certificates for all the domains of our customers.
After a few weeks the performance on this server became worse and worse and now it takes 20 minutes to issue a single certificate. What we can see is that certbot loops through ALL the certificates on the server whenever it is issuing a new certificate or reissuing an existing one, which also explains why it takes longer and longer the more certificates we have.
Right now there are about 170’000 certificates in the /etc/letsencrypt/live directory. After seeing certbot processes using up a lot of load we followed them with strace, which is how we found out that it’s looping through all the other certificates on the server.
I did not find any information about why this is happening and how to stop it. It could very well be a feature which is supposed to prevent duplicates or something like that as I can’t imagine any other reason why it would search through everything.
The parameters we’re using for certbot usually look as follows:
certonly --webroot -w /srv/www/challenges/example.com -d example.com --non-interactive --email firstname.lastname@example.org --agree-tos --force-renewal
We’re using an Ubuntu 16.04 system and we’re redirecting http-traffic from our production webservers to this server for validation. This worked without problems for more than 170’000 domains, but if certbot insists on reading through all existing certificates every time we want to get a new one then it won’t even be possible to renew all those domains without adding new ones because that would still take much much longer than 3 months.