Typical VPS with a Xeon CPU from not-this-generation.
Certbot used less than 8 seconds of CPU time.
As I wrote, I didn't use the Nginx plugin. (Or --allow-subset-of-names.)
After combining/ordering those log files, this is what stood out to me:
5 minute interval before generating backups of nginx config (a couple of times)
2018-08-02 08:06:34,912:INFO:certbot.auth_handler:http-01 challenge for www.houstonclassified.com
2018-08-02 08:06:55,004:DEBUG:certbot.crypto_util:Generating key (1024 bits): /var/lib/letsencrypt/snakeoil/0320_key.pem
2018-08-02 08:11:23,592:DEBUG:certbot.reverter:Creating backup of /etc/nginx/config-templates/common
and
2018-08-02 08:23:09,558:INFO:certbot.auth_handler:http-01 challenge for www.houstonclassified.com
2018-08-02 08:23:27,915:DEBUG:certbot.crypto_util:Generating key (1024 bits): /var/lib/letsencrypt/snakeoil/0321_key.pem
2018-08-02 08:27:46,006:DEBUG:certbot.reverter:Creating backup of /etc/nginx/config-templates/common
10 minute interval writing nginx config
2018-08-02 08:11:24,432:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/sites-enabled/toughdomains.conf:
2018-08-02 08:21:45,097:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:
and
2018-08-02 08:27:46,014:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:
2018-08-02 08:27:46,854:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/sites-enabled/toughdomains.conf:
2018-08-02 08:37:51,843:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:
Given that performance problems have been reported with libaugeas before and that @mnordhoff gets good performance when avoiding involving augeas, nginx pluginâs usage of libaugeas is probably the problem.
In the first case, could the problem be generating lots of snakeoil RSA keys?
Er, you might be right. The reason I dismissed it was because of a misread when Certbot got ^câd.
But I think itâs more likely that Augeus is doing unlogged work at that stage - we can see that the next operation always involves nginx config.
There is some evidence to indicate that key generation isnât to blame, and thatâs when the next operation isnât anything to do with nginx:
2018-08-02 09:07:47,902:INFO:certbot.main:Obtaining a new certificate
2018-08-02 09:07:48,190:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0323_key-certbot.pem
2018-08-02 09:07:48,197:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0323_csr-certbot.pemFWIW the nginx authenticator/configurator doesnât use augeas, only the Apache one does. certbot has its own parser. @erica corrected me on this a few months ago when a visitor found a bug in it. 
Ahh, that's helpful.
I tried the same parameters as the OP (75 vhosts x 100 aliases), and Certbot 0.26.1 + nginx plugin issues and deploys certificates in a very reasonable timeframe. Parsing of such a config file takes only 4 seconds with the pyparsing-based parser. So much for that theory!
@timkou, I guess @_azâs experience and contrast with your situation represents good news and bad news:
- In theory it should be possible for your configuration to have much faster certificate issuance.
- But we donât know much at all about why your configuration is slow.
Would you be willing to modify your copy of Certbot to save more profiling information to help narrow down which actions may be slow?
1 Like
@timkou, my colleague who created a lot of Certbotâs nginx support mentioned that your configuration file is extremely large, and she wonders how long it takes to restart nginx even without Certbot when using this file. (This might also be related because some Certbot functions internally tell nginx to reload its configuration, which might be a time-consuming task if nginx itself canât parse this file very quickly.)
@schoen thank you for all your helpful input.
We have certs for about 18k domains, one cert for 100 names, for some reason some names in some certs not working.
As we use --redirect certbot write to nginx conf to redirect this domain to https and it trigger error in browser. Renew do not solve this problem. Is any way to solve it without knowing broken domain names?
Can you give an example?
Here are several examples:
https://bermudashort.com/
https://armakeup.com/
https://bestbodydiet.com/
Please advise
And do you have an example of the command that you ran to create that certificate?
You arenât running Certbot with --allow-subset-of-names, are you?
@schoen here is an example of the command used to run a certificate.
/root/certbot-auto --quiet --nginx --redirect --allow-subset-of-names -w /home/toughdomains/application/current/web/ -d downs.in,www.dog-training.ca,doginstructions.com,dog-training.ca,www.dominicanjob.com,drafterjob.com,www.dogrescue.ca,www.downloadjobs.com,doll.info,domaindevelopersfund.com,www.dogfood.info,dogtraining.info,www.downloadsongfree.com,www.domescout.com,www.dolphins.co.in,www.domainclubhouse.com,www.dogz.biz,domainers.biz,www.doggiedoor.ca,www.downloads.biz,dogschool.biz,downloadsongfree.com,www.drains.co.in,dovexo.com,donatewedding.com,dogz.biz,www.donalds.top,domainclubhouse.com,www.downloadgames.app,www.doughnut.co.in,domainnamesbucket.com,downloadgames.app,dormer.space,domainstore.ca,downloads.biz,www.domainers.biz,www.dorval.info,donateformywedding.com,dorval.info,dogschool.ca,www.dogschool.biz,drawingar.com,downloadingmusicfree.com,domaining.place,doggiedoors.ca,doughnut.co.in,www.dormer.space,domain4sale.ca,domainbuyrent.com,donutstreet.com,www.domainstore.ca,www.donuts.co.in,www.domainbuyrent.com,donalds.top,www.downtownjob.com,dogrescue.ca,www.dogtraining.info,www.doll.info,domescout.com,dolphins.co.in,www.domaining.place,www.dogbreed.info,www.donateformywedding.com,www.drafterjob.com,dowoa.com,www.dominostore.com,www.downloadingmusicfree.com,dominostore.com,www.downs.in,www.donutstreet.com,www.dongguanjobs.com,downloadjobs.com,www.domstrahovka.com,www.dragonballsuper.ca,dogbreed.info,dogfood.info,dogschool.info,dom-strahovka.com,www.dovexo.com,www.doughnuts.co.in,doughnuts.co.in,www.drawingar.com,dominicanjob.com,www.domaindevelopersfund.com,drains.co.in,donuts.co.in,dongguanjobs.com,doggiedoor.ca,www.dowoa.com,www.dom-strahovka.com,domstrahovka.com,www.domainnamesbucket.com,dragonballsuper.ca,www.donatewedding.com,www.domain4sale.ca,downtownjob.com,www.dogschool.info,www.doginstructions.com,www.dogschool.ca,www.doggiedoors.ca
It's because of your use of --allow-subset-of-names. This tells Certbot to remove names from the certificate (instead of aborting the effort to obtain the certificate) if they fail at the time that the certificate request is made. This has the advantage that one failing domain won't prevent a renewal of a much larger certificate, but the disadvantage that the failing domains will be removed from the certificate. This could happen even in the case of an ephemeral or momentary failure, like if a DNS server fails to respond to a particular DNS lookup request.
To avoid this problem, you could stop using --allow-subset-of-names. This means that if there are temporary or transitory failures, they'll block the issuance or renewal of associated certificates until they're resolved or clear up.
There isn't an easy way around this with Certbot because a certificate must either be issued or not issued, and it can't be issued for a certificate that the CA couldn't validate. With a custom client, it might be possible to work around this by keeping a list of desired domains and opportunistically requesting new certificates periodically for those that had failed renewal during the most recent renewal attempt. Unfortunately, Certbot doesn't include any functionality for this.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.