~4 Minute delay while issuing certificates

rreess · April 4, 2019, 6:11pm

I’ve got a vhost setup with a few thousand sites on it, on a CentOS 7 machine running Apache 2.4.6 + Certbot 0.31.0. I’m in the process of issuing certs via Certbot + a cron job, which runs a command like this:

certbot --apache certonly -d jassonbautista.xyz -d www.jassonbautista.xyz

The first few certs went fast, but now that I’m a few hundred sites deep, things were moving pretty slowly (~7-8 minutes per cert).

I disabled IPv6, and that cut the times to around 4 minutes, but there’s still a very consistent 4 minute delay at a specific step in the process, between these two steps in the command output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
[~4 Minute delay]
Plugins selected: Authenticator apache, Installer apache

And between these two steps mentioned in log:

2019-04-04 12:23:45,810:DEBUG:certbot_apache.configurator:Apache version is 2.4.6
[~4 Minute delay]
2019-04-04 12:27:48,633:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache

Once it gets to the latter step, the process only takes a few seconds, but there seems to be some kind of timeout between these steps?

Here’s a bit more of the log. I’m not the most experienced backend dev, so I’m sure I’m missing some information here. Any help or direction would be greatly appreciated.

2019-04-04 12:23:45,005:DEBUG:certbot.main:certbot version: 0.31.0
2019-04-04 12:23:45,005:DEBUG:certbot.main:Arguments: ['--apache', '-q', '-d', 'jassonbautista.xyz', '-d', 'www.jassonbautista.xyz']
2019-04-04 12:23:45,005:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-04-04 12:23:45,042:DEBUG:certbot.log:Root logging level set at 30
2019-04-04 12:23:45,042:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-04-04 12:23:45,043:DEBUG:certbot.plugins.selection:Requested authenticator apache and installer apache
2019-04-04 12:23:45,810:DEBUG:certbot_apache.configurator:Apache version is 2.4.6
2019-04-04 12:27:48,633:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7f7c3170a850>
Prep: True
2019-04-04 12:27:48,634:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7f7c3170a850>
Prep: True
2019-04-04 12:27:48,635:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_apache.override_centos.CentOSConfigurator object at 0x7f7c3170a850> and installer <certbot_apache.override_centos.CentOSConfigurator object at 0x7f7c3170a850>
2019-04-04 12:27:48,635:INFO:certbot.plugins.selection:Plugins selected: Authenticator apache, Installer apache
2019-04-04 12:27:48,674:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, terms_of_service_agreed=None, agreement=None, only_return_existing=None, contact=(), key=None, external_account_binding=None), uri=u'https://acme-v02.api.letsencrypt.org/acme/acct/51249897', new_authzr_uri=None, terms_of_service=None), a247801cefefd4727a427c22189decb6, Meta(creation_host=u'localhost.localdomain', creation_dt=datetime.datetime(2019, 2, 10, 22, 39, 49, tzinfo=<UTC>)))>
2019-04-04 12:27:48,690:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-04-04 12:27:48,698:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2019-04-04 12:27:48,952:DEBUG:requests.packages.urllib3.connectionpool:"GET /directory HTTP/1.1" 200 658
2019-04-04 12:27:48,953:DEBUG:acme.client:Received response:
HTTP 200

...

JuergenAuer · April 4, 2019, 6:18pm

Hi @rreess

that's curious.

2019-04-02 11:49:15,507:DEBUG:certbot_apache.configurator:Apache version is 2.4.38
2019-04-02 11:55:03,705:DEBUG:certbot.plugins.selection:Single candidate plugin: * apache
Description: Apache Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_centos.CentOSConfigurator object at 0x7fe1ab0307d0>
Prep: True

6 minutes, two days earlier, another user.

schoen · April 4, 2019, 6:19pm

Hi @rreess,

I think this is similar to another problem recently reported by another user who had a very large Apache configuration. It may just be that Certbot’s current Apache integration is inherently slow on sites with extremely large numbers of virtual hosts due to limitations in the parser.

One option would be to use --webroot (which doesn’t require parsing your Apache configuration at all).

Another option would be to ensure that your Apache virtual hosts are all broken out into individual files and that they don’t unnecessarily use conditional directives.

Can you think of any other useful options, @joohoi?

rreess · April 4, 2019, 6:34pm

Hmmm, each vhost config is indeed broken out into individual files without conditionals, but it does use a name-based vhost config, so maybe that’s incurring a significant parsing cost?

I’ll look into switching to --webroot, that sounds like the way I should have set things up to begin with…

rreess · April 4, 2019, 6:58pm

Good lordy, --webroot works beautifully. Problem solved. Thanks so much, @schoen. I can’t afford much at the moment, but just started a monthly donation. Really appreciate what you guys do, and the absolutely insane quality of help you provide <3

schoen · April 4, 2019, 7:14pm

Thanks, I really appreciate that!

Both ISRG (which operates the Let's Encrypt CA) and EFF (which develops Certbot) can use donations; whichever one you've donated to, if you get more spare money in the future, please consider also donating to the other one.

system · May 4, 2019, 7:14pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.