Certbot crashes Nginx while renewing certificates

Hello,

I'm having issues with that Certbot crashes Nginx while it does the renewal process when it attempts to reload the Nginx webserver. See attached logs for additonal information.

Distro: Debian Buster (unstable)
Cerbot version: 0.24.0-2
Nginx version: 1.13.12

Logs:

May 30 01:43:51 vmi162505 systemd[1]: Starting Certbot...
May 30 01:43:53 vmi162505 systemd[1]: Stopping A high performance web server and a reverse proxy server...
May 30 01:43:54 vmi162505 systemd[1]: Stopped A high performance web server and a reverse proxy server.
May 30 01:43:58 vmi162505 certbot[6696]: nginx: [error] open() "/var/run/nginx.pid" failed (2: No such file or directory)
May 30 01:44:00 vmi162505 systemd[1]: Starting A high performance web server and a reverse proxy server...
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
May 30 01:44:00 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
May 30 01:44:01 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
May 30 01:44:02 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:80 failed (98: Address already in use)
May 30 01:44:02 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
May 30 01:44:02 vmi162505 nginx[6758]: nginx: [emerg] bind() to 0.0.0.0:443 failed (98: Address already in use)
May 30 01:44:02 vmi162505 nginx[6758]: nginx: [emerg] bind() to [::]:443 failed (98: Address already in use)
May 30 01:44:02 vmi162505 nginx[6758]: nginx: [emerg] still could not bind()
May 30 01:44:02 vmi162505 systemd[1]: nginx.service: Control process exited, code=exited status=1
May 30 01:44:02 vmi162505 systemd[1]: nginx.service: Failed with result 'exit-code'.
May 30 01:44:02 vmi162505 systemd[1]: Failed to start A high performance web server and a reverse proxy server.
May 30 01:44:02 vmi162505 certbot[6696]: Hook command "service nginx start" returned error code 1
May 30 01:44:02 vmi162505 certbot[6696]: Error output from service:
May 30 01:44:02 vmi162505 certbot[6696]: Job for nginx.service failed because the control process exited with error code.
May 30 01:44:02 vmi162505 certbot[6696]: See "systemctl status nginx.service" and "journalctl -xe" for details.

Could you also please paste the log from /var/log/letsencrypt/ that correlates with that systemd log ?

Renewal parameters from /etc/letsencrypt/renewal would also help explain what Certbot is trying to do.

This indicates that some of your renewal parameters use pre and post hooks to stop and start nginx, which can wreak havoc on other renewals if they are configured with --nginx or --webroot.

There was some issues in the past with the authenticator, which made me use the pre-hook commands as it basically said it was unsupported (Solution: Client with the currently selected authenticator does not support any combination of challenges that will satisfy the CA):

2018-05-30 01:43:52,296:DEBUG:certbot.main:certbot version: 0.24.0
2018-05-30 01:43:52,297:DEBUG:certbot.main:Arguments: ['-q']
2018-05-30 01:43:52,297:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-05-30 01:43:52,314:DEBUG:certbot.log:Root logging level set at 30
2018-05-30 01:43:52,314:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-05-30 01:43:52,342:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7fce0bf29940> and installer <certbot.cli._Default object at 0x7fce0bf29940>
2018-05-30 01:43:52,358:INFO:certbot.renewal:Cert not yet due for renewal
2018-05-30 01:43:52,365:INFO:certbot.renewal:Cert not yet due for renewal
2018-05-30 01:43:52,370:INFO:certbot.renewal:Cert not yet due for renewal
2018-05-30 01:43:52,375:INFO:certbot.renewal:Cert not yet due for renewal
2018-05-30 01:43:52,381:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2018-06-28 22:01:55 UTC.
2018-05-30 01:43:52,381:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2018-05-30 01:43:52,381:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer nginx
2018-05-30 01:43:52,879:DEBUG:certbot.plugins.selection:Single candidate plugin: * nginx
2018-05-30 01:43:53,079:DEBUG:certbot.plugins.selection:Single candidate plugin: * standalone
2018-05-30 01:43:53,079:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7fce0bf4f6a0> and installer <certbot_nginx.configurator.NginxConfigurator object at 0x7fce0bf29a58>
2018-05-30 01:43:53,079:INFO:certbot.plugins.selection:Plugins selected: Authenticator standalone, Installer nginx
2018-05-30 01:43:53,083:DEBUG:certbot.main:Picked account: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:53,085:DEBUG:acme.client:Sending GET request to https://acme-v01.api.letsencrypt.org/directory.
2018-05-30 01:43:53,120:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
2018-05-30 01:43:53,370:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2018-05-30 01:43:53,371:DEBUG:acme.client:Received response:
2018-05-30 01:43:53,372:INFO:certbot.hooks:Running pre-hook command: service nginx stop
2018-05-30 01:43:54,218:INFO:certbot.main:Renewing an existing certificate
2018-05-30 01:43:54,306:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0009_key-certbot.pem
2018-05-30 01:43:54,308:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0009_csr-certbot.pem
2018-05-30 01:43:54,309:DEBUG:acme.client:Requesting fresh nonce
2018-05-30 01:43:54,309:DEBUG:acme.client:Sending HEAD request to https://acme-v01.api.letsencrypt.org/acme/new-authz.
2018-05-30 01:43:54,483:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "HEAD /acme/new-authz HTTP/1.1" 405 0
2018-05-30 01:43:54,484:DEBUG:acme.client:Received response:
2018-05-30 01:43:54,484:DEBUG:acme.client:Storing nonce: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:54,484:DEBUG:acme.client:JWS payload:
2018-05-30 01:43:54,488:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-authz:
2018-05-30 01:43:54,676:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/new-authz HTTP/1.1" 201 993
2018-05-30 01:43:54,676:DEBUG:acme.client:Received response:
2018-05-30 01:43:54,676:DEBUG:acme.client:Storing nonce: 6d9YMxIJ5PQUSptIotEOJEVYXcXOafyGI_PRjJ56tXg
2018-05-30 01:43:54,677:INFO:certbot.auth_handler:Performing the following challenges:
2018-05-30 01:43:54,678:INFO:certbot.auth_handler:tls-sni-01 challenge for xxxxxxxxx
2018-05-30 01:43:54,678:DEBUG:acme.standalone:Failed to bind to :443 using IPv4
2018-05-30 01:43:54,686:INFO:certbot.auth_handler:Waiting for verification...
2018-05-30 01:43:54,686:DEBUG:acme.client:JWS payload:
2018-05-30 01:43:54,689:DEBUG:acme.client:Sending POST request to xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:54,874:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx" 202 339
2018-05-30 01:43:54,875:DEBUG:acme.client:Received response:
2018-05-30 01:43:54,875:DEBUG:acme.client:Storing nonce: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:55,525:DEBUG:acme.crypto_util:Performing handshake with ('2600:3000:2710:200::1d', 57184, 0, 0)
2018-05-30 01:43:55,686:DEBUG:acme.standalone:2600:3000:2710:200::1d - - Incoming request
2018-05-30 01:43:57,879:DEBUG:acme.client:Sending GET request to https://xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx HTTP/1.1" 200 1391
2018-05-30 01:43:58,059:DEBUG:acme.client:Received response:
2018-05-30 01:43:58,059:DEBUG:certbot.error_handler:Calling registered functions
2018-05-30 01:43:58,060:INFO:certbot.auth_handler:Cleaning up challenges
2018-05-30 01:43:58,060:DEBUG:certbot.plugins.standalone:Stopping server at :::443...
2018-05-30 01:43:58,191:DEBUG:certbot.client:CSR: CSR(file='xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx', form='pem')
2018-05-30 01:43:58,192:DEBUG:acme.client:Requesting issuance...
2018-05-30 01:43:58,192:DEBUG:acme.client:JWS payload:
2018-05-30 01:43:58,195:DEBUG:acme.client:Sending POST request to https://acme-v01.api.letsencrypt.org/acme/new-cert:
2018-05-30 01:43:58,543:DEBUG:urllib3.connectionpool:https://acme-v01.api.letsencrypt.org:443 "POST /acme/new-cert HTTP/1.1" 201 1550
2018-05-30 01:43:58,544:DEBUG:acme.client:Received response:
2018-05-30 01:43:58,544:DEBUG:acme.client:Storing nonce: xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:58,752:DEBUG:certbot.storage:Writing new private key to xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:58,752:DEBUG:certbot.storage:Writing certificate to xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:58,752:DEBUG:certbot.storage:Writing chain to xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:58,752:DEBUG:certbot.storage:Writing full chain to xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:58,774:DEBUG:certbot.storage:Writing new config xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx
2018-05-30 01:43:59,877:INFO:certbot.renewal:Cert not yet due for renewal
2018-05-30 01:43:59,879:DEBUG:certbot.renewal:no renewal failures
2018-05-30 01:43:59,879:INFO:certbot.hooks:Running post-hook command: service nginx start

As I understand it, it’s a longstanding bug. Read this comment: https://github.com/certbot/certbot/issues/5486#issuecomment-363970559 . The nginx installer does not play nice at renewal time with the standalone authenticator.

That issue you linked should no longer be an issue as of Certbot 0.21.0 or so (not sure on the exact version, but you’re definitely above it).

I recommend just getting rid of all of your standalone authenticators and pre/post hooks used to manage nginx, it doesn’t work very well at renewal time.

You should be able to go back to using --authenticator nginx --installer nginx safely.

2 Likes

That basically answers that and I’ll definitely go back to the Nginx authenticator to prevent this from occurring in the future.

Thanks for the help!

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.