Certbot renew dry run works but not actual renew

Certbot renew does not work but dry run works. There are no firewall blocks and nginx configuration is correct. This same configuration used to work before (on this server) and it works on other servers (similar stack) but some servers including this one has this unknown issue.

My domain is: mail.codexplorermail.com

I ran this command: certbot renew

It produced this output:

root@mail:~# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.codexplorermail.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.codexplorermail.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/mail.codexplorermail.com.conf produced an unexpected error: Failed authorization procedure. mail.codexplorermail.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.codexplorermail.com/.well-known/acme-challenge/XHY-D0BRkWpeC1toKxhtA8zK7a8TFeCep4Z-KvvLP6Y: Connection refused. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mail.codexplorermail.com/fullchain.pem (failure)
Running post-hook command: service nginx start && service postfix reload && service dovecot reload
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mail.codexplorermail.com
   Type:   connection
   Detail: Fetching
   http://mail.codexplorermail.com/.well-known/acme-challenge/XHY-D0BRkWpeC1toKxhtA8zK7a8TFeCep4Z-KvvLP6Y:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

After this, I ran command certbot renew --dry-run and it worked. Output below:

root@mail:~# certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.codexplorermail.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.codexplorermail.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0041_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0041_csr-certbot.pem
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/mail.codexplorermail.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
Running post-hook command: service nginx start && service postfix reload && service dovecot reload

My web server is: Nginx 1.6.2

My certbot is: certbot 0.10.2

The operating system my web server runs on is: Debian 8.9

My hosting provider, if applicable, is: n/a. Installed official debian certbot package using apt-get install certbot -t jessie-backports

I can login to a root shell on my machine: Yes

I’m using a control panel to manage my site: No

Strange things happening: The command above is running pre-hook and post-hook although as evident from the command it wasn’t typed on the console. These pre-hook and post-hook actions are mentioned in cron.d like so 06 */12 * * * root certbot renew --quiet --force-renewal --pre-hook 'service nginx stop' --post-hook 'service nginx start && service postfix reload && service dovecot reload'.

I really appreciate any help in troubleshooting this. Its rather very strange and I’m at a loss as to what’s happening here.

EDIT: Traceback with debug:

root@mail:~# certbot renew --dry-run --debug
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.codexplorermail.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.codexplorermail.com
Waiting for verification...
Cleaning up challenges
Generating key (2048 bits): /etc/letsencrypt/keys/0042_key-certbot.pem
Creating CSR: /etc/letsencrypt/csr/0042_csr-certbot.pem
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/mail.codexplorermail.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
Running post-hook command: service nginx start && service postfix reload && service dovecot reload
root@mail:~# certbot renew --debug
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.codexplorermail.com.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Running pre-hook command: service nginx stop
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.codexplorermail.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert from /etc/letsencrypt/renewal/mail.codexplorermail.com.conf produced an unexpected error: Failed authorization procedure. mail.codexplorermail.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.codexplorermail.com/.well-known/acme-challenge/V9pXO0OuoFTNo3kPQG9PwjRjCEK5oWUTKuR81i34c2A: Connection refused. Skipping.

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/mail.codexplorermail.com/fullchain.pem (failure)
Running post-hook command: service nginx start && service postfix reload && service dovecot reload
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.10.2', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 849, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 655, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 430, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
Error: 1 renew failure(s), 0 parse failure(s)


IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mail.codexplorermail.com
   Type:   connection
   Detail: Fetching
   http://mail.codexplorermail.com/.well-known/acme-challenge/V9pXO0OuoFTNo3kPQG9PwjRjCEK5oWUTKuR81i34c2A:
   Connection refused

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Can you take a look in /etc/letsencrypt/renewal/mail.codexplorermail.com.conf and see if the hooks are also defined there? If so, they would be run automatically during the renewal.

Can you take a look in /etc/letsencrypt/renewal/mail.codexplorermail.com.conf and see if the hooks are also defined there? If so, they would be run automatically during the renewal.

Thanks for that. The hooks are defined there and I suppose automatically. I'm positive I didn't put it there.

I believe if you specify them with the command that originally obtained the certificate, they are saved as part of the renewal parameters.

That makes sense. I used this certbot certonly --quiet --force-renewal --agree-tos -m user@xxx.com --webroot -w /opt/certbot-webroot -d mail.codexplorermail.com --pre-hook 'service nginx stop' --post-hook 'service nginx start && service postfix reload && service dovecot reload' command when the renewal issue happened previously to resissue the certificate. I have edited the original post with traceback obtained with debug flag. Hopefully that sheds some light on what's happening.

I believe this is intended behavior. Certbot doesn’t really understand the notion of “one-off” hook use as opposed to hooks that are a necessary, permanent part of the certificate issuance process.

1 Like

Hello @rooty,

Maybe I missed something but you are using certonly+webroot so you need a web server listening on port 80 to answer Let's Encrypt requests but you are stopping nginx using pre-hook parameter... I would understand that command if you were using standalone authenticator but you are not...

Cheers,
sahsanu

Maybe I missed something but you are using certonly+webroot so you need a web server listening on port 80 to answer Let’s Encrypt requests but you are stopping nginx using pre-hook parameter… I would understand that command if you were using standalone authenticator but you are not…

That solved it. Thanks @sahsanu and @schoen. Some of the servers used certbot's standalone option which required Nginx to be stopped so that port 443 was not in use during renewal, some used cerbot's webroot option which needed port 443 to be accepting connections (Nginx) during renewal. I completely missed it; helps to have a fresh look :slight_smile:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.