A couple of months I raised this issue and although I got it resolved under urgency never really got to the bottom of it.
Today I have managed to do so and it seems like a concerning behavior that is happening.
I've been running this command to renew a certificate.
/opt/letsencrypt/certbot-auto renew
The command errors when it is unable to start apache after the certificate is renewed (or not renewed as it's not due).
Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address 127.0.0.1:443
no listening sockets available, shutting down
AH00015: Unable to open logs
Attempting to renew cert (mydomain.com) from /etc/letsencrypt/renewal/mydomain.com.conf produced an unexpected error: Error while running apache2ctl graceful.
httpd not running, trying to start
Action 'graceful' failed.
The Apache error log may have more information.
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1. Set the 'ServerName' directive globally to suppress this message
(98)Address already in use: AH00072: make_sock: could not bind to address 127.0.0.1:443
no listening sockets available, shutting down
AH00015: Unable to open logs
. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mydomain.com/fullchain.pem (failure)
After running this it was clear both nginx and apache were trying to use port 443 but what I couldn't figure out was why. Naturally I trawled through the config for both trying to find out what I had done wrong.
It eventually became clear, confirmed through tests, that the certbot-auto renew command is editing the ports.conf file for apache. The original section is
Listen 127.0.0.1:80
But it is being changed to
Listen 127.0.0.1:443
As I use Jenkins jobs to automate certificate renewals, this means we cannot automate this one as I have to edit this file every time afterwards.
Should this be happening? Does anyone know how I can change it?
Sorry about that, my original version used ^ at the beginning and $ at the end, and then I thought “well, there could be spaces or tabs at the beginning, so let’s give the less correct and more general version instead”.
Here’s what’s going on here. certbot-auto is using our Apache plugin to obtain and renew your certificate. For this to work, Apache needs to temporarily handle traffic coming in on port 443 in order to perform the domain validation required for Let’s Encrypt to issue your cert. When you run certbot-auto renew, Certbot sees that Apache isn’t listening on this port, tries to add this line so it can solve the challenge, but because Nginx is listening on this port, renewal fails.
The solution is to use another Certbot plugin to renew your cert. If Nginx is listening on port 443 and using the certs obtained by Let’s Encrypt, I’d recommend using the Nginx plugin. Alternatively, you can use the webroot plugin if your server is serving files out of a directory for the domain you want to obtain a cert for.
To change the plugin being used, you should run certbot-auto renew --cert-name <name> --force-renewal --<plugin> once for each cert where <name> is the name of the directory in /etc/letsencrypt/live containing the cert and <plugin> the lowercase name of the plugin you want to use. If you want to use the webroot plugin, you should also include --webroot-path <path> on the command line where <path> is the path to the directory files are being served out of for the domain(s) in the certificate.
@schoen - understood and appreciated. I was working on your method, but having issues getting permissions sorted for the user that runs the renewals. I was slowly getting there.
@bmw - thank you so much. That method of forcing a renewal with nginx worked perfectly. Presumably I ran it incorrectly at the start using the apache plugin and the problem stemmed from there. Everything running as expected.