Certbot failing to restart Apache after renew

strongn · August 15, 2018, 6:30pm

H
I wonder if anyone can help - I’ve a recurring issue where certbot fails to restart the web server.

( I’m running Debian Stretch, Apache2 web server )

It seems random, doesn’t happen every day, and doesn’t even seem to be related to the time certbot is scheduled to run, but the times in the logs correlate

The run starts in letsencrypt.log as follows:

2018-08-15 12:31:08,589:INFO:certbot.hooks:Running pre-hook command: apachectl -k stop
2018-08-15 12:31:08,708:ERROR:certbot.hooks:Error output from apachectl:
[Wed Aug 15 12:31:08.650940 2018] [alias:warn] [pid 27477] AH00671: The Alias directive in /etc/phpmyadmin/apache.conf at line 3 will probably never match because it overlaps an earlier Alias.
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

The final lines in letsencrypt.log are as follows:

2018-08-15 12:31:14,247:DEBUG:certbot.renewal:no renewal failures
2018-08-15 12:31:14,247:INFO:certbot.hooks:Running post-hook command: apachectl -k start
2018-08-15 12:31:14,326:ERROR:certbot.hooks:Error output from apachectl:
[Wed Aug 15 12:31:14.303140 2018] [alias:warn] [pid 27491] AH00671: The Alias directive in /etc/phpmyadmin/apache.conf at line 3 will probably never match because it overlaps an earlier Alias.
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 127.0.1.1. Set the 'ServerName' directive globally to suppress this message

The apache error.log shows this:

[Wed Aug 15 12:31:08.741571 2018] [mpm_prefork:notice] [pid 17154] AH00169: caught SIGTERM, shutting down
[Wed Aug 15 12:31:14.433358 2018] [mpm_prefork:notice] [pid 27492] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Wed Aug 15 12:31:14.433424 2018] [core:notice] [pid 27492] AH00094: Command line: '/usr/sbin/apache2'
[Wed Aug 15 12:31:14.567663 2018] [mpm_prefork:notice] [pid 27492] AH00169: caught SIGTERM, shutting down

And finally, this is when I performed a manual apachectl start which worked fine.

[Wed Aug 15 19:12:39.401637 2018] [mpm_prefork:notice] [pid 11062] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2l configured -- resuming normal operations
[Wed Aug 15 19:12:39.401690 2018] [core:notice] [pid 11062] AH00094: Command line: '/usr/sbin/apache2'

The cron entry is as follows - I can’t even understand why it ran at 12:31!?

00 22 * * * certbot renew --renew-hook "systemctl restart apache2"

Any assistance gratefuly received

JuergenAuer · August 15, 2018, 6:36pm

Hi @strongn

are there other jobs? at? systemd?

strongn · August 15, 2018, 6:44pm

systemctl list jobs says “No Jobs Running” and I’m not using ‘at’

Is there another way to schedule the certbot run. I’m slightly puzzled why it decided to run in the middle of the day…

jmorahan · August 15, 2018, 6:45pm

maybe try systemctl list-timers

strongn · August 15, 2018, 6:53pm

Aha.

Thu 2018-08-16 01:39:08 BST 5h 53min left Wed 2018-08-15 12:31:07 BST 7h ago certbot.timer certbot.service

I’m not familiar with systemctl jobs. I found a file called certbot.timer in /etc/systemd/system/timers.target.wants.

It contains this:

[Unit]
Description=Run certbot twice daily

[Timer]
OnCalendar=--* 00,12:00:00
RandomizedDelaySec=43200
Persistent=true

[Install]
WantedBy=timers.target

It doesn’t seem to have the command in it…? At least that explains why it runs in the middle of the day!

jmorahan · August 15, 2018, 6:59pm

The command is in the corresponding certbot.service file. However it’s probably just the standard certbot -q renew which should be fine.

The problem is more likely in your certbot configuration under /etc/letsencrypt/renewal/ - would you mind sharing the contents of those files?

Also, did you install Apache from Debian’s repositories, or are you using a separate Apache install such as Bitnami/Serverpilot/etc?

strongn · August 15, 2018, 6:59pm

Ok, now I’m getting somewhere.

There’s a file in /lib/systemd/system called certbot.service. It looks like this:

[Unit]
Description=Certbot
Documentation=file:///usr/share/doc/python-certbot-doc/html/index.html
Documentation=https://letsencrypt.readthedocs.io/en/latest/
[Service]
Type=oneshot
ExecStart=/usr/bin/certbot -q renew
PrivateTmp=true

strongn · August 15, 2018, 7:03pm

It was installed from the debian apt repository I think. There are 5 files in the renewal dir. Do you need the contents of each of them?

jmorahan · August 15, 2018, 7:08pm

I expect at least one of them contains a reference to those apachectl -k stop/start commands, and I’d like to see that one please

strongn · August 15, 2018, 7:10pm

They all do: here’s one of them:

# renew_before_expiry = 30 days
version = 0.23.0
archive_dir = /etc/letsencrypt/archive/api.school360.co.uk
cert = /etc/letsencrypt/live/api.school360.co.uk/cert.pem
privkey = /etc/letsencrypt/live/api.school360.co.uk/privkey.pem
chain = /etc/letsencrypt/live/api.school360.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/api.school360.co.uk/fullchain.pem

# Options used in the renewal process
[renewalparams]
post_hook = apachectl -k start
installer = None
account = 3828f3e8e33653f5ee5687a24fe6aaae
pre_hook = apachectl -k stop
authenticator = standalone

jmorahan · August 15, 2018, 7:24pm

Okay so you’re using the standalone authenticator, my first question is do you have a specific reason for that? As it would usually be easier and safer to use the apache authenticator. For example does this test work?

sudo certbot certonly --dry-run -a apache --preferred-challenges http --cert-name api.school360.co.uk

If you do need to stick with standalone for some reason, I suspect that using systemd to stop and start apache might work better than apachectl -k for example if you changed those lines in the configuration to:

pre_hook = systemctl stop apache2
post_hook = systemctl start apache2

strongn · August 15, 2018, 7:30pm

It’s been a while since I installed, and the server wasn’t ‘live’ at the time ( THe school360.co.uk domain was pointed elsewhere at the time, so maybe that’s why )

The command seemed to work OK.

$ sudo certbot certonly --dry-run -a apache --preferred-challenges http --cert-name api.school360.co.uk

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer None
Cert not due for renewal, but simulating renewal for dry run
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for api.school360.co.uk
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - The dry run was successful.
 - Your account credentials have been saved in your Certbot
   configuration directory at /etc/letsencrypt. You should make a
   secure backup of this folder now. This configuration directory will
   also contain certificates and private keys obtained by Certbot so
   making regular backups of this folder is ideal.

So should I be modifying those config files manually?

I can change the pre_hook and post_hook ok, but what should I change the authenticator to?

jmorahan · August 15, 2018, 7:36pm

You don’t need to modify the config files manually.

Since the apache authenticator seems to be working, you could try this:

sudo certbot --cert-name api.school360.co.uk --apache --preferred-challenges http

That should switch you to the apache authenticator and installer - when the cert is already installed, all the installer does is reload your configuration.

If that doesn’t work for some reason, you could try this instead:

sudo certbot certonly --cert-name api.school360.co.uk -a apache --preferred-challenges http --deploy-hook "systemctl reload apache2"

That just uses the apache authenticator and uses a deploy-hook to reload the configuration instead of the installer.

You don’t need any pre-hook or post-hook in either case - I think they get removed automatically if those commands succeed, but no harm to double-check the config afterwards to make sure.

(Note that these commands will renew your cert immediately before updating the configuration, which might be something to be aware of if you happen to have any domains that are already close to their rate limits)

strongn · August 15, 2018, 7:42pm

The first command didn’t change the config files, but the second did ( I told it to renew the certs regardless )

New config file looks like this:

# renew_before_expiry = 30 days
version = 0.23.0
archive_dir = /etc/letsencrypt/archive/api.school360.co.uk
cert = /etc/letsencrypt/live/api.school360.co.uk/cert.pem
privkey = /etc/letsencrypt/live/api.school360.co.uk/privkey.pem
chain = /etc/letsencrypt/live/api.school360.co.uk/chain.pem
fullchain = /etc/letsencrypt/live/api.school360.co.uk/fullchain.pem

# Options used in the renewal process
[renewalparams]
installer = None
account = 3828f3e8e33653f5ee5687a24fe6aaae
authenticator = apache
pref_challs = http-01,
renew_hook = systemctl reload apache2

… AND the web server it still alive!

Thanks John. I’ll run that against the other 4 domains and see how that goes. Fingers crossed!

Many thanks for your time and assistance!

jmorahan · August 15, 2018, 7:46pm

No problem, good luck

system · September 14, 2018, 7:46pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Troubleshoot apache2 not starting after cert renewal Help	4	984	December 1, 2019
Certbot failed to restart apache Help	8	3701	January 9, 2019
Apache fails to start again after `certbot-auto renew` script Server	3	3097	March 4, 2018
Apache2 does not come back up, certbot finishes abnormally Help	23	1325	June 29, 2020
After years of successful use, 'certbot renew' failing on apache2 Help	3	1063	February 26, 2022

Certbot failing to restart Apache after renew

Related topics