Apache sigterm without restart after certbot

Help
1

Hi. I have used certbot for a few years now.

The issue is that after certbot runs, Apache does not restart for me.

I used a band-aid a few years ago with some setting that would ALWAYS restart Apache immediately after getting shut down - I don't remember where that settings was, though.

A few weeks ago, I updated certbot. Now the issue is back - Apache shuts down sometimes after certbot runs, but it does not come back up.

So far, I have not found a reason for this.

Syslog shows Certbot starting, Apache stopping, and Apache starting.
Letsencrypt log shows it running "systemctl start apache2"
Apache Error log shows SIGTERM, then Apache starting, then immediately another SIGTERM at the same time that Certbot is still working on stuff.

My environment is with Digital Ocean and running Debian 9.6.

I noticed that the letsencrypt logs are owned by root and not accessible by my regular user - is it possible that I installed it wrong and that this could be the issue?

Here are the logs for the time frame:

syslog:

May 18 10:13:05 myserver systemd[1]: Starting Certbot...
May 18 10:13:07 myserver systemd[1]: Stopping The Apache HTTP Server...
May 18 10:13:07 myserver systemd[1]: Stopped The Apache HTTP Server.
May 18 10:13:12 myserver systemd[1]: Starting The Apache HTTP Server...
May 18 10:13:12 myserver apachectl[19772]: httpd (pid 19769) already running
May 18 10:13:12 myserver systemd[1]: Started The Apache HTTP Server.
May 18 10:13:13 myserver systemd[1]: Started Certbot.
May 18 10:13:13 myserver systemd[1]: certbot.timer: Adding 9h 31min 31.742731s random time.
May 18 10:13:13 myserver systemd[1]: certbot.timer: Adding 3h 40min 47.792668s random time.

apache2/error.log:

[Tue May 18 10:13:07.499235 2021] [mpm_prefork:notice] [pid 11519] AH00169: caught SIGTERM, shutting down
[Tue May 18 10:13:12.774131 2021] [mpm_prefork:notice] [pid 19769] AH00163: Apache/2.4.25 (Debian) OpenSSL/1.0.2u configured -- resuming normal operations
[Tue May 18 10:13:12.774224 2021] [core:notice] [pid 19769] AH00094: Command line: '/usr/sbin/apache2'
[Tue May 18 10:13:12.971035 2021] [mpm_prefork:notice] [pid 19769] AH00169: caught SIGTERM, shutting down

letsencrypt.log:

2021-05-18 10:13:12,533:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2021-05-18 10:13:12,701:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer apache
2021-05-18 10:13:12,702:DEBUG:certbot.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7fdcf6dcf5f8>
Prep: True
2021-05-18 10:13:12,714:INFO:certbot.renewal:Cert not yet due for renewal
2021-05-18 10:13:12,715:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer apache
2021-05-18 10:13:12,720:DEBUG:certbot.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7fdcfb2906d8>
2021-05-18 10:13:12,720:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2021-05-18 10:13:12,729:INFO:certbot.renewal:Cert not yet due for renewal
2021-05-18 10:13:12,730:DEBUG:certbot.plugins.selection:Requested authenticator standalone and installer apache
2021-05-18 10:13:12,730:DEBUG:certbot.plugins.selection:Selecting plugin: * apache
Description: Apache Web Server plugin - Beta
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: apache = certbot_apache.entrypoint:ENTRYPOINT
Initialized: <certbot_apache.override_debian.DebianConfigurator object at 0x7fdcf6de8470>
2021-05-18 10:13:12,730:DEBUG:certbot.plugins.storage:Plugin storage file /etc/letsencrypt/.pluginstorage.json was empty, no values loaded
2021-05-18 10:13:12,730:DEBUG:certbot.renewal:no renewal failures
2021-05-18 10:13:12,731:INFO:certbot.hooks:Running post-hook command: systemctl start apache2

Based on a similar post here that doesn't show a solution ( Certbot causing apache shutdown on debian 9 stretch (SIGTERM) - #7 by Osiris ), I'm adding more data:

crontab -l only shows two unrelated jobs.
sudo crontab -l is empty
systemctl list-timers shows the following timers:

  • phpsessionclean.timer
  • certbot.timer
  • systemd-tmpfiles-clean-timer
  • apt-daily.timer
  • apt-daily-upgrade.timer

Thanks for any advice you can give me!

EDIT: I found how I force-restart Apache (which doesn't seem to work as planned):
/lib/systemd/system/apache2.service
Restart=on-abort
I may have to set it to "always" if I can't get a hold of this issue.

2

No, certbot usually (i.e., almost always) runs as root, that's normal.

The certbot apache plugin uses apache2ctl graceful (or apachectl, depending on the OS) to reload Apache. It seems you're using the standalone plugin with certbot hooks to stop and start Apache. Is there a specific reason for using the standalone plugin and not the apache or webroot plugin?

3

Thanks for the quick reply!

Thanks for confirming this.

Nope, I just went with the tutorials that made the most sense to me when I set everything up.

Which would be the best way to set this up?

4

If you don't mind certbot managing parts of your Apache config files, usually one uses --apache in stead of --standalone without all the Apache stopping and starting commands.

If you do mind the Apache config file modifying part, one could use --webroot in combination with the -w option to specify the webroot of your site (i.e., the same path as your DocumentRoot option in Apaches configuration.

See the official certbot documentation about how those plugins work and about their options.

5

Thanks so much for your time - I'll look into these options.

6

I'm still running into issues.

I ran:
certbot --apache
and followed the setup for all my sites.
I could not find any other reference on plugins.

Then I ran
sudo certbot renew --dry-run

At the end, certbot shows having started Apache again, but it is not running - I had to manually start it again - see the commands and output I used:

Running post-hook command: systemctl start apache2
3 renew failure(s), 0 parse failure(s)

myuser@myserver:/var/log$ systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Tue 2021-05-18 16:26:58 UTC; 9s ago
Process: 24710 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
Process: 24706 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 24543 (code=exited, status=0/SUCCESS)

May 18 16:26:58 myserver systemd[1]: Starting The Apache HTTP Server...
May 18 16:26:58 myserver apachectl[24706]: httpd (pid 24651) already running
May 18 16:26:58 myserver systemd[1]: Started The Apache HTTP Server.

myuser@myserver:/var/log$ sudo systemctl start apache2

myuser@myserver:/var/log$ systemctl status apache2
● apache2.service - The Apache HTTP Server
Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
Active: active (running) since Tue 2021-05-18 16:27:17 UTC; 5s ago
Process: 24710 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
Process: 24723 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
Main PID: 24727 (apache2)
Tasks: 6 (limit: 4915)
CGroup: /system.slice/apache2.service
├─24727 /usr/sbin/apache2 -k start
├─24728 /usr/sbin/apache2 -k start
├─24729 /usr/sbin/apache2 -k start
├─24730 /usr/sbin/apache2 -k start
├─24731 /usr/sbin/apache2 -k start
└─24732 /usr/sbin/apache2 -k start

May 18 16:27:17 myserver systemd[1]: Starting The Apache HTTP Server...
May 18 16:27:17 myserver systemd[1]: Started The Apache HTTP Server.
myuser@myserver:/var/log$

Also, I just checked, and the other methods you mentioned are not available on my server:

myuser@myserver:/var/log$ apachectl
-bash: apachectl: command not found
myuser@myserver:/var/log$ apache2ctl
-bash: apache2ctl: command not found

Please let me know what I'm doing wrong.

Thanks!

7

That command shouldn't be run when using --apache. Perhaps it's somewhere in your certbot configuration files? Or did you include it on the command line?

8

I don't think I made any manual changes to the configuration files (it's been a couple of years) - I think I just let it install and do what it did.

Otherwise, I typed all the commands exactly as I included them here.

Maybe certbot recognizes that the other commands are not available?

apachectl/apache2ctl don't exist on my system.

But yes, it is listed in at least one of the certbot config files:

# Options used in the renewal process
[renewalparams]
authenticator = standalone
post_hook = systemctl start apache2
account = [...]
pre_hook = systemctl stop apache2
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory

I'm pretty sure I did not edit those before.

What should I do?

9

Certbot has OS specific commands for different distributions, those 2 were examples.

You also had successfully run certbot with the --apache option, right? Where did that certificate go? Did certbot make a duplicate? Please check and paste the output of certbot certificates

10

When I ran it, some config changes had issues - something about binding port 80.

Here is the output from certbot certificates (changing the actual domain names):

$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: www.mydomain1.com
    Domains: mydomain2.com dev.mydomain2.com mydomain3.info mydomain3.us mydomain1.com www.mydomain2.com www.mydomain3.info www.mydomain3.us www.mydomain1.com
    Expiry Date: 2021-06-29 18:55:56+00:00 (VALID: 42 days)
    Certificate Path: /etc/letsencrypt/live/www.mydomain1.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/www.mydomain1.com/privkey.pem
  Certificate Name: www.mydomain2.com
    Domains: www.mydomain2.com
    Expiry Date: 2021-08-15 02:30:38+00:00 (VALID: 88 days)
    Certificate Path: /etc/letsencrypt/live/www.mydomain2.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/www.mydomain2.com/privkey.pem
  Certificate Name: mydomain1.com
    Domains: mydomain1.com
    Expiry Date: 2021-08-16 09:13:11+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/mydomain1.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mydomain1.com/privkey.pem
  Certificate Name: mydomain2.com
    Domains: mydomain2.com
    Expiry Date: 2021-08-16 14:40:53+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/mydomain2.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mydomain2.com/privkey.pem
  Certificate Name: dev.mydomain2.com
    Domains: dev.mydomain2.com
    Expiry Date: 2021-06-17 23:44:10+00:00 (VALID: 30 days)
    Certificate Path: /etc/letsencrypt/live/dev.mydomain2.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/dev.mydomain2.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Here is the output from the dryrun (again, replaced domain names):

$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mydomain1.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain2.com
http-01 challenge for dev.mydomain2.com
http-01 challenge for mydomain3.info
http-01 challenge for mydomain3.us
http-01 challenge for mydomain1.com
http-01 challenge for www.mydomain2.com
http-01 challenge for www.mydomain3.info
http-01 challenge for www.mydomain3.us
http-01 challenge for www.mydomain1.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/www.mydomain1.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mydomain2.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer apache
Running pre-hook command: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.mydomain2.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/www.mydomain2.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain1.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer apache
Pre-hook command already run, skipping: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain1.com
Cleaning up challenges
Attempting to renew cert (mydomain1.com) from /etc/letsencrypt/renewal/mydomain1.com.conf produced an unexpected error: Problem binding to port 80: Could not bind to IPv4 or IPv6.. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain2.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer apache
Pre-hook command already run, skipping: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain2.com
Cleaning up challenges
Attempting to renew cert (mydomain2.com) from /etc/letsencrypt/renewal/mydomain2.com.conf produced an unexpected error: Problem binding to port 80: Could not bind to IPv4 or IPv6.. Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/dev.mydomain2.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator standalone, Installer apache
Pre-hook command already run, skipping: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dev.mydomain2.com
Cleaning up challenges
Attempting to renew cert (dev.mydomain2.com) from /etc/letsencrypt/renewal/dev.mydomain2.com.conf produced an unexpected error: Problem binding to port 80: Could not bind to IPv4 or IPv6.. Skipping.
The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain1.com/fullchain.pem (failure)
  /etc/letsencrypt/live/mydomain2.com/fullchain.pem (failure)
  /etc/letsencrypt/live/dev.mydomain2.com/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

The following certs were successfully renewed:
  /etc/letsencrypt/live/www.mydomain1.com/fullchain.pem (success)
  /etc/letsencrypt/live/www.mydomain2.com/fullchain.pem (success)

The following certs could not be renewed:
  /etc/letsencrypt/live/mydomain1.com/fullchain.pem (failure)
  /etc/letsencrypt/live/mydomain2.com/fullchain.pem (failure)
  /etc/letsencrypt/live/dev.mydomain2.com/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: systemctl start apache2
3 renew failure(s), 0 parse failure(s)
11

You have a few redundant certificates.

The www.mydomain1.com certificate already uses the apache plugin and it seems you've renewed that certificate 48 days ago..

The certificate mydomain1.com (using the standalone authenticator and the apache plugin, which is a very, VERY unusual combination) is redundant and is covered by the www.mydomain1.com cert mentioned above.

The certificates mydomain2.com and www.mydomain2.com could easily be combined into a single certificate. They are currently using the standalone authenticator plugin and apache installer plugin, but I'm not seeing any reason not to use the apache plugin for authentication too.

dev.mydomain2.com could probably use the apache authenticator too.

I recommend you to clean up your certificates, integrate the two mydomain2 certs into a single cert and use the apache plugin for both authentication and installation. Don't use the standalone plugin unless you've got a very good reason to (and I'm not seeing it..)

12

OK, so I found references in the documentation for deleting certificates - would you recommend I use "revoke" on the redundant certificates or just "delete"?

Otherwise, what is the best way to get rid of the redundant entries?

So just to double-check - if I have a certificate for www.domain.com, I do NOT need another for domain.com? I also saw a reference to "expand" which I could use to include domain.com within www.domain.com - which is the better option?

And last, how do I switch plugins? I figured running "certbot --apache" would switch, but it didn't. I see entries in the documentation that explain the plugins, but I couldn't find how to change.

Would it be better to start from scratch, and if yes, what is the safest way to do that?

Sorry for the many questions, and thanks for your help!

13

Revoking is only necessary if the private key has been compromised.

Before deleting a cert, make sure it's not in use any longer in Apache or any other software. Certbot isn't smart enough to remove any directive related to a to-be deleted cert, not even if it was installed by certbot itself.

You don't need a separate cert for that. Note that usually one would like to have both hostnames in a cert, but usually those two hostnames would be included in a single cert.

That's a good option, indeed. You should include all the required hostnames in the command when using --expand, including the one already in the to-be expanded cert.

Hm, that should have done the trick. Note that using --dry-run doesn't actually change the certbot configuration. There's not really any way in certbot to change the certificate parameters unfortunately.. Even when it's the recommended client, it's lacking key user interface features...

1 Like
14

Thanks SO much for the explanations! That should take care of everything except the plugin switch.

I did run "certbot --apache" on its own without "--dry-run" earlier today, so it should have switched. And based on my bash history, I did run it as "sudo"...

I only used "--dry-run" along with "renew".

You didn't mean that I should combine the two, right? Like "sudo certbot renew --apache", correct?

So how do I fix this?

15

It should have, indeed.

Well, as far as I know that's also a possibility. You could try that the next time you need to renew.

16

The problem is that right now, every time certbot runs automatically, it shuts down Apache and for some reason fails to start it back up, and I have to log into my server to restart...

I guess for now I'll install have to a bash file that'll start Apache automatically if it's down.

Would it be worth deleting all certbot settings and starting from scratch? If yes, how would I do that?

What else could I do to switch into Apache mode and remove standalone?

17

Most of the files in /etc/letsencrypt/renewal/ (not all) would contain a line authenticator = standalone in the renewal configuration file. It probably would be sufficient to change standalone into apache so it would read authenticator = apache. Before you should do such a manual change, please always backup the directory/files you're about to change, just in case.

Afterwards you should be able to run certbot renew --dry-run and check the "Plugins selected" -line in the output.

18

I'll give that a try, thanks!

19

The good news: The port 80 errors went away.

The bad news: certbot still seems unable to start Apache after it is done:

$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mydomain3.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain1.com
http-01 challenge for dev.mydomain1.com
http-01 challenge for mydomain2.info
http-01 challenge for mydomain2.us
http-01 challenge for mydomain3.com
http-01 challenge for www.mydomain1.com
http-01 challenge for www.mydomain2.info
http-01 challenge for www.mydomain2.us
http-01 challenge for www.mydomain3.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/www.mydomain3.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.mydomain1.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Running pre-hook command: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.mydomain1.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/www.mydomain1.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain3.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Pre-hook command already run, skipping: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain3.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/mydomain3.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/mydomain1.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Pre-hook command already run, skipping: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mydomain1.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/mydomain1.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/dev.mydomain1.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator apache, Installer apache
Pre-hook command already run, skipping: systemctl stop apache2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for dev.mydomain1.com
Waiting for verification...
Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of apache server; fullchain is
/etc/letsencrypt/live/dev.mydomain1.com/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/www.mydomain3.com/fullchain.pem (success)
  /etc/letsencrypt/live/www.mydomain1.com/fullchain.pem (success)
  /etc/letsencrypt/live/mydomain3.com/fullchain.pem (success)
  /etc/letsencrypt/live/mydomain1.com/fullchain.pem (success)
  /etc/letsencrypt/live/dev.mydomain1.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Running post-hook command: systemctl start apache2
$ ps aux | grep httpd
user     2548  0.0  0.0  12716   972 pts/0    S+   16:23   0:00 grep httpd
$ ps aux | grep apache
user     2556  0.0  0.0  12716   932 pts/0    S+   16:23   0:00 grep apache
$ systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: enabled)
   Active: inactive (dead) since Wed 2021-05-19 16:23:28 UTC; 25s ago
  Process: 2539 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS)
  Process: 2530 ExecStart=/usr/sbin/apachectl start (code=exited, status=0/SUCCESS)
 Main PID: 675 (code=exited, status=0/SUCCESS)

May 19 16:23:28 mydomain1 systemd[1]: Starting The Apache HTTP Server...
May 19 16:23:28 mydomain1 apachectl[2530]: httpd (pid 2333) already running
May 19 16:23:28 mydomain1 systemd[1]: Started The Apache HTTP Server.

My new bash script caught it and started Apache again after a few minutes, so that also works.

Any idea what could cause certbot not able to restart Apache? Any idea how to track this? I can't find anything in the logs so far...

As a band-aid, could I use "--manual-cleanup-hook"? When does this run - at the very end of the process, or before it tries to restart Apache?

Huh, I just discovered something weird: apachectl DOES exist - it's in /usr/sbin but for some reason it is not found when run without giving it a path... Is this normal?

20

Well, I forgot to mention that above, but all those pre- and post-hooks with stopping and starting Apache aren't necessary when using the apache plugin. And it seems that plugin does actually work nicely, so you could remove those pre- and post-hooks with stopping and starting Apache from your renewal configuration files too. (Don't forget to backup!)

1 Like