Apache2 does not come back up, certbot finishes abnormally

Matched (these seem OK):

         port 80 namevhost home.baldockery.com		/etc/apache2/sites-enabled/000-default.conf:27
         port 443 namevhost home.baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:179

         port 80 namevhost openhab.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:36
         port 443 namevhost openhab.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:50

         port 80 namevhost nodered.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:45
         port 443 namevhost nodered.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:66

         port 80 namevhost dakboard.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:54
         port 443 namevhost dakboard.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:89

         port 80 namevhost mypi.baldockery.com		/etc/apache2/sites-enabled/000-default.conf:63
         port 443 namevhost mypi.baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:113

Unmatched sections or containing irregularity:

         port 80 namevhost baldockery.com		/etc/apache2/sites-enabled/000-default.conf:1
         port 443 namevhost baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:1
                 alias baldockery.com

         port 80 namevhost enphase.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:72

         port 443 namevhost emonpi.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:136

         port 443 namevhost test.baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:157
1 Like

From the little you show of the error, it is hard to say which section(s) failed to renew, nor even which needed to be renewed.

Please matchup the unmatched sections and correct the redundant alias.
Also, please show:
certbot certificates

1 Like

Can you help me with what you mean by matching the unmatched sections and correcting the redundant alias?

Output of certbot certficates:

Found the following certs:
  Certificate Name: baldockery.com
    Domains: baldockery.com
    Expiry Date: 2020-08-13 22:15:09+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/baldockery.com/privkey.pem
  Certificate Name: dakboard.baldockery.com
    Domains: dakboard.baldockery.com
    Expiry Date: 2020-08-14 08:35:23+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/dakboard.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/dakboard.baldockery.com/privkey.pem
  Certificate Name: emoncmsnodered.baldockery.com
    Domains: emoncmsnodered.baldockery.com
    Expiry Date: 2020-06-15 08:48:17+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/emoncmsnodered.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/emoncmsnodered.baldockery.com/privkey.pem
  Certificate Name: emonpi.baldockery.com
    Domains: emonpi.baldockery.com
    Expiry Date: 2020-06-15 08:48:54+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/emonpi.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/emonpi.baldockery.com/privkey.pem
  Certificate Name: home.baldockery.com
    Domains: home.baldockery.com
    Expiry Date: 2020-08-14 08:36:25+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/home.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/home.baldockery.com/privkey.pem
  Certificate Name: mypi.baldockery.com
    Domains: mypi.baldockery.com
    Expiry Date: 2020-06-15 08:49:51+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/mypi.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mypi.baldockery.com/privkey.pem
  Certificate Name: nodered.baldockery.com
    Domains: nodered.baldockery.com
    Expiry Date: 2020-08-14 08:36:57+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/nodered.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/nodered.baldockery.com/privkey.pem
  Certificate Name: noderedemoncms.baldockery.com
    Domains: noderedemoncms.baldockery.com
    Expiry Date: 2020-06-15 08:51:36+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/noderedemoncms.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/noderedemoncms.baldockery.com/privkey.pem
  Certificate Name: openhab.baldockery.com
    Domains: openhab.baldockery.com
    Expiry Date: 2020-08-14 08:37:41+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/openhab.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/openhab.baldockery.com/privkey.pem
  Certificate Name: test.baldockery.com
    Domains: test.baldockery.com
    Expiry Date: 2020-08-14 08:37:53+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/test.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/test.baldockery.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Like

The redundant alias is simple.
The config has the same server name twice.

Matching the unmatched has to do with making HTTP and HTTPS enabled sections for each of the server names.

1 Like

I corrected the redundant alias and the unmatched sections for the various virtual servers. And I updated the Route53 IP address for the ones that I had let get out-of-date. Now sudo certbot renew runs without errors. I restarted the timer. Hopefully when it next runs apache2 stays up.

2 Likes

I am curious why, when I had these errors in my configuration, I was able to restart apache2, but certbot was not able to restart apache2.

Thanks for all your help with this!

1 Like

May act differently than however you stopped and started apache.

1 Like

True. I use systemctl start apache2. I’ve read that is the better way to do it (https://www.configserverfirewall.com/linux-tutorials/apachectl-command/) but maybe certbot doesn’t want to depend on systemd being in place on various platforms.

1 Like

OK, if you issue that command, does it still complain?
apachectl -k start

1 Like

It still complains.

pi@emonpi:~$ sudo apachectl -k stop
pi@emonpi:~$ sudo apachectl -k start
pi@emonpi:~$ sudo systemctl status apache2
● apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: 
   Active: inactive (dead) since Wed 2020-05-27 17:30:28 CDT; 31s ago
     Docs: https://httpd.apache.org/docs/2.4/
  Process: 8969 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS
 Main PID: 6583 (code=exited, status=0/SUCCESS)

May 27 12:02:51 emonpi systemd[1]: Starting The Apache HTTP Server...
May 27 12:02:51 emonpi systemd[1]: Started The Apache HTTP Server.
May 27 17:30:28 emonpi apachectl[8969]: httpd (no pid file) not running
May 27 17:30:28 emonpi systemd[1]: apache2.service: Succeeded.
1 Like

hmm…

Does certbot still show that problem?

1 Like

After I corrected my configuration I ran certbot renew manually and it worked, so now when I try to test if certbot will stop and start apache2 successfully, it doesn’t ever stop it because none of the certificates are up for renewal.

2 Likes

I have some time to look at this again today, and I’m wondering if you have suggestions on how to check if certbot still shows the problem.

My other question is if I should go back to using cron to periodically run certbot, and use post-hook “systemctl start apache”, since systemctl is working for me and apachectl is not. Obviously it would be more satisfying to figure out what is going on with apachectl, but in the meantime is there any harm in going with this cron idea?

1 Like

--post-hook will always execute
--deploy-hook will only execute when the cert is actually renewed [recommended]

How did you get/generate the previous error and output?

1 Like

Thanks! How would deploy-hook be used in a cron command like this? In place of --pre-hook or --post-hook, and then what goes in the other spot?

@weekly certbot renew --authenticator standalone --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2" >> /home/pi/data/certbot.log 2>&1

Before I updated Route53 with my current IP address for a few of those sites that I had let get out of date, every time the systemd certbot service ran it would try to renew the certificates for those sites, bring down apache and try to bring it back up. Two days ago, when I updated Route53 while working on this, I did a certbot renew and they successfully renewed. So now the certbot service doesn’t find anything that needs renewing, so it never brings down apache. Thus the best simulation I know of now is to just try “apachectl -k start”. When you asked if certbot still showed the problem, I wondered if you knew a way to make it do so.

1 Like

standalone breaks the deploy hook logic.
As it requires you to stop the web service to spin up a new temporary web service.
You really should find a way to use the existing web server (and leave it running at all times).
Then you can use --deploy-hook

1 Like

I’m not 100% certain that it works that way.
Or at least the way you expect and on all versions of certbot.

I would test that out that command and see if it stops/starts the web server even now when it doesn’t need to renew anything.

certbot renew --authenticator standalone --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2"

[or leave the post hook off and see if apache is still running]

I can confirm that in version 0.31.0, certbot does work as you expected and will not stop apache when no renewal is required.

1 Like

Why is certbot trying to run this command?

What are your hook configurations?

Look in

  • /etc/letsencrypt/renewal/
  • /etc/letsencrypt/renewal-hooks/
  • /etc/letsencrypt/cli.ini

and tell us.


If you really want to use --standalone you don’t need to stop apache. You can just reverse proxy certbot’s server through it:

<Location "/.well-known/acme-challenge">
   ProxyPass "http://localhost:23782/.well-known/acme-challenge"
   # 23782 == CERTB, you can choose another free port.
</Location>

and then tell certbot to use that port:

certbot --standalone --http-01-port 23782
2 Likes
pi@emonpi:~$ ls -lh /etc/letsencrypt/renewal/
total 40K
-rw-r--r-- 1 root root 574 May 15 18:15 baldockery.com.conf
-rw-r--r-- 1 root root 619 May 16 04:35 dakboard.baldockery.com.conf
-rw-r--r-- 1 root root 649 May 27 12:00 emoncmsnodered.baldockery.com.conf
-rw-r--r-- 1 root root 609 May 27 12:01 emonpi.baldockery.com.conf
-rw-r--r-- 1 root root 599 May 16 04:36 home.baldockery.com.conf
-rw-r--r-- 1 root root 599 May 27 12:01 mypi.baldockery.com.conf
-rw-r--r-- 1 root root 614 May 16 04:36 nodered.baldockery.com.conf
-rw-r--r-- 1 root root 649 May 27 12:01 noderedemoncms.baldockery.com.conf
-rw-r--r-- 1 root root 614 May 16 04:37 openhab.baldockery.com.conf
-rw-r--r-- 1 root root 599 May 16 04:37 test.baldockery.com.conf
pi@emonpi:~$ cat /etc/letsencrypt/renewal/*.conf
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/baldockery.com
cert = /etc/letsencrypt/live/baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/dakboard.baldockery.com
cert = /etc/letsencrypt/live/dakboard.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/dakboard.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/dakboard.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/dakboard.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/emoncmsnodered.baldockery.com
cert = /etc/letsencrypt/live/emoncmsnodered.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/emoncmsnodered.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/emoncmsnodered.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/emoncmsnodered.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/emonpi.baldockery.com
cert = /etc/letsencrypt/live/emonpi.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/emonpi.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/emonpi.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/emonpi.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/home.baldockery.com
cert = /etc/letsencrypt/live/home.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/home.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/home.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/home.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/mypi.baldockery.com
cert = /etc/letsencrypt/live/mypi.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/mypi.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/mypi.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/mypi.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/nodered.baldockery.com
cert = /etc/letsencrypt/live/nodered.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/nodered.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/nodered.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/nodered.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/noderedemoncms.baldockery.com
cert = /etc/letsencrypt/live/noderedemoncms.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/noderedemoncms.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/noderedemoncms.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/noderedemoncms.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/openhab.baldockery.com
cert = /etc/letsencrypt/live/openhab.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/openhab.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/openhab.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/openhab.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/test.baldockery.com
cert = /etc/letsencrypt/live/test.baldockery.com/cert.pem
privkey = /etc/letsencrypt/live/test.baldockery.com/privkey.pem
chain = /etc/letsencrypt/live/test.baldockery.com/chain.pem
fullchain = /etc/letsencrypt/live/test.baldockery.com/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = fba61c080ee8d787595e4fdf81b6560c
pre_hook = apachectl -k stop
post_hook = apachectl -k start
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory

Three empty directories:

pi@emonpi:~$ ls -lh /etc/letsencrypt/renewal-hooks
total 12K
drwxr-xr-x 2 root root 4.0K Nov 17  2019 deploy
drwxr-xr-x 2 root root 4.0K Nov 17  2019 post
drwxr-xr-x 2 root root 4.0K Nov 17  2019 pre
pi@emonpi:~$ cat /etc/letsencrypt/cli.ini
# Because we are using logrotate for greater flexibility, disable the
# internal certbot logrotation.
max-log-backups = 0

I have no specific attachment to using --standalone. This is just the command I learned when from the blogs and posts I read when I first installed certbot way back when. I confess I don’t even know what it does, only that it was working for me back when I was using cron to run an older certbot version. But this is good to know. My favorite idea is to figure out why the newer systems-based service is not working for me.

2 Likes

Then don’t use it. It’s not very useful when you already have a webserver running.

It spins up a temporary webserver to perform an http validation. But you can do that perfectly well using the apache webserver you already have, using --apache or --webroot.

Is this the command you’re trying?

Try using certbot renew --apache --pre-hook "" --post-hook "" for the next ~100 days.

2 Likes