Apache2 does not come back up, certbot finishes abnormally

My domain is: baldockery.com

I ran this command: whenever systemd is running the default certbot.timer task, I get errors (below) and apache2 fails to come back up. sudo certbot renew --dry-run produces the same result - apache2 is dead after running it.

It produced this output: I can provide the entire output if necessary, but here is the output after the certbot.timer task tries to bring apache2 back up:

2020-05-26 21:51:04,811:INFO:certbot.hooks:Running post-hook command: apachectl -k start
2020-05-26 21:51:05,214:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1272, in renew
    renewal.handle_renewal_request(config)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 477, in handle_renewal_request
    len(renew_failures), len(parse_failures)))
certbot.errors.Error: 4 renew failure(s), 0 parse failure(s)

My web server is (include version)/The operating system my web server runs on is (include version): Apache/2.4.38 (Raspbian)

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.31.0

1 Like

Please show the output of:
apachectl -S

1 Like
VirtualHost configuration:
*:80                   is a NameVirtualHost
         default server baldockery.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost baldockery.com (/etc/apache2/sites-enabled/000-default.conf:1)
         port 80 namevhost home.baldockery.com (/etc/apache2/sites-enabled/000-default.conf:27)
         port 80 namevhost openhab.baldockery.com (/etc/apache2/sites-enabled/000-default.conf:36)
         port 80 namevhost nodered.baldockery.com (/etc/apache2/sites-enabled/000-default.conf:45)
         port 80 namevhost dakboard.baldockery.com (/etc/apache2/sites-enabled/000-default.conf:54)
         port 80 namevhost mypi.baldockery.com (/etc/apache2/sites-enabled/000-default.conf:63)
         port 80 namevhost enphase.baldockery.com (/etc/apache2/sites-enabled/000-default.conf:72)
*:443                  is a NameVirtualHost
         default server baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:1)
         port 443 namevhost baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:1)
                 alias baldockery.com
         port 443 namevhost openhab.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:50)
         port 443 namevhost nodered.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:66)
         port 443 namevhost dakboard.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:89)
         port 443 namevhost mypi.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:113)
         port 443 namevhost emonpi.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:136)
         port 443 namevhost test.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:157)
         port 443 namevhost home.baldockery.com (/etc/apache2/sites-enabled/default-ssl.conf:179)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/emoncms/apache2-error.log"Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default 
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex proxy-balancer-shm: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33
1 Like

Matched (these seem OK):

         port 80 namevhost home.baldockery.com		/etc/apache2/sites-enabled/000-default.conf:27
         port 443 namevhost home.baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:179

         port 80 namevhost openhab.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:36
         port 443 namevhost openhab.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:50

         port 80 namevhost nodered.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:45
         port 443 namevhost nodered.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:66

         port 80 namevhost dakboard.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:54
         port 443 namevhost dakboard.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:89

         port 80 namevhost mypi.baldockery.com		/etc/apache2/sites-enabled/000-default.conf:63
         port 443 namevhost mypi.baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:113

Unmatched sections or containing irregularity:

         port 80 namevhost baldockery.com		/etc/apache2/sites-enabled/000-default.conf:1
         port 443 namevhost baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:1
                 alias baldockery.com

         port 80 namevhost enphase.baldockery.com	/etc/apache2/sites-enabled/000-default.conf:72

         port 443 namevhost emonpi.baldockery.com	/etc/apache2/sites-enabled/default-ssl.conf:136

         port 443 namevhost test.baldockery.com		/etc/apache2/sites-enabled/default-ssl.conf:157
1 Like

From the little you show of the error, it is hard to say which section(s) failed to renew, nor even which needed to be renewed.

Please matchup the unmatched sections and correct the redundant alias.
Also, please show:
certbot certificates

1 Like

Can you help me with what you mean by matching the unmatched sections and correcting the redundant alias?

Output of certbot certficates:

Found the following certs:
  Certificate Name: baldockery.com
    Domains: baldockery.com
    Expiry Date: 2020-08-13 22:15:09+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/baldockery.com/privkey.pem
  Certificate Name: dakboard.baldockery.com
    Domains: dakboard.baldockery.com
    Expiry Date: 2020-08-14 08:35:23+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/dakboard.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/dakboard.baldockery.com/privkey.pem
  Certificate Name: emoncmsnodered.baldockery.com
    Domains: emoncmsnodered.baldockery.com
    Expiry Date: 2020-06-15 08:48:17+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/emoncmsnodered.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/emoncmsnodered.baldockery.com/privkey.pem
  Certificate Name: emonpi.baldockery.com
    Domains: emonpi.baldockery.com
    Expiry Date: 2020-06-15 08:48:54+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/emonpi.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/emonpi.baldockery.com/privkey.pem
  Certificate Name: home.baldockery.com
    Domains: home.baldockery.com
    Expiry Date: 2020-08-14 08:36:25+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/home.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/home.baldockery.com/privkey.pem
  Certificate Name: mypi.baldockery.com
    Domains: mypi.baldockery.com
    Expiry Date: 2020-06-15 08:49:51+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/mypi.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/mypi.baldockery.com/privkey.pem
  Certificate Name: nodered.baldockery.com
    Domains: nodered.baldockery.com
    Expiry Date: 2020-08-14 08:36:57+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/nodered.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/nodered.baldockery.com/privkey.pem
  Certificate Name: noderedemoncms.baldockery.com
    Domains: noderedemoncms.baldockery.com
    Expiry Date: 2020-06-15 08:51:36+00:00 (VALID: 18 days)
    Certificate Path: /etc/letsencrypt/live/noderedemoncms.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/noderedemoncms.baldockery.com/privkey.pem
  Certificate Name: openhab.baldockery.com
    Domains: openhab.baldockery.com
    Expiry Date: 2020-08-14 08:37:41+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/openhab.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/openhab.baldockery.com/privkey.pem
  Certificate Name: test.baldockery.com
    Domains: test.baldockery.com
    Expiry Date: 2020-08-14 08:37:53+00:00 (VALID: 78 days)
    Certificate Path: /etc/letsencrypt/live/test.baldockery.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/test.baldockery.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 Like

The redundant alias is simple.
The config has the same server name twice.

Matching the unmatched has to do with making HTTP and HTTPS enabled sections for each of the server names.

1 Like

I corrected the redundant alias and the unmatched sections for the various virtual servers. And I updated the Route53 IP address for the ones that I had let get out-of-date. Now sudo certbot renew runs without errors. I restarted the timer. Hopefully when it next runs apache2 stays up.

2 Likes

I am curious why, when I had these errors in my configuration, I was able to restart apache2, but certbot was not able to restart apache2.

Thanks for all your help with this!

1 Like

May act differently than however you stopped and started apache.

1 Like

True. I use systemctl start apache2. I’ve read that is the better way to do it (https://www.configserverfirewall.com/linux-tutorials/apachectl-command/) but maybe certbot doesn’t want to depend on systemd being in place on various platforms.

1 Like

OK, if you issue that command, does it still complain?
apachectl -k start

1 Like

It still complains.

pi@emonpi:~$ sudo apachectl -k stop
pi@emonpi:~$ sudo apachectl -k start
pi@emonpi:~$ sudo systemctl status apache2
â—Ź apache2.service - The Apache HTTP Server
   Loaded: loaded (/lib/systemd/system/apache2.service; enabled; vendor preset: 
   Active: inactive (dead) since Wed 2020-05-27 17:30:28 CDT; 31s ago
     Docs: https://httpd.apache.org/docs/2.4/
  Process: 8969 ExecStop=/usr/sbin/apachectl stop (code=exited, status=0/SUCCESS
 Main PID: 6583 (code=exited, status=0/SUCCESS)

May 27 12:02:51 emonpi systemd[1]: Starting The Apache HTTP Server...
May 27 12:02:51 emonpi systemd[1]: Started The Apache HTTP Server.
May 27 17:30:28 emonpi apachectl[8969]: httpd (no pid file) not running
May 27 17:30:28 emonpi systemd[1]: apache2.service: Succeeded.
1 Like

hmm…

Does certbot still show that problem?

1 Like

After I corrected my configuration I ran certbot renew manually and it worked, so now when I try to test if certbot will stop and start apache2 successfully, it doesn't ever stop it because none of the certificates are up for renewal.

2 Likes

I have some time to look at this again today, and I'm wondering if you have suggestions on how to check if certbot still shows the problem.

My other question is if I should go back to using cron to periodically run certbot, and use post-hook "systemctl start apache", since systemctl is working for me and apachectl is not. Obviously it would be more satisfying to figure out what is going on with apachectl, but in the meantime is there any harm in going with this cron idea?

1 Like

--post-hook will always execute
--deploy-hook will only execute when the cert is actually renewed [recommended]

How did you get/generate the previous error and output?

1 Like

Thanks! How would deploy-hook be used in a cron command like this? In place of --pre-hook or --post-hook, and then what goes in the other spot?

@weekly certbot renew --authenticator standalone --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2" >> /home/pi/data/certbot.log 2>&1

Before I updated Route53 with my current IP address for a few of those sites that I had let get out of date, every time the systemd certbot service ran it would try to renew the certificates for those sites, bring down apache and try to bring it back up. Two days ago, when I updated Route53 while working on this, I did a certbot renew and they successfully renewed. So now the certbot service doesn't find anything that needs renewing, so it never brings down apache. Thus the best simulation I know of now is to just try "apachectl -k start". When you asked if certbot still showed the problem, I wondered if you knew a way to make it do so.

1 Like

standalone breaks the deploy hook logic.
As it requires you to stop the web service to spin up a new temporary web service.
You really should find a way to use the existing web server (and leave it running at all times).
Then you can use --deploy-hook

1 Like

I'm not 100% certain that it works that way.
Or at least the way you expect and on all versions of certbot.

I would test that out that command and see if it stops/starts the web server even now when it doesn't need to renew anything.

certbot renew --authenticator standalone --pre-hook "systemctl stop apache2" --post-hook "systemctl start apache2"

[or leave the post hook off and see if apache is still running]

I can confirm that in version 0.31.0, certbot does work as you expected and will not stop apache when no renewal is required.

1 Like