How can I prevent a failed renewal from shutting down Apache?

Help
1

My domain is: halapp.com

Hi, I had an unexplained Apache shutdown today, and can't figure out why. Let's Encrypt tried and failed to renew at that time. Can this be related?

If I follow, Let's Encrypt shut down Apache to attempt a renewal, the renewal failed, and then somehow Apache shut down rather than restarting.

I have updated my certbot setup since this incident, and am renewing things fine now, but I want to know if the renewal failure, "libgomp: could not create thread pool destructor." and "AH00169: caught SIGTERM, shutting down" are somehow related. I have found very little online about "libgomp: could not create thread pool destructor." Someone said it related to an ImageMagick bug after 1024 Apache restarts (This Is Why Our 3000 Apache Servers Went Down On The First Day of 2022 | by Ali Josie | Medium), but I have a newer version of ImageMagick so that shouldn't be the issue.

Can you please assist to advise if you know anything about what caused this Apache shutdown and how to prevent a similar issue from recurring? Thanks!!!

My syslog from the relevant moment:
Sep 2 08:35:23 halapp apache2[21520]: * Stopping Apache httpd web server apache2
Sep 2 08:35:23 halapp apache2[21520]: *
Sep 2 08:35:23 halapp certbot.renew[21390]: Failed to renew certificate austinwines.com-0001 with error: Some challenges have failed.
Sep 2 08:35:37 halapp certbot.renew[21390]: Failed to renew certificate austinwines.com with error: Some challenges have failed.
Sep 2 08:35:37 halapp certbot.renew[21390]: All renewals failed. The following certificates could not be renewed:
Sep 2 08:35:37 halapp certbot.renew[21390]: /etc/letsencrypt/live/austinwines.com-0001/fullchain.pem (failure)
Sep 2 08:35:37 halapp certbot.renew[21390]: /etc/letsencrypt/live/austinwines.com/fullchain.pem (failure)

My error.log from the same time:
[Fri Sep 02 08:33:41.299193 2022] [php7:error] [pid 21284] [client 20.118.188.137:55303] script '/var/www/canopywines.com/public_html/xmlrpc.php' not found or unable to stat
[Fri Sep 02 08:35:15.097193 2022] [mpm_prefork:notice] [pid 436] AH00171: Graceful restart requested, doing restart
[Fri Sep 02 08:35:15.169893 2022] [mpm_prefork:notice] [pid 436] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations
[Fri Sep 02 08:35:15.169908 2022] [core:notice] [pid 436] AH00094: Command line: '/usr/sbin/apache2'
[Fri Sep 02 08:35:23.723106 2022] [mpm_prefork:notice] [pid 436] AH00171: Graceful restart requested, doing restart

libgomp: could not create thread pool destructor.
[Fri Sep 02 08:35:29.247220 2022] [core:warn] [pid 21552] AH00098: pid file /var/run/apache2/apache2.pid overwritten -- Unclean shutdown of previous Apache run?
[Fri Sep 02 08:35:29.252487 2022] [mpm_prefork:notice] [pid 21552] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations
[Fri Sep 02 08:35:29.252513 2022] [core:notice] [pid 21552] AH00094: Command line: '/usr/sbin/apache2'
[Fri Sep 02 08:35:37.576308 2022] [mpm_prefork:notice] [pid 21552] AH00171: Graceful restart requested, doing restart
[Fri Sep 02 08:35:37.638903 2022] [mpm_prefork:notice] [pid 21552] AH00163: Apache/2.4.18 (Ubuntu) OpenSSL/1.0.2g configured -- resuming normal operations
[Fri Sep 02 08:35:37.638919 2022] [core:notice] [pid 21552] AH00094: Command line: '/usr/sbin/apache2'
[Fri Sep 02 08:35:37.927077 2022] [mpm_prefork:notice] [pid 21552] AH00169: caught SIGTERM, shutting down

My web server is (include version): Apache/2.4.18 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 16.04

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.29.0

2

Hi @katillery, and welcome to the LE community forum :slight_smile:

Whenever I see:

and also:

I can't help but to think that something has gone off the rails.

Please show the output of:
apachectl -t -D DUMP_VHOSTS

3 Likes
3

Usually, one does not require Apache to stop for a certificate renewal. Are you using the --standalone authenticator plugin in Certbot by any chance? If so, I recommend to change the authenticator plugin to either the --apache plugin or the --webroot plugin. See more about the different Certbot plugins here: User Guide — Certbot 1.29.0 documentation

3 Likes
4

Sure thing:

VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com-le-ssl.conf:2)
         port 443 namevhost bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com-le-ssl.conf:2)
                 alias www.bloemwine.com
                 alias bloemwines.com
                 alias www.bloemwines.com
         port 443 namevhost canopyclearing.com (/etc/apache2/sites-enabled/canopyclearing.com-le-ssl.conf:2)
                 alias www.canopyclearing.com
         port 443 namevhost canopywineclub.com (/etc/apache2/sites-enabled/canopywineclub.com-le-ssl.conf:2)
                 alias www.canopywineclub.com
         port 443 namevhost canopywines.com (/etc/apache2/sites-enabled/canopywines.com.conf:1)
                 alias www.canopywines.com
                 alias invoer.com
                 alias www.invoer.com
                 alias invoerekke.com
                 alias www.invoerekke.com
         port 443 namevhost cellardoornigeria.com (/etc/apache2/sites-enabled/cellardoornigeria.com-le-ssl.conf:2)
                 alias www.cellardoornigeria.com
         port 443 namevhost cosecharestaurant.com (/etc/apache2/sites-enabled/cosecharestaurant.com-le-ssl.conf:2)
                 alias www.cosecharestaurant.com
         port 443 namevhost halapp.com (/etc/apache2/sites-enabled/halapp.com.conf:1)
                 alias www.halapp.com
                 alias heuristic.cc
                 alias www.heuristic.cc
                 alias quickcrm.app
                 alias www.quickcrm.app
         port 443 namevhost kinleywine.com (/etc/apache2/sites-enabled/kinleywine.com-le-ssl.conf:2)
                 alias www.kinleywine.com
         port 443 namevhost liqr.app (/etc/apache2/sites-enabled/liqr.app-le-ssl.conf:2)
                 alias www.liqr.app
         port 443 namevhost noblehill.com (/etc/apache2/sites-enabled/noblehill.com-le-ssl.conf:2)
                 alias www.noblehill.com
                 alias austinwines.co.za
                 alias www.austinwines.co.za
                 alias austinwines.com
                 alias www.austinwines.com
                 alias noblehill.co.za
                 alias www.noblehill.co.za
                 alias noblehill.de
                 alias www.noblehill.de
                 alias noblehillvineyards.com
                 alias www.noblehillvineyards.com
                 alias noblehillwines.com
                 alias www.noblehillwines.com
                 alias thenoblehill.com
                 alias www.thenoblehill.com
         port 443 namevhost noblehill.com (/etc/apache2/sites-enabled/noblehill.com.conf:1)
                 alias www.noblehill.com
         port 443 namevhost simonsbergwine.com (/etc/apache2/sites-enabled/simonsbergwine.com-le-ssl.conf:2)
                 alias www.simonsbergwine.com
                 alias paarlvineyards.com
                 alias www.paarlvineyards.com
                 alias paarlwines.com
                 alias www.paarlwines.com
                 alias simondium.com
                 alias www.simondium.com
                 alias simonsbergwines.com
                 alias www.simonsbergwines.com
                 alias winesofthesimonsberg.com
                 alias www.winesofthesimonsberg.com
         port 443 namevhost simonskop.com (/etc/apache2/sites-enabled/simonskop.com-le-ssl.conf:2)
                 alias www.simonskop.com
         port 443 namevhost vtoliveoil.com (/etc/apache2/sites-enabled/vtoliveoil.com-le-ssl.conf:2)
                 alias www.vtoliveoil.com
                 alias virginterritoryoliveoil.com
                 alias www.virginterritoryoliveoil.com
*:80                   is a NameVirtualHost
         default server bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com-le-ssl.conf:42)
         port 80 namevhost bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com-le-ssl.conf:42)
                 alias www.bloemwine.com
                 alias bloemwines.com
                 alias www.bloemwines.com
         port 80 namevhost bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com.conf:1)
                 alias www.bloemwine.com
                 alias bloemwines.com
                 alias www.bloemwines.com
         port 80 namevhost canopyclearing.com (/etc/apache2/sites-enabled/canopyclearing.com-le-ssl.conf:40)
                 alias www.canopyclearing.com
         port 80 namevhost canopyclearing.com (/etc/apache2/sites-enabled/canopyclearing.com.conf:1)
                 alias www.canopyclearing.com
         port 80 namevhost canopywineclub.com (/etc/apache2/sites-enabled/canopywineclub.com-le-ssl.conf:40)
                 alias www.canopywineclub.com
         port 80 namevhost canopywineclub.com (/etc/apache2/sites-enabled/canopywineclub.com.conf:1)
                 alias www.canopywineclub.com
         port 80 namevhost canopywines.com (/etc/apache2/sites-enabled/canopywines.com.conf:42)
                 alias www.canopywines.com
                 alias invoer.com
                 alias www.invoer.com
                 alias invoerekke.com
                 alias www.invoerekke.com
         port 80 namevhost cellardoornigeria.com (/etc/apache2/sites-enabled/cellardoornigeria.com-le-ssl.conf:40)
                 alias www.cellardoornigeria.com
         port 80 namevhost cellardoornigeria.com (/etc/apache2/sites-enabled/cellardoornigeria.com.conf:1)
                 alias www.cellardoornigeria.com
         port 80 namevhost cosecharestaurant.com (/etc/apache2/sites-enabled/cosecharestaurant.com-le-ssl.conf:40)
                 alias www.cosecharestaurant.com
         port 80 namevhost cosecharestaurant.com (/etc/apache2/sites-enabled/cosecharestaurant.com.conf:1)
                 alias www.cosecharestaurant.com
         port 80 namevhost halapp.com (/etc/apache2/sites-enabled/halapp.com.conf:42)
                 alias www.halapp.com
                 alias heuristic.cc
                 alias www.heuristic.cc
                 alias quickcrm.app
                 alias www.quickcrm.app
         port 80 namevhost kinleywine.com (/etc/apache2/sites-enabled/kinleywine.com-le-ssl.conf:42)
                 alias www.kinleywine.com
         port 80 namevhost kinleywine.com (/etc/apache2/sites-enabled/kinleywine.com.conf:1)
                 alias www.kinleywine.com
         port 80 namevhost liqr.app (/etc/apache2/sites-enabled/liqr.app.conf:1)
                 alias www.liqr.app
         port 80 namevhost noblehill.com (/etc/apache2/sites-enabled/noblehill.com-le-ssl.conf:28)
                 alias www.noblehill.com
                 alias austinwines.co.za
                 alias www.austinwines.co.za
                 alias austinwines.com
                 alias www.austinwines.com
                 alias noblehill.co.za
                 alias www.noblehill.co.za
                 alias noblehill.de
                 alias www.noblehill.de
                 alias noblehillvineyards.com
                 alias www.noblehillvineyards.com
                 alias noblehillwines.com
                 alias www.noblehillwines.com
                 alias thenoblehill.com
                 alias www.thenoblehill.com
         port 80 namevhost noblehill.com (/etc/apache2/sites-enabled/noblehill.com.conf:38)
                 alias www.noblehill.com
                 alias austinwines.co.za
                 alias www.austinwines.co.za
                 alias austinwines.com
                 alias www.austinwines.com
                 alias noblehill.co.za

truncated because "new users can only put 120 links in messages"

5

I use the --apache plugin already, thanks!

6

Hmm, the --apache plugin would not trigger a full Apache stop like it says in your logs? :roll_eyes:

2 Likes
7

Can you slice that up and post it all [at least the port 80 part]?
OR use backticks before and after the content.
Like:
```
text
```

3 Likes
8

Possibly a restart crash?

3 Likes
9

That wouldn't put a "Sep 2 08:35:23 halapp apache2[21520]: * Stopping Apache httpd web server apache2" line in the logs I think? This looks like a purposely tiggered stop of Apache. As far as I know, a graceful restart, as Certbot initiates, would not trigger such a thing/log entry.

4 Likes
10

Here's the rest of that output:

alias www.noblehill.co.za
                 alias noblehill.de
                 alias www.noblehill.de
                 alias noblehillvineyards.com
                 alias www.noblehillvineyards.com
                 alias noblehillwines.com
                 alias www.noblehillwines.com
                 alias thenoblehill.com
                 alias www.thenoblehill.com
         port 80 namevhost simonsbergwine.com (/etc/apache2/sites-enabled/simonsbergwine.com-le-ssl.conf:50)
                 alias www.simonsbergwine.com
                 alias paarlvineyards.com
                 alias www.paarlvineyards.com
                 alias paarlwines.com
                 alias www.paarlwines.com
                 alias simondium.com
                 alias www.simondium.com
                 alias simonsbergwines.com
                 alias www.simonsbergwines.com
                 alias winesofthesimonsberg.com
                 alias www.winesofthesimonsberg.com
         port 80 namevhost simonsbergwine.com (/etc/apache2/sites-enabled/simonsbergwine.com.conf:1)
                 alias www.simonsbergwine.com
                 alias paarlvineyards.com
                 alias www.paarlvineyards.com
                 alias paarlwines.com
                 alias www.paarlwines.com
                 alias simondium.com
                 alias www.simondium.com
                 alias simonsbergwines.com
                 alias www.simonsbergwines.com
                 alias winesofthesimonsberg.com
                 alias www.winesofthesimonsberg.com
         port 80 namevhost simonskop.com (/etc/apache2/sites-enabled/simonskop.com.conf:1)
                 alias www.simonskop.com
         port 80 namevhost vtoliveoil.com (/etc/apache2/sites-enabled/vtoliveoil.com-le-ssl.conf:42)
                 alias www.vtoliveoil.com
                 alias virginterritoryoliveoil.com
                 alias www.virginterritoryoliveoil.com
         port 80 namevhost vtoliveoil.com (/etc/apache2/sites-enabled/vtoliveoil.com.conf:1)
                 alias www.vtoliveoil.com
                 alias virginterritoryoliveoil.com
                 alias www.virginterritoryoliveoil.com
         port 80 namevhost 216.70.112.150 (/etc/apache2/sites-enabled/zzz.conf:1)
                 wild alias *
11

Hi, I'm following, but I'm a bit lost as to why Apache restarted unless related to Let's Encrypt... the syslog timing doesn't show anything else happening at that time, and Let's Encrypt failing to renew exactly at that time:

Including a few more lines from syslog to show what was happening up to that moment:
Sep 2 08:32:31 halapp postfix/smtp[21439]: 3A71B40162: to=notifications@halapp.com, relay=ASPMX.L.GOOGLE.com[172.253.115.27]:25, delay=31, delays=0.03/0.03/31/0.35, dsn=2.0.0, status=sent (250 2.0.0 OK 1662107551 s8-20020a0cf648000000b0049914aeda27si533558qvm.108 - gsmtp)
Sep 2 08:32:31 halapp postfix/qmgr[587]: 3A71B40162: removed
Sep 2 08:33:01 halapp CRON[21454]: (root) CMD (php /var/www/halapp.com/cron/hal_cron_task_action.php)
Sep 2 08:33:01 halapp CRON[21455]: (root) CMD (php /var/www/halapp.com/cron/hal_cron_task_add.php)
Sep 2 08:34:01 halapp CRON[21467]: (root) CMD (php /var/www/halapp.com/cron/hal_cron_task_add.php)
Sep 2 08:34:01 halapp CRON[21468]: (root) CMD (php /var/www/halapp.com/cron/hal_cron_task_action.php)
Sep 2 08:35:01 halapp CRON[21478]: (root) CMD (php /var/www/halapp.com/cron/hal_cron_task_action.php)
Sep 2 08:35:01 halapp CRON[21477]: (root) CMD (php /var/www/halapp.com/cron/hal_cron_task_add.php)
Sep 2 08:35:23 halapp apache2[21520]: * Stopping Apache httpd web server apache2
Sep 2 08:35:23 halapp apache2[21520]: *
Sep 2 08:35:23 halapp certbot.renew[21390]: Failed to renew certificate austinwines.com-0001 with error: Some challenges have failed.

The real concern I have from Apache is "libgomp: could not create thread pool destructor." because this seemed to prevent the restart from completing successfully. But I have no idea how to pursue this error further, as the only thing I could find was an instruction to update ImageMagick to a version I am already using...

12

There are some name:port overlaps:

         port 80 namevhost bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com-le-ssl.conf:42)
                 alias www.bloemwine.com
                 alias     bloemwines.com
                 alias www.bloemwines.com
         port 80 namevhost bloemwine.com (/etc/apache2/sites-enabled/bloemwine.com.conf:1)
                 alias www.bloemwine.com
                 alias     bloemwines.com
                 alias www.bloemwines.com

         port 80 namevhost canopyclearing.com (/etc/apache2/sites-enabled/canopyclearing.com-le-ssl.conf:40)
                 alias www.canopyclearing.com
         port 80 namevhost canopyclearing.com (/etc/apache2/sites-enabled/canopyclearing.com.conf:1)
                 alias www.canopyclearing.com

         port 80 namevhost canopywineclub.com (/etc/apache2/sites-enabled/canopywineclub.com-le-ssl.conf:40)
                 alias www.canopywineclub.com
         port 80 namevhost canopywineclub.com (/etc/apache2/sites-enabled/canopywineclub.com.conf:1)
                 alias www.canopywineclub.com

         port 80 namevhost cellardoornigeria.com (/etc/apache2/sites-enabled/cellardoornigeria.com-le-ssl.conf:40)
                 alias www.cellardoornigeria.com
         port 80 namevhost cellardoornigeria.com (/etc/apache2/sites-enabled/cellardoornigeria.com.conf:1)
                 alias www.cellardoornigeria.com

         port 80 namevhost cosecharestaurant.com (/etc/apache2/sites-enabled/cosecharestaurant.com-le-ssl.conf:40)
                 alias www.cosecharestaurant.com
         port 80 namevhost cosecharestaurant.com (/etc/apache2/sites-enabled/cosecharestaurant.com.conf:1)
                 alias www.cosecharestaurant.com

         port 80 namevhost noblehill.com (/etc/apache2/sites-enabled/noblehill.com-le-ssl.conf:28)
                 alias www.noblehill.com
                 alias     austinwines.co.za
                 alias www.austinwines.co.za
                 alias     austinwines.com
                 alias www.austinwines.com
                 alias     noblehill.co.za
                 alias www.noblehill.co.za
                 alias     noblehill.de
                 alias www.noblehill.de
                 alias     noblehillvineyards.com
                 alias www.noblehillvineyards.com
                 alias     noblehillwines.com
                 alias www.noblehillwines.com
                 alias     thenoblehill.com
                 alias www.thenoblehill.com
         port 80 namevhost noblehill.com (/etc/apache2/sites-enabled/noblehill.com.conf:38)
                 alias www.noblehill.com
                 alias     austinwines.co.za
                 alias www.austinwines.co.za
                 alias     austinwines.com
                 alias www.austinwines.com
                 alias     noblehill.co.za
                 alias www.noblehill.co.za
                 alias     noblehill.de
                 alias www.noblehill.de
                 alias     noblehillvineyards.com
                 alias www.noblehillvineyards.com
                 alias     noblehillwines.com
                 alias www.noblehillwines.com
                 alias     thenoblehill.com
                 alias www.thenoblehill.com

         port 80 namevhost simonsbergwine.com (/etc/apache2/sites-enabled/simonsbergwine.com-le-ssl.conf:50)
                 alias www.simonsbergwine.com
                 alias     paarlvineyards.com
                 alias www.paarlvineyards.com
                 alias     paarlwines.com
                 alias www.paarlwines.com
                 alias     simondium.com
                 alias www.simondium.com
                 alias     simonsbergwines.com
                 alias www.simonsbergwines.com
                 alias     winesofthesimonsberg.com
                 alias www.winesofthesimonsberg.com
         port 80 namevhost simonsbergwine.com (/etc/apache2/sites-enabled/simonsbergwine.com.conf:1)
                 alias www.simonsbergwine.com
                 alias     paarlvineyards.com
                 alias www.paarlvineyards.com
                 alias     paarlwines.com
                 alias www.paarlwines.com
                 alias     simondium.com
                 alias www.simondium.com
                 alias     simonsbergwines.com
                 alias www.simonsbergwines.com
                 alias     winesofthesimonsberg.com
                 alias www.winesofthesimonsberg.com

         port 80 namevhost kinleywine.com (/etc/apache2/sites-enabled/kinleywine.com-le-ssl.conf:42)
                 alias www.kinleywine.com
         port 80 namevhost kinleywine.com (/etc/apache2/sites-enabled/kinleywine.com.conf:1)
                 alias www.kinleywine.com

         port 80 namevhost vtoliveoil.com (/etc/apache2/sites-enabled/vtoliveoil.com-le-ssl.conf:42)
                 alias www.vtoliveoil.com
                 alias     virginterritoryoliveoil.com
                 alias www.virginterritoryoliveoil.com
         port 80 namevhost vtoliveoil.com (/etc/apache2/sites-enabled/vtoliveoil.com.conf:1)
                 alias www.vtoliveoil.com
                 alias     virginterritoryoliveoil.com
                 alias www.virginterritoryoliveoil.com
3 Likes
13

Do you want me to remove the "-le-ssl.conf" files or something? I'm a bit confused about them since they were generated by certbot, not by me...

For clarity, the setup is running and renewing 100% fine as at now.

The problem is a a restart which failed and caused Apache on my server to stop running. The error in the restart seems to be "libgomp: could not create thread pool destructor."

14

Show this file.
[let's see what's going on in there]
/etc/apache2/sites-enabled/vtoliveoil.com-le-ssl.conf

3 Likes
15 
<IfModule mod_ssl.c>
<VirtualHost *:443>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	#ServerName www.example.com

	ServerAdmin katillery@gmail.com
	ServerName vtoliveoil.com
	ServerAlias www.vtoliveoil.com
        ServerAlias virginterritoryoliveoil.com
	ServerAlias www.virginterritoryoliveoil.com
        DocumentRoot /var/www/vtoliveoil.com/public_html

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf


Include /etc/letsencrypt/options-ssl-apache.conf
SSLCertificateFile /etc/letsencrypt/live/vtoliveoil.com/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/vtoliveoil.com/privkey.pem
</VirtualHost>
</IfModule>
<IfModule mod_ssl.c>
<VirtualHost *:80>
	# The ServerName directive sets the request scheme, hostname and port that
	# the server uses to identify itself. This is used when creating
	# redirection URLs. In the context of virtual hosts, the ServerName
	# specifies what hostname must appear in the request's Host: header to
	# match this virtual host. For the default virtual host (this file) this
	# value is not decisive as it is used as a last resort host regardless.
	# However, you must set it for any further virtual host explicitly.
	#ServerName www.example.com

	ServerAdmin katillery@gmail.com
	ServerName vtoliveoil.com
	ServerAlias www.vtoliveoil.com
        ServerAlias virginterritoryoliveoil.com
	ServerAlias www.virginterritoryoliveoil.com
        DocumentRoot /var/www/vtoliveoil.com/public_html

	# Available loglevels: trace8, ..., trace1, debug, info, notice, warn,
	# error, crit, alert, emerg.
	# It is also possible to configure the loglevel for particular
	# modules, e.g.
	#LogLevel info ssl:warn

	ErrorLog ${APACHE_LOG_DIR}/error.log
	CustomLog ${APACHE_LOG_DIR}/access.log combined

	# For most configuration files from conf-available/, which are
	# enabled or disabled at a global level, it is possible to
	# include a line for only one particular virtual host. For example the
	# following line enables the CGI configuration for this host only
	# after it has been globally disabled with "a2disconf".
	#Include conf-available/serve-cgi-bin.conf
RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =www.virginterritoryoliveoil.com [OR]
# RewriteCond %{SERVER_NAME} =www.vtoliveoil.com [OR]
# RewriteCond %{SERVER_NAME} =virginterritoryoliveoil.com [OR]
# RewriteCond %{SERVER_NAME} =vtoliveoil.com
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]


</VirtualHost>
</IfModule>
16

@katillery I agree that the certbot graceful restart is the likely cause of the Apache Stopped message. My working theory (which could be wrong) is that the graceful failed with that libgomp error and that systemd stopped Apache as a result. It looks like systemd then retried your certbot command and it failed again.

As background, certbot apache plug-in will make temp changes to your Apache conf and do a graceful restart. The cert request / challenges are done and then certbot removes the temp changes and does another graceful restart. This second restart will pick up a new cert if issued (and remove the temp changes). Interestingly, it looks like just the second graceful failed.

Now, what should be done? There are a couple problems. Certainly Apache failing is a key but you also have overlapping domain names / ports that must be fixed. It looks like you also have redundant certs in the certbot folders.

Anyway, can you do a graceful restart from the command line? Does that work? Can you do repeated graceful restarts within seconds of each other?

If you can manually restart the best approach is probably to convert to using webroot authentication and quit using the apache plug-in. Given the, um, messy config state and the number of domains that will take some effort and likely need some Apache downtime.

Maybe another volunteer sees a better way but if it was my system that's what I'd do. First step is try repeated graceful restarts from command prompt.

4 Likes
17

Thanks for your response!

I can definitely restart gracefully normally:

user@halapp:~$ cd /var/www/
user@halapp:/var/www$ sudo apachectl -k graceful
[sudo] password for user: 
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful
user@halapp:/var/www$ sudo apachectl -k graceful

So that is good news.

The overlapping domain/ports I am confused about only because it was all very tidy until certbot was installed. So I don't want to point fingers, but there were never multiple .conf files for the same domain and files with "-le-ssl" in the filename until certbot was installed.

I'm happy to try to go in and clean up, but I don't want to delete something that is needed by certbot, since the duplicative conflicting stuff was added by certbot.

Basically each domain has a <VirtualHost> for port 80 in the domain name file, and a two <VirtualHost>s for ports 80 and 443 in the "le-ssl" file. I'm just not sure which one to delete?

Should I disable/remove the sites that are NOT "le-ssl" now that certbot is installed? Sorry if I am clueless on this point, I didn't see it covered in the setup guide...

Thanks for your help!

1 Like
18

That's possible. The Apache plug-in not only gets a cert but also configures your HTTPS VirtualHost(s). It does this by using your HTTP VirtualHost as a model and looks like something went wrong. That doesn't happen often with the current certbot (which you use) so not sure why.

If we avoid the plugin and use certbot certonly with webroot it will only obtain a cert. You will need to configure the HTTPS VirtualHost(s) yourself. That is, once you have a working apache config you don't have to worry about certbot changing it using certonly webroot.

I don't see a way forward with the apache plug-in but maybe another volunteer will know a trick to avoid the libgomp fault.

Your config is fairly involved to convert to webroot. It is worth waiting to see if any other ideas appear. It is a national US holiday today so responses may not be as prompt. Maybe even the @certbot-devs have insight on why the graceful restart works in command prompt but faults from the plug-in.

In the meantime, you could try posting on apache or even imagemagick forums to see if any other ideas appear for that fault. I didn't see anything from a quick google.

3 Likes
20

Why is the HTTP vhost also wrapped in a required IF Module MOD_SSL?
[probably won't fix anything by removing it, but it just seems completely unnecessary]

As for the actual problem, I would switch to using --webroot authentication OR nginx - LOL

3 Likes
21

I don't have much to comment without first seeing the /var/log/letsencrypt.log from the failed run.

I agree that the syslog is mega suspicious and that it looks very much like somebody/something has invoked the sysvinit script to stop Apache. Maybe a --pre-hook, but it's for sure not part of the Apache plugin's stock behavior.

6 Likes