FreeBSD: sudden failures to start Apache

Help
1

My domain is: pepak.net

I ran this command: /usr/local/bin/certbot renew as root

It produced this output:

Processing /usr/local/etc/letsencrypt/renewal/pepak.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Unable to read ssl_module file; not disabling session tickets.
Renewing an existing certificate for forum.pepak.net and 5 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: forum.pepak.net
  Type:   connection
  Detail: 185.216.75.212: Fetching http://forum.pepak.net/.well-known/acme-challenge/jp9rnsg4JR8Awlzr4I-MOUWhPS1bE7BzXLw_6rue3Ug: Connection refused

  Domain: riva.pepak.net
  Type:   connection
  Detail: 185.216.75.212: Fetching http://riva.pepak.net/.well-known/acme-challenge/iOuDW2O6w9eQQui-0lRoekWnIxaS9ay04KrILbULQX8: Connection refused

  Domain: temp.pepak.net
  Type:   connection
  Detail: 185.216.75.212: Fetching http://temp.pepak.net/.well-known/acme-challenge/68_yFfXaFCGM_SFrUEYR_nhKGBl9vW2C5fW-a2T1_R8: Connection refused

  Domain: update.pepak.net
  Type:   connection
  Detail: 185.216.75.212: Fetching http://update.pepak.net/.well-known/acme-challenge/sA8_9gHaNM-YcJmmx_B7gaMeFo1J0VA5gWU9vOcKLaM: Connection refused

  Domain: www.pepak.net
  Type:   connection
  Detail: 185.216.75.212: Fetching http://www.pepak.net/.well-known/acme-challenge/2Yn5Ugg0yBWhYoJeeClY5Ovi6W9zEU0CZZkfSU24aB8: Connection refused

  Domain: ytd.pepak.net
  Type:   connection
  Detail: 185.216.75.212: Fetching http://ytd.pepak.net/.well-known/acme-challenge/YFuF9dm-aMsVmMrksGW0VV1kNWm1VSXjJSifmwdyOcA: Connection refused

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).


Unable to restart apache using ['apachectl', 'graceful']
Encountered exception during recovery: certbot.errors.MisconfigurationError: Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).
Failed to renew certificate pepak.net with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /usr/local/etc/letsencrypt/live/pepak.net/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

My web server is (include version): 2.4.56

The operating system my web server runs on is (include version): FreeBSD 13.2

My hosting provider, if applicable, is: Contabo

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.4.0

Looking through the log file, it seems that the problem is that after DEBUG:certbot.reverter:Creating backup of /usr/local/etc/apache24/extra/httpd-vhosts.conf, Certbot fails to restart Apache (ERROR:certbot.util:Error while running apachectl graceful.). Which would explain why the Let's Encrypt server is complaining that connection to my site are refused and why the webserver stops. What I don't understand is why would this change suddenly occur, the renewal had worked fine for at least a year or more until a short time ago. A manual apachectl graceful works perfectly with no complaints from Apache, but the next time I try to renew the certificates, I get the same error. I expect certbot writes something into the configuration files that prevents Apache from restarting and then reverts the change so that my manual restarts work, but how can I fugure out what exactly it is? The certbot logfile doesn't seem to be telling me that:

2023-05-28 05:00:57,193:DEBUG:certbot_apache._internal.http_01:Adding a temporary challenge validation Include for name: None in: /usr/local/etc/apache24/extra/httpd-vhosts.conf
2023-05-28 05:00:57,193:DEBUG:certbot_apache._internal.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/db/letsencrypt/http_challenges/$1 [END]
    
2023-05-28 05:00:57,194:DEBUG:certbot_apache._internal.http_01:writing a post config file with text:
         <Directory /var/db/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>
    
2023-05-28 05:00:57,225:DEBUG:certbot.reverter:Creating backup of /usr/local/etc/apache24/extra/httpd-vhosts.conf
2023-05-28 05:00:57,461:ERROR:certbot.util:Error while running apachectl graceful.
apache24 not running? (check /var/run/httpd.pid).

The apache error log doesn't seem to contain anything useful either:

[Sun May 28 05:12:51.957833 2023] [ssl:warn] [pid 77624] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:12:51.993588 2023] [ssl:warn] [pid 77625] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:12:52.122037 2023] [core:warn] [pid 77625] AH00098: pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Sun May 28 05:12:52.270672 2023] [mpm_prefork:notice] [pid 77625] AH00163: Apache/2.4.56 (FreeBSD) OpenSSL/1.1.1t-freebsd configured -- resuming normal operations
[Sun May 28 05:12:52.270807 2023] [core:notice] [pid 77625] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Sun May 28 05:13:00.225675 2023] [mpm_prefork:notice] [pid 77625] AH00171: Graceful restart requested, doing restart
[Sun May 28 05:13:00.267516 2023] [ssl:warn] [pid 77625] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:14:13.469579 2023] [ssl:warn] [pid 77749] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:14:13.509851 2023] [ssl:warn] [pid 77750] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:14:13.631389 2023] [core:warn] [pid 77750] AH00098: pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Sun May 28 05:14:13.720380 2023] [mpm_prefork:notice] [pid 77750] AH00163: Apache/2.4.56 (FreeBSD) OpenSSL/1.1.1t-freebsd configured -- resuming normal operations
[Sun May 28 05:14:13.720498 2023] [core:notice] [pid 77750] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Sun May 28 05:14:19.189581 2023] [mpm_prefork:notice] [pid 77750] AH00171: Graceful restart requested, doing restart
[Sun May 28 05:14:19.229678 2023] [ssl:warn] [pid 77750] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:16:02.025232 2023] [ssl:warn] [pid 77809] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:16:02.047596 2023] [ssl:warn] [pid 77810] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
[Sun May 28 05:16:02.119452 2023] [core:warn] [pid 77810] AH00098: pid file /var/run/httpd.pid overwritten -- Unclean shutdown of previous Apache run?
[Sun May 28 05:16:02.193223 2023] [mpm_prefork:notice] [pid 77810] AH00163: Apache/2.4.56 (FreeBSD) OpenSSL/1.1.1t-freebsd configured -- resuming normal operations
[Sun May 28 05:16:02.193392 2023] [core:notice] [pid 77810] AH00094: Command line: '/usr/local/sbin/httpd -D NOHTTPACCEPT'
[Sun May 28 05:16:21.357071 2023] [mpm_prefork:notice] [pid 77810] AH00171: Graceful restart requested, doing restart
[Sun May 28 05:16:21.400894 2023] [ssl:warn] [pid 77810] AH01909: pepak.net:443:0 server certificate does NOT include an ID which matches the server name
2

This looks a little bit weird to me: a Python "None" value as "name"?

Futher more, to debug this issue you might want to use --debug-challenges, which pauses Certbot at the moment the challenge(s) has/have loaded (i.e.: made the changes to Apache), but before it instructs the ACME server to validate them.

While Certbot is paused, you can try to debug the issue within Apache further.

3 Likes
3

Thank you. This helped, to a degree. Apparently, certbot started to modify my files in a way that is incompatible with my apache setup: instead of copying all the domains again and again, I created a macro (shortened to just the HTTP port):

<Macro VHost $domain $docdir $aliases $maindomain>
	<VirtualHost *:80>
		Use VHostSettings "$domain" "$docdir" "$aliases" "$maindomain"
	</VirtualHost>
</Macro>

Certbot replaces that with:

<Macro VHost $domain $docdir $aliases $maindomain>
	<VirtualHost *:80>
		Include /usr/local/etc/apache24/le_http_01_challenge_pre.conf
Use VHostSettings "$domain""$docdir""$aliases""$maindomain"
Include /usr/local/etc/apache24/le_http_01_challenge_post.conf
	</VirtualHost>
</Macro>

which obviously doesn't work because of the missing spaces on the Use VHostSettings line. When I added them back, everything started to work and I got the new certificates, which prevents me from debugging the issue further. I expect I will need to return to it in two weeks.

The question is, what would be the best way of solving this issue? I would prefer if I didn't have to remove the macro. I am thinking of adding the contents of le_http_01_challenge_pre.conf and le_http_01_challenge_post.conf manually, it doesn't seem like it would be harmful for the server, but how do I prevent certbot from adding the same thing again when run? Should I report an issue? It seems like a bug when certbot removes spaces from my virtual host definitions...

1 Like
4

I'm not a Certbot expert, but I think those spaces should not be removed by Certbot. I'd recommend opening an issue on th Github repo (Issues · certbot/certbot · GitHub) with preferably a short Apache configuration file demonstrating the bug so the developers can easily reproduce this.

3 Likes
5

Use the --webroot method instead of --apache and configure the VHosts for port 443 manually

3 Likes
6

Thank you both.

3 Likes
7

Reported: Certbot 2.4.0 + Apache: incorrect removal of spaces in httpd-vhosts.conf · Issue #9701 · certbot/certbot · GitHub

2 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.