Apache ceases to work, no errors from Apache, errors from CertBot

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: www.journeyman.cc

I ran this command: service apache2 restart

It produced this output: (none)

My web server is (include version): apache2 2.4.61-1~deb12u1

The operating system my web server runs on is (include version): Linux notary 6.1.0-18-amd64 #1 SMP PREEMPT_DYNAMIC Debian 6.1.76-1 (2024-02-01) x86_64 GNU/Linux

My hosting provider, if applicable, is: Linode

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.1.0

Problem description

OK, this is not strictly a certbot problem, it's a certbot-adjacent problem. And it's entirely my fault, I deserve no sympathy. But I've messed up my SSL config, and, ironically, I messed it up trying to tighten security.

The default-le-ssl.conf was generated by certbot; it worked. I added a stanza to enable webgit, that worked. I did not keep a copy of the last known-good config. I then tried to password protect webgit, and the default site ceased to serve at all on HTTPS (still serves on HTTP). Apache reports no errors, not even in the log:

root@notary:/etc/apache2/sites-available# apachectl configtest
Syntax OK
root@notary:/etc/apache2/sites-available# service apache2 restart
root@notary:/etc/apache2/sites-available# root@notary:/etc/apache2/sites-available# tail /var/log/apache2/error.log
[Wed Sep 18 08:16:34.773990 2024] [mpm_prefork:notice] [pid 3549146:tid 3549146] AH00170: caught SIGWINCH, shutting down gracefully
[Wed Sep 18 08:16:34.855549 2024] [mpm_prefork:notice] [pid 3549557:tid 3549557] AH00163: Apache/2.4.61 (Debian) mod_jk/1.2.48 OpenSSL/3.0.13 configured -- resuming normal operations
[Wed Sep 18 08:16:34.855651 2024] [core:notice] [pid 3549557:tid 3549557] AH00094: Command line: '/usr/sbin/apache2'

I then commented out the entire gitweb stanza, but this made no difference: same absolute lack of any diagnostic errors or log entries.

Attempting to rerun certbot to regenerate default-le-ssl.conf fails as follows:

root@notary:/etc/apache2/sites-available# certbot --apache -v -d notary.journeyman.cc -d www.journeyman.cc
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Certificate not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn't close to expiry.
(ref: /etc/letsencrypt/renewal/notary.journeyman.cc.conf)

What would you like to do?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Attempt to reinstall this existing certificate
2: Renew & replace the certificate (may be subject to CA rate limits)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Keeping the existing certificate
Deploying certificate
Could not install certificate
An unexpected error occurred:
StopIteration
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

(a very similar failure occurs when I attempt Renew & replace the certificate (may be subject to CA rate limits))

The full default-le-ssl.conf, as it stands now, is as follows:

<VirtualHost *:80>
        ServerAdmin simon@journeyman.cc
        ServerName notary.journeyman.cc
	ServerAlias www.journeyman.cc

	DocumentRoot /var/www/http
	<Directory />
                Options FollowSymLinks
                AllowOverride None
	</Directory>
        <Directory /var/www/http>
		Require all granted
                Options Indexes FollowSymLinks MultiViews
                AllowOverride None
		Order allow,deny
                allow from all
	</Directory>

        ScriptAlias /cgi-bin/ /usr/lib/cgi-bin/
        <Directory "/usr/lib/cgi-bin">
		AllowOverride None
                Options +ExecCGI -MultiViews +SymLinksIfOwnerMatch
	        Order allow,deny
		Allow from all
	</Directory>

        ErrorLog ${APACHE_LOG_DIR}/error.log

	# Possible values include: debug, info, notice, warn, error, crit,
        # alert, emerg.
	LogLevel warn

        CustomLog ${APACHE_LOG_DIR}/access.log combined

        Alias /awstatsclasses "/usr/share/awstats/lib/"
	Alias /awstats-icon/ "/usr/share/awstats/icon/"
	Alias /awstatscss "/usr/share/doc/awstats/examples/css"
        ScriptAlias /awstats/ /usr/lib/cgi-bin/
#       Options ExecCGI -MultiViews +SymLinksIfOwnerMatch

        RewriteEngine on
	RewriteCond %{HTTPS} off

	<IfModule headers_module>
                  header set X-Falling-Outside-the-Normal-Moral-Constraints "Ia\
in M Banks"
                  header set X-Lambda "λάμβδα"
	</IfModule>
#       RewriteCond %{SERVER_NAME} =projecthope.scot [OR]
#       RewriteCond %{SERVER_NAME} =www.projecthope.scot
#       RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
#       Alias /gitweb "/usr/share/gitweb"
#       <Directory "/usr/share/gitweb">
#           <IfModule mod_auth_digest>
#               AuthType Digest
#                AuthName "Journeyman"
#               AuthDigestProvider file
#               AuthUserFile "/etc/apache2/auth/passwords"
#               AuthGroupFile "/etc/apache2/auth/groups"
#               Require group git
#           </IfModule>
#           DirectoryIndex gitweb.cgi
#           Options ExecCGI
#           Require all granted
#           <Files gitweb.cgi>
#                 SetHandler cgi-script
#           </Files>
#           SetEnv  GITWEB_CONFIG  /etc/gitweb.conf
#       </Directory>
</VirtualHost>

Any suggestions gratefully received!

That is far from an SSL config file.

What shows?:
sudo apachectl -t -D DUMP_VHOSTS

2 Likes
simon@notary:~$ sudo apachectl -t -D DUMP_VHOSTS
[sudo] password for simon: 
VirtualHost configuration:
*:443                  www.act.scot (/etc/apache2/sites-enabled/act.scot-le-ssl.conf:2)
*:80                   is a NameVirtualHost
         default server notary.journeyman.cc (/etc/apache2/sites-enabled/000-default-ssl.conf:1)
         port 80 namevhost notary.journeyman.cc (/etc/apache2/sites-enabled/000-default-ssl.conf:1)
                 alias www.journeyman.cc
         port 80 namevhost notary.journeyman.cc (/etc/apache2/sites-enabled/000-default.conf:1)
                 alias www.journeyman.cc
         port 80 namevhost www.act.scot (/etc/apache2/sites-enabled/act.scot-le-ssl.conf:47)
                 alias act.scot
         port 80 namevhost www.act.scot (/etc/apache2/sites-enabled/act.scot.conf:1)
                 alias act.scot

So Apache isn't seeing www.journeyman.cc as a valid hostname for port 443 at all, which is... interesting.

... and that isn't surprising, since the default-le-ssl.conf starts


<VirtualHost *:80>
	ServerAdmin simon@journeyman.cc
	ServerName notary.journeyman.cc
	ServerAlias www.journeyman.cc
	
	DocumentRoot /var/www/http

D'oh! Thanks. So, a quick change of 80 to 443 and of /var/www/http to /var/www/https, restart Apache, find the certificate is broken, rerun certbot and... it works.

Many thanks indeed. The worst problems are always hiding in plain sight!

2 Likes

I'm seeing double:

Double-Double:

2 Likes

So true.

Get rid of the duplicates and you get rid of the problem(s).

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.