Site crashes when I add ssl certificate but Apache still runs fine

asle · October 30, 2020, 2:06pm

I can run certbot on domains and get success message but then I can not reach the site(s). I am running Apache/2.4.46 installed with Homebrew on OS X 10.13.6. This worked before on OS X 10.12. I think this started failing when I upgraded to 10.13. I have upgraded HTTPD from homebrew after OS X 10.13 upgrade. When I run certbot everything seems ok. Then I can not access the domain. Apache does not crash but the domain with SSL does not work.

I see no error in error_log so I have no idea what is happening. I have to remove SSL from the domain to get it up running again. Since Apache is not crashing but only the domain, how can I find out what is wrong?

My web server is: Apache/2.4.46

The operating system my web server runs on is: Mac OS X 10.13.6

I can login to a root shell on my machine: yes

The version of my client is: certbot 1.9.0 (from homebrew)

Here is my config file (now I have disabled SSL)
<VirtualHost 85.196.85.68:80>
ServerName borchbio.no
ServerAlias www.borchbio.no
ServerAdmin hostmaster@borchbio.no
DocumentRoot "/Library/WebServer/Documents/BorchBio"
CustomLog "/var/log/apache2/borchbio.no/access_log" common
ErrorLog "/var/log/apache2/borchbio.no/error_log"
DirectoryIndex index.html index.php index.htm
Redirect "/narin/" "https://biofokus.no/narin/"
<Directory "/Library/WebServer/Documents/BorchBio">
Options All +MultiViews -ExecCGI -Indexes
AllowOverride AuthConfig
Order deny,allow
Allow from all
<IfModule mod_dav.c>
DAV Off
</IfModule>
</Directory>
RewriteEngine on
RewriteCond %{SERVER_NAME} =borchbio.no [OR]
RewriteCond %{SERVER_NAME} =www.borchbio.no
RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
<VirtualHost 85.196.85.68:443>
ServerName borchbio.no
ServerAlias www.borchbio.no
ServerAdmin hostmaster@borchbio.no
DocumentRoot "/Library/WebServer/Documents/BorchBio"
Redirect "/narin/" "https://biofokus.no/narin/"
CustomLog /var/log/apache2/borchbio.no/access_log example
ErrorLog /var/log/apache2/borchbio.no/error_log
DirectoryIndex index.html index.php index.htm
<Directory "/Library/WebServer/Documents/BorchBio">
Options All +MultiViews -ExecCGI -Indexes
AllowOverride AuthConfig
Order deny,allow
Allow from all
<IfModule mod_dav.c>
DAV Off
</IfModule>
</Directory>
SSLCertificateFile /etc/letsencrypt/live/borchbio.no/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/borchbio.no/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>

rg305 · October 30, 2020, 2:44pm

Please show:
apachectl -S

This may not be doing what you might expect:

Unless that IP is actually on the system (not a NAT), it will not bind correctly.
Replace that with:
<VirtualHost *:443>

READERS: Get involved and participate: If you read something you like, then click to like it

asle · October 30, 2020, 3:35pm

Thanks. That is a real IP so it is correct. <VirtualHost 85.196.85.68:443>
I wrote that all other sites work fine and you see in my config I use the same IP for port 80.

Here is the output of apachectl:
~ /usr/local/opt/httpd/bin/apachectl -S
VirtualHost configuration:
85.196.85.68:443 is a NameVirtualHost
default server babkart.no (/usr/local/etc/httpd/vhosts/babkart.no.conf:19)
port 443 namevhost babkart.no (/usr/local/etc/httpd/vhosts/babkart.no.conf:19)
alias www.babkart.no
port 443 namevhost borchbio.no (/usr/local/etc/httpd/vhosts/borchbio.no.conf:27)
alias www.borchbio.no
85.196.85.68:80 is a NameVirtualHost
default server babkart.no (/usr/local/etc/httpd/vhosts/babkart.no.conf:1)
port 80 namevhost babkart.no (/usr/local/etc/httpd/vhosts/babkart.no.conf:1)
alias www.babkart.no
port 80 namevhost borchbio.no (/usr/local/etc/httpd/vhosts/borchbio.no.conf:1)
alias www.borchbio.no
port 80 namevhost globalnor.no (/usr/local/etc/httpd/vhosts/globalnor.no.conf:1)
alias www.globalnor.no
alias globalnor.xyz
alias www.globalnor.xyz
port 80 namevhost jeloykameratene.no (/usr/local/etc/httpd/vhosts/jeloykameratene.no.conf:1)
alias www.jeloykameratene.no
alias xn--jelykameratene-sqb.no
alias www.xn--jelykameratene-sqb.no
port 80 namevhost jordforeningen.no (/usr/local/etc/httpd/vhosts/jordforeningen.no.conf:1)
alias www.jordforeningen.no
port 80 namevhost kulturminnekart.no (/usr/local/etc/httpd/vhosts/kulturminnekart.no.conf:1)
alias www.kulturminnekart.no
port 80 namevhost narin.biofokus.no (/usr/local/etc/httpd/vhosts/narin.biofokus.no.conf:1)
port 80 namevhost naturkart.no (/usr/local/etc/httpd/vhosts/naturkart.no.conf:1)
alias www.naturkart.no
alias natur2000.no
alias www.natur2000.no
ServerRoot: "/usr/local/opt/httpd"
Main DocumentRoot: "/Library/WebServer/Documents"
Main ErrorLog: "/usr/local/var/log/httpd/error_log"
Mutex proxy-balancer-shm: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/usr/local/var/run/httpd/" mechanism=default
Mutex mpm-accept: using_defaults
PidFile: "/usr/local/var/run/httpd/httpd.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="_www" id=70 not_used
Group: name="_www" id=70 not_used

rg305 · October 30, 2020, 3:48pm

Please show the other working SSL vhost config.

I'm thinking the broken config is missing:
SSLEngine On

It might/should be in this file.
Let's have a look at it:
Include /etc/letsencrypt/options-ssl-apache.conf

rg305 · October 30, 2020, 3:53pm

This is not happening:

curl -Iki http://borchbio.no/
HTTP/1.1 200 OK
Date: Fri, 30 Oct 2020 15:50:40 GMT
Server: Apache/2.4.46 (Unix) OpenSSL/1.1.1h PHP/7.2.33
Last-Modified: Fri, 24 Aug 2007 23:56:56 GMT
ETag: "136-4387ac306a200"
Accept-Ranges: bytes
Content-Length: 310
Vary: Accept-Encoding
Content-Type: text/html

This is not encrypting:

curl -Iki http://borchbio.no:443/
HTTP/1.1 200 OK
Date: Fri, 30 Oct 2020 15:51:49 GMT
Server: Apache/2.4.46 (Unix) OpenSSL/1.1.1h PHP/7.2.33
Last-Modified: Fri, 24 Aug 2007 23:56:56 GMT
ETag: "136-4387ac306a200"
Accept-Ranges: bytes
Content-Length: 310
Vary: Accept-Encoding
Content-Type: text/html

It's almost like this file is loaded but not loaded:

I would stop Apache and check for any Apache processes that remain running.
If any, please kill them all and then restart Apache.

asle · October 30, 2020, 6:19pm

Here is /usr/local/etc/httpd/vhosts/borchbio.no.conf

SSLEngine on

# Intermediate configuration, tweak to your needs
SSLProtocol             all -SSLv2 -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite          ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384
SSLHonorCipherOrder     off

SSLOptions +StrictRequire

# Add vhost name to log entries:
LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-agent}i\"" vhost_combined
LogFormat "%v %h %l %u %t \"%r\" %>s %b" vhost_common

asle · October 30, 2020, 6:20pm

Hi, I turned off everything for letsencrypt to get the site up again. Do you want me to turn it on again for testing?

rg305 · October 30, 2020, 6:22pm

Please explain that.
"turned off" ?

asle · October 30, 2020, 6:24pm

Commented out everything SSL. I turned it on again now. You see the site is redirecting and fails to load.

rg305 · October 30, 2020, 6:25pm

I recommended for you to...

Do you know how to do those steps?

asle · October 30, 2020, 6:27pm

This is a list of all httpd processes. I see one is run by root. Is that wrong? I killed the one from root. Did not help.
➜ vhosts ps aux | grep httpd
_www 41514 0.0 0.0 4493944 1928 ?? S 7:23PM 0:00.00 /usr/local/opt/httpd/bin/httpd -k start
_www 41513 0.0 0.0 4494200 2868 ?? S 7:23PM 0:00.00 /usr/local/opt/httpd/bin/httpd -k start
_www 41512 0.0 0.0 4493944 1932 ?? S 7:23PM 0:00.00 /usr/local/opt/httpd/bin/httpd -k start
_www 41511 0.0 0.0 4493944 1956 ?? S 7:23PM 0:00.00 /usr/local/opt/httpd/bin/httpd -k start
_www 41510 0.0 0.0 4494200 2852 ?? S 7:23PM 0:00.00 /usr/local/opt/httpd/bin/httpd -k start
_www 41509 0.0 0.0 4493944 1952 ?? S 7:23PM 0:00.00 /usr/local/opt/httpd/bin/httpd -k start
root 80338 0.0 0.1 4493944 13872 ?? Ss 11:59PM 0:04.90 /usr/local/opt/httpd/bin/httpd -k start

rg305 · October 30, 2020, 7:17pm

First STOP apache/httpd (before killing anything).
Then, if that is still there after the stop go ahead and clear that problem.

asle · October 30, 2020, 8:37pm

Thanks for helping out! It seems httpd is started as root. Homebrew prefers running as local user. When I stop that httpd services and run from

/Users/hb/Library/LaunchAgents/homebrew.mxcl.httpd.plist -> the command in the plist is this:
/usr/local/opt/httpd/bin/httpd
I now get this error:

AH00526: Syntax error on line 23 of /usr/local/etc/httpd/vhosts/borchbio.no.conf:
SSLCertificateFile: file '/etc/letsencrypt/live/borchbio.no/fullchain.pem' does not exist or is empty

But the fullchain.pem does indeed exist and is not empty!

asle · October 30, 2020, 8:44pm

Here is an observation:
When I try to start httpd as user
/usr/local/opt/httpd/bin/httpd
I get the error of the missing fullchain.pem and httpd will not start.

When I start httpd as root
sudo /usr/local/opt/httpd/bin/httpd
I get no error and httpd can serve the sites that do not have SSL

Something is strange, permissions?

rg305 · October 31, 2020, 4:19am

So it needs to start as root (to get the certificate files).
But is everything working, as expected, when you do?

asle · October 31, 2020, 10:31am

HTTPD starts but the sites with SSL do not work. This site is using SSL
borchbio.no
this site is on the same server not using SSL: kulturminnekart.no
I get no error in the logfile for borchbio.no. I am logging to /usr/local/var/log/httpd/ but there are no entries in the error_log.

asle · October 31, 2020, 11:39am

Progress! I was able to add a certificate to this site on the same server: https://narin.biofokus.no/
So something is wrong with borchbio.no still. Not sure how to fix that.

asle · November 2, 2020, 8:37pm

I have solved it now. Here is what I have learned:

Homebrew version of Apache (httpd) must be run as root since it is under port 1024. Standard for Homebrew software versions is to run as local user, which is very practical really.
The Apache binaries are located in "/usr/local/opt/httpd/bin/"
To start httpd manually: /usr/local/opt/httpd/bin/apachectl start (or httpd start)
"brew services start httpd" fixes startup as root by adding a plist file in /Library/LaunchDaemons/
Fixed the problem with letsencrypt certificates by removing the <VirtualHost *:443> section for the vhosts conf file.

I deleted everything about "SSL" and certificates in the vhost conf and all files regarding the domain in /etc/letsencrypt. They are in different subfolders and you must be root to delete them.
Then I ran sudo certbot and setup the domain again. Everything is ok now. Thanks to @rg305 for hanging on with me!