Apache intermittently gets wrong certificate

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: api.mightysmallhomes.com

I ran this command: openssl s_client -showcerts -connect 23.253.218.119:443

It produced this output:
Sometimes it is (wrong!):
depth=1 C = US, O = Let’s Encrypt, CN = Let’s Encrypt Authority X3
depth=0 CN = resultsbydesign.com

And a second later it might be (correct):
depth=1 C = US, ST = Arizona, L = Scottsdale, O = “GoDaddy.com, Inc.”, OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Aut
hority - G2
depth=0 OU = Domain Control Validated, CN = api.mightysmallhomes.com

And …

apachectl -S

VirtualHost configuration:
23.253.218.119:443 api.mightysmallhomes.com
(/etc/apache2/conf/vhosts/vhosts_staging.conf:1582)
23.253.221.93:443 www.pinsiaa.com (/etc/apache2/conf/vhosts/vhosts_ssl.conf:941)
23.253.221.91:443 is a NameVirtualHost
etc, etc.

The second one referenced by openssl should be correct. I need to make the first one (letsencrypt) go away I guess. Not sure why apache is so utterly confused. I deleted any reference to resultsbydesign.com in /etc/letsencrypt (there was a wildcard cert there at one time), and restarted apache a number of times.

My web server is (include version): Apache 2.4.18

The operating system my web server runs on is (include version): Ubuntu 18.04 LTS

My hosting provider, if applicable, is: Rackspace

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.27.0

Thanks!

2

Hi,

What's the output if you run
openssl s_client -showcerts -connect api.mightysmallhomes.com:443

If you run the command with IP address, you'll see the first domain in your Apache server sequence (if you didn't designate a default server), this was previously crucial to clients that doesn't support SNI.

https://www.ssllabs.com/ssltest/analyze.html?d=api.mightysmallhomes.com&latest
The test above does prove that your Apache instance has designated the old domain (the one you were trying to delete) as the default instance.

You might need to share the entire virtual host configuration files since the apachectl -S might not show the certificate loaded to each files.

Also, did you restart (not reload) Apache?

Thank you

3

I can reproduce it:

$ openssl s_client -connect api.mightysmallhomes.com:443 -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -dates
subject=CN = resultsbydesign.com
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
notBefore=Sep 27 14:48:18 2019 GMT
notAfter=Dec 26 14:48:18 2019 GMT
^C

$ openssl s_client -connect api.mightysmallhomes.com:443 -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -dates
subject=CN = resultsbydesign.com
issuer=C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
notBefore=Sep 27 14:48:18 2019 GMT
notAfter=Dec 26 14:48:18 2019 GMT
^C

$ openssl s_client -connect api.mightysmallhomes.com:443 -showcerts 2>/dev/null | openssl x509 -noout -subject -issuer -dates
subject=OU = Domain Control Validated, CN = api.mightysmallhomes.com
issuer=C = US, ST = Arizona, L = Scottsdale, O = "GoDaddy.com, Inc.", OU = http://certs.godaddy.com/repository/, CN = Go Daddy Secure Certificate Authority - G2
notBefore=Dec  2 21:16:27 2019 GMT
notAfter=Dec  2 21:16:27 2021 GMT

In the past when I’ve seen this, it’s because there have been orphaned Apache processes which have the old certificate loaded, which don’t get killed when Apache is restarted the normal way.

The resolution to that is generally:

service apache2 stop
killall -9 apache2
service apache2 restart

and seeing whether the problem is still reproducible.

If it still happens, then I’d look to the configuration.

Edit: I realized that killall may not always work. It’s best to confirm that after you stop Apache, that you cannot manually find any apache2 processes running.

4

BINGO _az! That indeed was the problem and solution! After stopping apache2 there was about 8 processes still active. Thanks! This was driving me crazy.

5

Good thinking! :clap: