Multiple certs being sent by server, one is revoked and expired, how can I remove?

liquidgnome · July 27, 2018, 3:52pm

I’ve run out of options and can’t seem to find the answer I need, so hopefully somebody here can be of assistance.

I run letsencrypt on a server I own, with several domains hosted from there. I am using a single cert for every domain at once, and am renewing certs via letsencrypt-auto. At some point a few months ago, I tried to add a new domain, was working with a new registrar, and messed up my DNS. As I couldn’t get a cert for the domain, I did something at that time, which i think might have been to revoke the existing cert.

I installed a new certificate, and part of what I did during/after the revoke was to remove letsencrypt completely and start again. I realize now the error of my ways, but I don’t have a backup that goes that far back, and I don’t think it would matter if I did.

My server is currently supplying two certificates, one of which is valid and one of which is revoked. I let the revoked one expire, thinking that would remove the SSL errors, but it did not.

Looking at ssllabs for the various domains, I am getting two certs listed, but I can’t for the life of me figure out if the revoked one is still lurking somewhere on the machine. I don’t think it is, but that’s what I need to know.

Here is a domain with an “A” rating that is still sending two certs:
https://www.ssllabs.com/ssltest/analyze.html?d=www.northless.com

Here is a domain with a “T” rating that is sending the two certs but in reverse order so the revoked one is coming through first:
https://www.ssllabs.com/ssltest/analyze.html?d=www.missuswalkah.com

And here’s a third domain that was probably the cause of this trouble, which is showing an “A” rating but a cert mismatch as it is not listed in the set of domains for the bad certificate:
https://www.ssllabs.com/ssltest/analyze.html?d=www.lindsaysmonarchs.com

My domain is: northless.com, missuswalkah.com, lindsaysmonarchs.com, liquidgnome.com

My web server is (include version): apache2

The operating system my web server runs on is (include version): ubuntu 16.04.9

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

My understanding is that the certs are maintained on the CA level, which means I might be SOL. How can I get rid of this revoked cert in favor of the one that actually works and is trusted? Is that even possible? I have looked at every key and pem file on my server and can’t find anything with the same value as the revoked cert is sending.

I feel as though I’m out of options. Any help somebody can provide would be great.

sahsanu · July 27, 2018, 5:39pm

Hi @liquidgnome,

To try to understand what is going on, could you please show the output of these commands (as root or using sudo)?.

letsencrypt-auto certificates

ls -lR /etc/letsencrypt/live/

Cheers,
sahsanu

liquidgnome · July 27, 2018, 7:14pm

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: arcticsleep.com
    Domains: arcticsleep.com api.gleisner.net lindsaysmonarchs.com liquidgnome.com missuswalkah.com northless.com streetlightinterference.com www.arcticsleep.com www.lindsaysmonarchs.com www.liquidgnome.com www.missuswalkah.com www.northless.com www.streetlightinterference.com
    Expiry Date: 2018-10-25 14:26:33+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/arcticsleep.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/arcticsleep.com/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

/etc/letsencrypt/live/:
total 4
drwxr-xr-x 2 root root 4096 Jul 27 15:26 arcticsleep.com

/etc/letsencrypt/live/arcticsleep.com:
total 4
lrwxrwxrwx 1 root root  39 Jul 27 15:26 cert.pem -> ../../archive/arcticsleep.com/cert5.pem
lrwxrwxrwx 1 root root  40 Jul 27 15:26 chain.pem -> ../../archive/arcticsleep.com/chain5.pem
lrwxrwxrwx 1 root root  44 Jul 27 15:26 fullchain.pem -> ../../archive/arcticsleep.com/fullchain5.pem
lrwxrwxrwx 1 root root  42 Jul 27 15:26 privkey.pem -> ../../archive/arcticsleep.com/privkey5.pem
-rw-r--r-- 1 root root 543 Jun  9 16:17 README

i did at some point also install certbot on the machine, but the output of "certbot certificates" is the same.

sahsanu · July 27, 2018, 7:40pm

You only have 1 certificate covering all domains and all is good but right now you have 6 domains/subdomains not showing the right cert:

api.gleisner.net
liquidgnome.com
missuswalkah.com
streetlightinterference.com
www.liquidgnome.com
www.streetlightinterference.com

You should check the Apache conf defined for all of them and double check that they are pointing to the right cert and key:

/etc/letsencrypt/live/arcticsleep.com/fullchain.pem
/etc/letsencrypt/live/arcticsleep.com/privkey.pem

liquidgnome · July 27, 2018, 7:50pm

Ok, so how did you determine that? cert.sh or something?

I noticed in looking at the apache configs for the vhosts that the -le-ssl.conf files that letsencrypt generated at some point include port designations for both 443 and 80, where there are also .conf files in place for port 80 for those domains.

I checked, and the cert/key paths are correct. I might see about recreating the *-le-ssl.conf files from a couple of the so-called “good” ones on the server, but I am not brimming with confidence that that will work.

I disabled the missuswalkah.com.conf file (which was previously serving at port 80 and redirecting to https://) and the site is still throwing cert_revoked errors after apache was reloaded.

sahsanu · July 27, 2018, 10:06pm

@liquidgnome, do you have some kind of load balancer, reverse proxy or similar?. I’m asking because for example, checking liquidgnome.com 10 times in a row, it is showing the expired certificate 4 times and the right certificate 6 times:

$ for i in {1..10};do echo "Test $i";echo | openssl s_client -connect liquidgnome.com:443 -servername liquidgnome.com 2>/dev/null | openssl x509 -noout -dates; echo "";done
Test 1
notBefore=Apr 28 00:51:18 2018 GMT
notAfter=Jul 27 00:51:18 2018 GMT

Test 2
notBefore=Jul 27 14:26:33 2018 GMT
notAfter=Oct 25 14:26:33 2018 GMT

Test 3
notBefore=Apr 28 00:51:18 2018 GMT
notAfter=Jul 27 00:51:18 2018 GMT

Test 4
notBefore=Apr 28 00:51:18 2018 GMT
notAfter=Jul 27 00:51:18 2018 GMT

Test 5
notBefore=Jul 27 14:26:33 2018 GMT
notAfter=Oct 25 14:26:33 2018 GMT

Test 6
notBefore=Apr 28 00:51:18 2018 GMT
notAfter=Jul 27 00:51:18 2018 GMT

Test 7
notBefore=Jul 27 14:26:33 2018 GMT
notAfter=Oct 25 14:26:33 2018 GMT

Test 8
notBefore=Jul 27 14:26:33 2018 GMT
notAfter=Oct 25 14:26:33 2018 GMT

Test 9
notBefore=Jul 27 14:26:33 2018 GMT
notAfter=Oct 25 14:26:33 2018 GMT

Test 10
notBefore=Jul 27 14:26:33 2018 GMT
notAfter=Oct 25 14:26:33 2018 GMT

It also happens with other domains like www.missuswalkah.com, northless.com, etc. so seems you are using some kind of load balancer and looks like there are 2 Apache instances and you only have updated the cert in one of them, the other one is using the old certificate, does make sense?.

Cheers,
sahsanu

liquidgnome · July 28, 2018, 6:39pm

I do not have a load balancer or a reverse proxy installed. (The proxy bit is a little scary. I’m going to have to do some digging and see what is going on here. I’ve looked for information on any kind of back door or malware that could install a proxy of some kind, but I’ve come up blank.)

This behavior does seem consistent with what I’ve seen: namely one can reload the site multiple times and get cert errors sometimes, but not others.

Thank you!

sahsanu · July 28, 2018, 7:03pm

@liquidgnome, that is really strange. If you don’t mind to share the whole apache web server conf I could take a look just in case.

tar zcvf liquidgnome-apache.tar.gz /etc/apache2/

and put liquidgnome-apache.tar.gz in one of your sites so we can download it.

Also, could you please show the output of these commands?

ss -tlpn | cat

iptables --list-rules

liquidgnome · July 28, 2018, 8:27pm

iptables --list-rules
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT

# ss -tlpn | cat
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      128    127.0.0.1:11211                    *:*                   users:(("memcached",pid=885,fd=26))
LISTEN     0      20     127.0.0.1:25                       *:*                   users:(("exim4",pid=1231,fd=4))
LISTEN     0      80          :::3306                    :::*                   users:(("mysqld",pid=911,fd=14))
LISTEN     0      128         :::80                      :::*                   users:(("apache2",pid=14605,fd=4),("apache2",pid=14592,fd=4),("apache2",pid=14591,fd=4),("apache2",pid=14590,fd=4),("apache2",pid=14589,fd=4),("apache2",pid=14588,fd=4),("apache2",pid=14585,fd=4))
LISTEN     0      128         :::80                      :::*                   users:(("apache2",pid=20390,fd=4),("apache2",pid=18874,fd=4),("apache2",pid=18873,fd=4),("apache2",pid=16669,fd=4),("apache2",pid=16668,fd=4),("apache2",pid=16667,fd=4),("apache2",pid=16665,fd=4),("apache2",pid=16664,fd=4),("apache2",pid=16650,fd=4),("apache2",pid=16649,fd=4),("apache2",pid=957,fd=4))
LISTEN     0      20         ::1:25                      :::*                   users:(("exim4",pid=1231,fd=5))
LISTEN     0      128         :::443                     :::*                   users:(("apache2",pid=14605,fd=6),("apache2",pid=14592,fd=6),("apache2",pid=14591,fd=6),("apache2",pid=14590,fd=6),("apache2",pid=14589,fd=6),("apache2",pid=14588,fd=6),("apache2",pid=14585,fd=6))
LISTEN     0      128         :::443                     :::*                   users:(("apache2",pid=20390,fd=6),("apache2",pid=18874,fd=6),("apache2",pid=18873,fd=6),("apache2",pid=16669,fd=6),("apache2",pid=16668,fd=6),("apache2",pid=16667,fd=6),("apache2",pid=16665,fd=6),("apache2",pid=16664,fd=6),("apache2",pid=16650,fd=6),("apache2",pid=16649,fd=6),("apache2",pid=957,fd=6))

I just killed memcached, but there was no effect.

https://www.liquidgnome.com/liquidgnome-apache.tar.gz

So weird.

sahsanu · July 28, 2018, 8:44pm

@liquidgnome, I’m wondering how do you have two instances of apache running at the same time in the same ports…

Could you please stop apache, check again the processes, start apache and check listening services?.

systemctl stop apache2
ss -tlpn | cat
systemctl start apache2
ss -tlpn | cat

liquidgnome · July 28, 2018, 8:51pm

Welp. I think that’s the problem.

I stopped Apache, but Apache is still running and when I run the script to check the cert, it shows 100% bad cert information.

So, Apache is probably owned. Great.

sahsanu · July 28, 2018, 8:54pm

So, what is the output of ss -tlpn | cat right now?. Maybe you should kill the apache instance that is already running.

liquidgnome · July 28, 2018, 9:02pm

ss -tlpn | cat
State      Recv-Q Send-Q Local Address:Port               Peer Address:Port              
LISTEN     0      20     127.0.0.1:25                       *:*                   users:(("exim4",pid=1231,fd=4))
LISTEN     0      80          :::3306                    :::*                   users:(("mysqld",pid=911,fd=14))
LISTEN     0      128         :::80                      :::*                   users:(("apache2",pid=20390,fd=4),("apache2",pid=18874,fd=4),("apache2",pid=18873,fd=4),("apache2",pid=16669,fd=4),("apache2",pid=16668,fd=4),("apache2",pid=16667,fd=4),("apache2",pid=16665,fd=4),("apache2",pid=16664,fd=4),("apache2",pid=16650,fd=4),("apache2",pid=16649,fd=4),("apache2",pid=957,fd=4))
LISTEN     0      20         ::1:25                      :::*                   users:(("exim4",pid=1231,fd=5))
LISTEN     0      128         :::443                     :::*                   users:(("apache2",pid=20390,fd=6),("apache2",pid=18874,fd=6),("apache2",pid=18873,fd=6),("apache2",pid=16669,fd=6),("apache2",pid=16668,fd=6),("apache2",pid=16667,fd=6),("apache2",pid=16665,fd=6),("apache2",pid=16664,fd=6),("apache2",pid=16650,fd=6),("apache2",pid=16649,fd=6),("apache2",pid=957,fd=6))

Killing now with extreme prejudice.

liquidgnome · July 28, 2018, 9:04pm

Killed those procs, started Apache, showing only good certs at the moment.

sahsanu · July 28, 2018, 9:08pm

Great, It’s working fine from my side too

liquidgnome · July 28, 2018, 9:11pm

Well, aside from the archaeological expedition to go figure out why rogue apache was just running for a couple months, I really appreciate your help and insight!

Thank you so much for spending your time looking into this with me!

Cheers,

system · August 27, 2018, 9:11pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.