Installed a new certificate but the browsers still use the old expired one

Hi there!
My certificate expired, and I renewed it successfully, however, this had no effect in the clients so I decided to remove it and install a new one. This was successful according to the console output, however, I the browsers still pick up the old expired certificate. Any idea why this might be?
My domain is: www.bebetto.bg

I ran this command:
Removed the old certificate and installed a new one. Now running certbot certificates
It produced this output:
Found the following certs:
Certificate Name: www.bebetto.bg
Serial Number: 439a80f1af767971d5b45ae9110d9e19081
Key Type: RSA
Domains: www.bebetto.bg
Expiry Date: 2022-09-09 09:08:45+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/www.bebetto.bg/fullchain.pem
Private Key Path: /etc/letsencrypt/live/www.bebetto.bg/privkey.pem

My web server is (include version): apache/1.4.29

The operating system my web server runs on is (include version):Ubuntu 18.04

My hosting provider, if applicable, is:
digitalocean with WAF provided by prophase

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.28.0

How did you install your previous and your new certificate in Apache?

Also note that you've issued 5 certificates today (crt.sh | bebetto.bg) and thus are rate limited. You should understand the process better: if installation of an issued certificate is the problem, but issuance is not, it does not matter how much times you re-issue a certificate: it won't fix the installation problem.

Now you've issued 4 certs unnecessarily, increasing unnecessary load to the Let's Encrypt systems. Please don't do that.

2 Likes

Thanks for responding.
I used certbot --apache
The when I tried to install the new one, it gave me an error that bebetto-le-ssl-config already exists, so I renamed it and run certbot --apache again. This time it was installed successfully.
I tried certbot -senronly --apache too.. No difference on the browser side.

How can I undo this?
Also, my new certificate is issued to www.bebetto.bg . I cannot issue one for bebetto.bg (without www) because the WAF does not route the requested to this domain to my server.

Interestingly you've issued two other certificates since the currently served certificate, which expired today. On 27 April and 8 May. Nothing was done with those? Doesn't look like automated renewals..

Anyway, please show the output of the following command:

sudo apachectl -t -D DUMP_VHOSTS

And the contents of the file:

/etc/letsencrypt/renewal/www.bebetto.bg.conf

There's nothing to be undone? Once you've issued a certificate, the Let's Encrypt resources have been spend, nothing you can do about that. Also, you already have a valid certificate. Just don't loose that one.

1 Like
[Sat Jun 11 11:07:36.839477 2022] [so:warn] [pid 2388] AH01574: module php7_modu                                                                                         le is already loaded, skipping
VirtualHost configuration:
*:443                  www.bebetto.bg (/etc/apache2/sites-enabled/bebetto-le-ssl                                                                                         .conf:2)
*:80                   is a NameVirtualHost
         default server bebetto.bg-1643092430872-s-1vcpu-1gb-fra1-01 (/etc/apach                                                                                         e2/sites-enabled/000-default.conf:1)
         port 80 namevhost bebetto.bg-1643092430872-s-1vcpu-1gb-fra1-01 (/etc/ap                                                                                         ache2/sites-enabled/000-default.conf:1)
         port 80 namevhost bebe1.ddns.net (/etc/apache2/sites-enabled/bebe1.conf                                                                                         :2)
                 alias bebe1.ddns.net
         port 80 namevhost bebe2.ddns.net (/etc/apache2/sites-enabled/bebe2.conf                                                                                         :2)
                 alias bebe2.ddns.net
         port 80 namevhost www.bebetto.bg (/etc/apache2/sites-enabled/bebetto.co                                                                                         nf:1)
                 wild alias *.bebetto.bg
         port 80 namevhost opencart.ddns.net (/etc/apache2/sites-enabled/opencar                                                                                         t.conf:2)
                 alias opencart.ddns.net
root@bebetto:~#
# renew_before_expiry = 30 days
version = 1.28.0
archive_dir = /etc/letsencrypt/archive/www.bebetto.bg
cert = /etc/letsencrypt/live/www.bebetto.bg/cert.pem
privkey = /etc/letsencrypt/live/www.bebetto.bg/privkey.pem
chain = /etc/letsencrypt/live/www.bebetto.bg/chain.pem
fullchain = /etc/letsencrypt/live/www.bebetto.bg/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = d670e7a7741810e0c8d14b2a65c17cc8
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa

There's nothing to be undone

Apologies, I will not do this anymore.

1 Like

Hm, not really anything weird with that..

Please show the contents of the file /etc/apache2/sites-enabled/bebetto-le-ssl.conf

2 Likes

Yeah, I did not experience any errors when installing the certificate. The old certificate was also renewed without errors.

<IfModule mod_ssl.c>
<VirtualHost *:443>
     ServerAdmin admin@example.com
     DocumentRoot /var/www/html/bebe2/
     ServerName www.bebetto.bg
     ServerAlias *.bebetto.bg

     <Directory /var/www/html/bebe2/>
        Options FollowSymlinks
        AllowOverride All
        Order allow,deny
        allow from all
     </Directory>

     ErrorLog ${APACHE_LOG_DIR}/error.log
     CustomLog ${APACHE_LOG_DIR}/access.log combined

RewriteEngine on
# Some rewrite rules in this file were disabled on your HTTPS site,
# because they have the potential to create redirection loops.

# RewriteCond %{SERVER_NAME} =*.bebetto.bg [OR]
# RewriteCond %{SERVER_NAME} =bebetto.bg
# RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]


SSLCertificateFile /etc/letsencrypt/live/www.bebetto.bg/fullchain.pem
SSLCertificateKeyFile /etc/letsencrypt/live/www.bebetto.bg/privkey.pem
Include /etc/letsencrypt/options-ssl-apache.conf
</VirtualHost>
</IfModule>

Nothing wrong with that.

Looks like your issue is in your WAF. It probably handles all the certificate stuff too but there is also something wrong with how it handles the backend.

See the output below when I try to connect to your site:

osiris@erazer ~ $ curl -LIvk --http1.1 https://www.bebetto.bg
*   Trying 65.21.250.191:443...
* Connected to www.bebetto.bg (65.21.250.191) port 443 (#0)
* ALPN: offers http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
* TLSv1.3 (IN), TLS handshake, Certificate (11):
* TLSv1.3 (IN), TLS handshake, CERT verify (15):
* TLSv1.3 (IN), TLS handshake, Finished (20):
* TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.3 (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
* ALPN: server accepted http/1.1
* Server certificate:
*  subject: CN=www.bebetto.bg
*  start date: Mar 13 00:38:23 2022 GMT
*  expire date: Jun 11 00:38:22 2022 GMT
*  issuer: C=US; O=Let's Encrypt; CN=R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
> HEAD / HTTP/1.1
> Host: www.bebetto.bg
> User-Agent: curl/7.83.1
> Accept: */*
> 
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
* old SSL session ID is stale, removing
* Mark bundle as not supporting multiuse
< HTTP/1.1 504 Gateway Time-out
HTTP/1.1 504 Gateway Time-out
< Date: Sat, 11 Jun 2022 11:22:34 GMT
Date: Sat, 11 Jun 2022 11:22:34 GMT
< Content-Type: text/html
Content-Type: text/html
< Content-Length: 10316
Content-Length: 10316
< Connection: keep-alive
Connection: keep-alive
< ETag: "60fa8f1f-284c"
ETag: "60fa8f1f-284c"
< Server: Prophaze EagleEye
Server: Prophaze EagleEye

< 
* Connection #0 to host www.bebetto.bg left intact
osiris@erazer ~ $ 

Notice the error "gateway timeout"? It probably means that your WAF can't connect to your Apache. And even without your Apache, it's sending the older certificate, which means you probably need to update the renewed certificate in your WAF.

2 Likes

I will get them on the case and will report back as soon as I have a reply.
Maybe it is time to look for other WAF alternatives? Any recommendation (apart from Cloudflare)?
Thanks for helping me troubleshoot this! Much appreciated!

I personally have no experience with WAFs at all, so sorry. Maybe others do.

2 Likes

Please show output of:
netstat -pant | grep -i listen

1 Like

Hi rg305!

tcp        0      0 0.0.0.0:1234            0.0.0.0:*               LISTEN      910/sshd
tcp        0      0 127.0.0.1:4321          0.0.0.0:*               LISTEN      1110/mysqld
tcp        0      0 0.0.0.0:2314              0.0.0.0:*               LISTEN      908/vsftpd
tcp        0      0 127.0.0.53:1231           0.0.0.0:*               LISTEN      736/systemd-resolve
tcp6       0      0 :::443                  :::*                    LISTEN      4273/apache2
tcp6       0      0 :::1234                 :::*                    LISTEN      910/sshd
tcp6       0      0 :::80                   :::*                    LISTEN      4273/apache2

Please show:
ps -ef | grep apache | grep -v grep

1 Like

@rg305 I'm pretty sure the Prophaze EagleEye WAF is before OPs Apache. And even before OPs own host.

Although that might have changed since earlier. Currently, I'm getting a false certificate from "cert-manager.local"? Weird.. Port 80 still reports the Prophaze EagleEye WAF and port 443 still gives a timeout on the backend, so probably also Prophaze EagleEye.

4 Likes

You are correct. No errors if I go to the server directly by mapping the domain to the server IP address. I have contacted Prophaze and they are on the case.
Thanks for your help!

2 Likes