Expired Cert On Only Some Pages, Active Certs from Unknown Source

Help
1

My domain is: humboldtutilities.com

I ran this command:

  1. sudo certbot certificates

  2. crt.sh search

It produced this output:

    Domains: humboldtutilities.com www.humboldtutilities.com
    Expiry Date: 2024-02-13 16:55:36+00:00 (VALID: 39 days)
    Certificate Path: /etc/letsencrypt/live/humboldtutilities.com/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/humboldtutilities.com/privkey.pem
  1. https://crt.sh/?q=humboldtutilities.com

My web server is (include version): Apache, not sure what version

The operating system my web server runs on is (include version): Ubuntu 18.04.1

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): ExpressionEngine 6

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

Note: My certificates are set to auto-renew so as of this moment I haven't made any changes/taken any actions on the server, so I know this is not something I caused.

So, this is a twofold issue kind of. I'm VERY new to all of this, and picking up where previous devs left off with little to no documentation, so please be patient and ELI5 whenever possible.

The site is showing as Not Secure but only sometimes and not on every page. On most pages, most of the time, it's successfully pulling the correct Certbot certificate that I can see on the server. When I hit the error, I can see exactly that it's pulling the previous certificate that expired in December.

The problem is I don't know or understand why it's doing that. As you can see from my output, the new February cert is right there, and it's pulling just fine most of the time. Is there any action I can take or is this just a matter of time??

Part 2 is that, in investigating this and running the crt.sh query, I have NO idea where half of the active certs even came from! There's the 2/13 expiry cert that I expect from Certbot. Then, this site is also on Cloudflare. Based on my Cloudflare account SSL settings, I'm expecting to see a certificate that expires on 3/24... and it's not there. Instead, I see 3(?) certs set to expire on 1/25.

How in the world do I track those down - I would need file location, or website service e.g. Cloudflare, or something - in the event that I ever need to check on, remove, adjust those?? Or if they are the Cloudflare certs, why is the expiry date so wildly different??

2

Sometimes old Apache processes with the old cert loaded keep lingering around. Try stopping Apache entirely, check all processes are gone and start Apache again. (Note: that would lead to downtime for a few seconds [if you start it up again fast enough just after checking], so don't do it when hundreds of visitors are visiting your site.)

Cloudflare can also issue Let's Encrypt certificates for domains using the Cloudflare CDN, but also uses other CAs for that purpose. Note that crt.sh also shows the pre-certificates, so it often shows double the certs if deduplication is not enabled (crt.sh | humboldtutilities.com)

4 Likes
3

When proxying your domain at Cloudflare you are using its CDN. For any CDN there are two HTTPS connections so two different certs involved (there are more complex possibilities I am keeping it simple for this example).

Cloudflare CDN gets certs on your behalf (sometimes from Let's Encrypt). From the public internet a client (browser) connects to CF CDN and sees that cert (always as long as your domain is proxied there).

The CF CDN then makes an HTTPS connection to your Apache server "behind the scenes" which uses the cert from Apache. This is the cert you got using Certbot. The client (browser) does not see this HTTPS connection or cert.

On your local network, if you somehow connect directly to your Apache server you will also be using the Apache cert.

The above is just how CDN's work and is explained on the Cloudflare docs.

The crt.sh log display is not showing the most recent certs that Cloudflare got for you. But, try a site like this (link here). When you click details the wildcard cert issued by "E1" is from CF (in your case). The other certs from "R1" are what you got using Certbot on your Apache server.

To know why you see one cert or another in a browser we need to know how that device connected to your domain. That is, from the public internet and so using Cloudflare. Or, in a local network with perhaps local DNS settings direct to your Apache server.

4 Likes
4

Both these replies have been really helpful! I appreciate breaking things down this much, it can be hard to diagnose a problem when all the documentation assumes a level of knowledge just a hop and a skip above where I'm at, so I was getting pretty lost in the weeds. At the very least, I think I understand why I'm seeing the certs I'm seeing now. :grimacing:

@MikeMcQ especially, thank you for that explanation. Based on it, I actually think I took the steps that I needed to and didn't know that they were really going to "do" anything - in Cloudflare, the DNS records were both set to DNS Only and not to Proxied, so I toggled them over to Proxied because that's how all others are set up and I like to be consistent. I wasn't sure if there was a reason the original dev set it up without proxy, and was hoping that wasn't the case, but toggling that sounds like it may have solved the issue.

I've confirmed this likelihood by checking the certificate details upon a successful page load, and it shows the details for the CF certificate as of this afternoon. This morning, BEFORE I changed those settings, the certificate details on a successful page load were the ones for the Apache/Certbot cert. Which sounds painfully obvious in hindsight, but like I said in the OP, I don't have much leftover instructions from the devs before me.

Thank you!!

ETA: I'm not going to mark this solved until tomorrow when I have a chance to comb through everything again and make sure the issue doesn't persist.

3 Likes
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.