The certificate is not trusted in all web browsers. You may need to install an Intermediate/chain certificate to link it to a trusted root certificate

Help
1

Hi Team,

I am using Lets encrypt certificates for all non www domains and currently we are facing The Certificate is not trusted in all web browsers issue.

The non www domains SSL are working intermittently. Mostly failing for some regions with SSL error that your connection is not private.

If checked through SSL checker it shows root or intermediate certs has expired.
Please find the below screenshot for your reference

image
image558×754 68.8 KB

My domain is: colgatepension.com

My web server is (include version): Apache/2.4.64 ()

The operating system my web server runs on is (include version): Amazon Linux 2

I can login to a root shell on my machine (yes or no, or I don't know): Yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.29.0

Just for reference if needed
SSLCertificateFile /opt/www/conf/certs/fullchain.pem
SSLCertificateKeyFile /opt/www/conf/certs/privkey.pem

Please note this is happening for our 2000 + non www sites which is very critical issue for us and needs to be resolved immediately with highest priority.

Please let me know the solution for this with the steps required to fix this.
Awaiting for your response.

2

The webserver for that domain is sending a really strange certificate chain:

Certificate chain
 0 s:CN=colgatepension.com
   i:C=US, O=Let's Encrypt, CN=R13
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct 19 15:29:37 2025 GMT; NotAfter: Jan 17 15:29:36 2026 GMT
 1 s:C=US, O=Let's Encrypt, CN=R3
   i:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 00:00:00 2020 GMT; NotAfter: Sep 15 16:00:00 2025 GMT
 2 s:C=US, O=Internet Security Research Group, CN=ISRG Root X1
   i:O=Digital Signature Trust Co., CN=DST Root CA X3
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jan 20 19:14:03 2021 GMT; NotAfter: Sep 30 18:14:03 2024 GMT

Notice that the first certificate, the end leaf certificate with your hostname, was issued by "R13". However, the next certificate is "R3", which is an intermediate certificate already retired for some time now, and not the R13 which is to be expected. And the next intermediate is an ancient cross-signed root which has expired more than a year already.

Thus: something very strange is happening with your certificate chain.

Now, normally Certbot would just use the chain provided by the ACME server and put it together with the end leaf certificate into fullchain.pem. However, CLEARLY that's not what's happening here.

Also, the path /opt/www/conf/certs/fullchain.pem is not a regular Certbot path either.

I suspect some manual or perhaps automated (but flawed [and probably unnecessary]) tinkering is being done to that fullchain.pem, resulting in this completely incorrect certificate chain. Or perhaps an incorrect webserver configuration elsewhere than the two single directives you have provided.

Also: if this has been the standard workflow for some time now, this is not something that just happened recently due to something with Let's Encrypt. This chain is flawed in such a way that it should have been noticed sooner, unless something has happened recently with regard to your setup.

4 Likes
3

Hi Osiris,

Thanks for the prompt reply.

Please disregard the /opt/www/conf/certs/fullchain.pem path.

The fullchain.pem is actually copied in a different path for automation.

sudo cp /etc/letsencrypt/live/${DOMAIN}/fullchain.pem /home/vmladmin/apacheconfig/certs/${DOMAIN}.pem

sudo cp /etc/letsencrypt/live/${DOMAIN}/privkey.pem /home/vmladmin/apacheconfig/certs/${DOMAIN}_key.pem

4

Now I'm confused.

Please provide all the relevant webserver configuration with regard to TLS and include how those files are being 'generated'/'placed'/'modified'/'whatever'.

3 Likes
5

The short answer is that those fullchain and privkey files from certbot are what your server needs to use, and it isn't. It's clearly using some other files instead. So you need to fix whatever process you're using so that your server actually can use those files (or copies of them, assuming that you're using certbot's deploy-hook or the like to copy them to where your server is configured to use them).

4 Likes
6

We are using a Dev Server to generate or renew the certificate and copies the fullchain and privkey on Prod servers.

This setup has been in use for many years now. This issue was reported in Oct first week all of a sudden. No change was done in the setup, we have auto renewal script in place.

This script is run in the dev server and it renews the cert, copies the new fullchain and privkey to different path and then those are deployed to prod servers.

Sharing the renewal script detail.
sudo certbot renew --webroot-path /opt/www/letsencrypt

for DOMAIN in $(sudo find /etc/letsencrypt/live/ -name fullchain.pem -mmin -120 | awk -F "/" '{print $5}')
do
cd /home/vmladmin/apacheconfig
git config --global user.name "Let's Encrypt"
git config --global user.email sysops@redfuse.com
sudo git checkout master
sudo git pull
sudo git checkout dev
sudo git pull
sudo cp /etc/letsencrypt/live/${DOMAIN}/fullchain.pem /home/vmladmin/apacheconfig/certs/${DOMAIN}.pem
sudo cp /etc/letsencrypt/live/${DOMAIN}/privkey.pem /home/vmladmin/apacheconfig/certs/${DOMAIN}_key.pem
sudo git add /home/vmladmin/apacheconfig/certs/${DOMAIN}.pem /home/vmladmin/apacheconfig/certs/${DOMAIN}_key.pem
sudo git commit -a -m "PS-28564 updating renewed let's encrypt cert for ${DOMAIN}"
sudo git push
done

Is there a way to manually fix this expired root or intermediate certificates ?

7

It's provided by Certbot. It should be used. Something is not going correctly. As mentioned before:

Currently you've only shared a single script which apparently just checks for fullchain.pem files updated the last 2 hours and apparently pushes them to some git repository.

5 Likes
8

where you found string /opt/www/conf/certs/fullchain.pem ?
I think certbot install cert at /home/vmladmin/apacheconfig... than third script moving it to vm's opt/www/.... and you looked at apache config file with that path.

2 Likes
9

This is the cross signed root, this chain is pretty old.

@amey that doesn't look like a fullchain.pem, it looks like a cert.pem cat a very old chain.pem

4 Likes
10

They've changed something because they now send a chain that includes both current RSA intermediates.

@amey It looks like something is "pinning" these intermediates to the reply rather than relying on the chain given to you by Let's Encrypt.

The R12 / R13 intermediates were only recently introduced. They replaced R10/R11 which replaced R3 that you showed in your first post.

"Pinning" (or hard-coding) the chain is poor practice and not recommended. Your "leaf" cert is the one with your domain name in it. It will be signed randomly by R12 or R13 and that's the only intermediate that you should send out. The leaf and its matching intermediate are the only two certs in Certbot's "fullchain.pem" file.

You should quit pinning these intermediates. At minimum, this ensures you future-proof your setup as Let's Encrypt changes intermediates in the future (and they plan to). Frankly, it seems harder to do what you've done than just rely on fullchain.pem

See: SSL Server Test: colgatepension.com (Powered by Qualys SSL Labs)

The chain your server currently sends:

 0 s:CN = colgatepension.com
   i:C = US, O = Let's Encrypt, CN = R13
   v:NotBefore: Oct 21 16:46:32 2025 GMT; NotAfter: Jan 19 16:46:31 2026 GMT
 1 s:C = US, O = Let's Encrypt, CN = R13
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Let's Encrypt, CN = R12
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
4 Likes