Trying to solve the issue that is popping up here with the expired DST Root CA X3 cert.
We've figured out a workaround but we're trying to understand why certificates even just renewed today are still showing that DST cert in the chain. It's causing some of our users and services to fail. While I understand we can fix things here, we're just modifying the pem files to remove the DST cert in the files and getting clients to update is another solution.
But we're still puzzled as to why even newly created certs are showing that expired cert in the chain at all and putting it into the cert files. We added this param --preferred-chain 'ISRG Root X1' to the cert creation but still seems to be including references to the DST cert in the file.
What are we missing here? We're using sed to modify existing certs to remove it from the chain but want to make sure future renewals or creations aren't including that part in the chain.
And in between, there are a small number of servers that can be fixed by serving the short chain. Specifically, those running OpenSSL < 1.1.0. If you do serve the short chain, older Android devices won't be able to access your site, but servers that (a) run older OpenSSL and (b) trust ISRG Root X1 will work again.
In theory, the --preferred-chain 'ISRG Root X1' flag you are passing should achieve that. Can you share the Certbot version you are running? [Edit: I see you already did that]
Thanks, I'll try to get you more and clearer info, could be user error here. And to be clear you used that param on a cert renewal and it works for you?
Hmmm... still looking, it seems we might not have been as up to date as I had thought.
We're still using certbot-auto and seems they might be on older versions 1.7.0 & 1.6.0 so perhaps that's why were seeing this still when renewing.
As for where we're seeing it in the chain. Looking in /etc/letsencrypt/archive/REDACTED.com/
We have these files:
sudo ls -ltr /etc/letsencrypt/archive/REDACTED.com/
-rw-r--r-- 1 root root 1704 Oct 1 17:39 privkey14.pem
-rw-r--r-- 1 root root 1854 Oct 1 17:39 cert14.pem
-rw-r--r-- 1 root root 3749 Oct 1 17:39 chain14.pem
-rw-r--r-- 1 root root 5603 Oct 1 17:39 fullchain14.pem
The chain14.pem and fullchain14.pem have this at the end of the chain.
Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Validity
Not Before: Sep 4 00:00:00 2020 GMT
Not After : Sep 15 16:00:00 2025 GMT
Subject: C=US, O=Let's Encrypt, CN=R3
And
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
We are guessing these certs aren't working due to the issuer for that last one being this Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 which devices probably stopped trusting yesterday. We manually removed that last cert in the chain from some certs today which seems to have made some devices happy, but now we're not sure if that caused issues for other devices.
To add more info we had someone on Android 7.1.1 yesterday who was failing to load due to the cert, this was with the longer chain above. When we removed the last portion of that chain they now seem to be working. So something about that longer chain was problematic it seems to the "older" but "newish" devices.
This likely has little to do with this particular problem, but worth the mention.
If at all possible, I would remove that version altogether.
And install the newest certbot available for your O/S.
[or even another ACME client]
Notice that using pip as the installation method is not the recommended method for most Linux distributions. While it could have been the only way for @onmeac to update their certbot, please refer to https://certbot.eff.org/ to check the recommended installation method for your distribution/OS.
For sure, thank you. Definitely something we'll need to consider doing. I didn't realize our version was that far behind what's current.
Also a tad bit hesitant to update as I didn't write the automation and scripting we have set up so we'll definitely need to do a bunch of testing to make sure it doesn't break our current setup. But this is probably something we should do regardless.
I have exactly the same issue than @onmeac :
I ran certbot renew --force-renewal --preferred-chain "ISRG Root X1",
in /etc/letsencrypt/renewal/<domain>.conf I have preferred_chain = ISRG Root X1 and version = 1.20.0, but despite this my cert.pem still depends on R3.
I'm checking the certificate by running the following command openssl s_client -servername mydomain.ca -showcerts -connect mydomain.ca:443, and I still have depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = mydomain.ca
Does someone have an idea of how to fix this ?
I tried the use of certbot installed by pip3 (after stopping any services listening op ports 80/443), as @onmeac did, but it didn't fix the issue.
Your chain is the expected one if you specify `preferred_chain = ISRG Root X1`.
The R3 intermediate is always the issuer for all Let's Encrypt certificates. If you used the long chain, you would also be serving a path that refers to the expired DST Root CA X3. Since you specified the preferred chain, you are not serving that path and your server's suggested trust path chains up to ISRG Root X1, via R3, like all other current Let's Encrypt certificates.
Sorry, I mistook what you were saying about your openssl output.
Can you tell us your real domain? And what's in your fullchain.pem file? You didn't copy chain.pem elsewhere or hard-code a reference to a particular version of it or anything, right?
This is a regular output for OpenSSL chaining up to the ISRG Root X1 certificate. For example, if I connect to acme-v02.api.letsencrypt.org, whhich is also using the "short chain", my OpenSSL outputs:
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = acme-v01.api.letsencrypt.org
verify return:1
---
Certificate chain
0 s:CN = acme-v01.api.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
So from that first output you can only see to which cert the chain is being build. But with the output just below that, you can see the chain send by the server.
