--preferred-chain not taking effect

Trying to solve the issue that is popping up here with the expired DST Root CA X3 cert.

We've figured out a workaround but we're trying to understand why certificates even just renewed today are still showing that DST cert in the chain. It's causing some of our users and services to fail. While I understand we can fix things here, we're just modifying the pem files to remove the DST cert in the files and getting clients to update is another solution.

But we're still puzzled as to why even newly created certs are showing that expired cert in the chain at all and putting it into the cert files. We added this param --preferred-chain 'ISRG Root X1' to the cert creation but still seems to be including references to the DST cert in the file.

What are we missing here? We're using sed to modify existing certs to remove it from the chain but want to make sure future renewals or creations aren't including that part in the chain.

3 Likes

To add a bit more info, it appears we're on the latest version of 1.19.0 of certbot.

2 Likes

Hi @mle-ii, welcome to the forum!

This is an intentional choice, so that most sites will continue to support older Android devices. See Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt.

There are also some users and servers that will simply not work anymore: Certificate Compatibility - Let's Encrypt.

And in between, there are a small number of servers that can be fixed by serving the short chain. Specifically, those running OpenSSL < 1.1.0. If you do serve the short chain, older Android devices won't be able to access your site, but servers that (a) run older OpenSSL and (b) trust ISRG Root X1 will work again.

In theory, the --preferred-chain 'ISRG Root X1' flag you are passing should achieve that. Can you share the Certbot version you are running? [Edit: I see you already did that]

5 Likes

Which file are you finding DST references in when you run with --preferred-chain?

2 Likes

Thanks, I'll try to get you more and clearer info, could be user error here. And to be clear you used that param on a cert renewal and it works for you?

4 Likes

I haven't personally used that param on a cert renewal but I have gotten reports from many folks here that they've used it successfully.

3 Likes

Hmmm... still looking, it seems we might not have been as up to date as I had thought.

We're still using certbot-auto and seems they might be on older versions 1.7.0 & 1.6.0 so perhaps that's why were seeing this still when renewing.

As for where we're seeing it in the chain. Looking in /etc/letsencrypt/archive/REDACTED.com/
We have these files:

sudo ls -ltr /etc/letsencrypt/archive/REDACTED.com/
-rw-r--r-- 1 root root 1704 Oct  1 17:39 privkey14.pem
-rw-r--r-- 1 root root 1854 Oct  1 17:39 cert14.pem
-rw-r--r-- 1 root root 3749 Oct  1 17:39 chain14.pem
-rw-r--r-- 1 root root 5603 Oct  1 17:39 fullchain14.pem

The chain14.pem and fullchain14.pem have this at the end of the chain.

        Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C=US, O=Let's Encrypt, CN=R3

And

        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1

We are guessing these certs aren't working due to the issuer for that last one being this Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3 which devices probably stopped trusting yesterday. We manually removed that last cert in the chain from some certs today which seems to have made some devices happy, but now we're not sure if that caused issues for other devices.

2 Likes

To add more info we had someone on Android 7.1.1 yesterday who was failing to load due to the cert, this was with the longer chain above. When we removed the last portion of that chain they now seem to be working. So something about that longer chain was problematic it seems to the "older" but "newish" devices.

4 Likes

This likely has little to do with this particular problem, but worth the mention.
If at all possible, I would remove that version altogether.
And install the newest certbot available for your O/S.
[or even another ACME client]

2 Likes

Found this thread because I too have issues with --preferred-chain not working as I had understood it to work.

I ran:
certbot renew --force-renewal --preferred-chain "ISRG Root X1"

preferred_chain = ISRG Root X1 is even in my /etc/letsencrypt/renewal/<domain>.conf

renewal was a success but both chain.pem and fullchain.pem still have a CA cert ISRG Root X1 signed by DST Root CA X3

Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3

certbot-1.11, on EL7.

4 Likes

Hi @onmeac welcome to the LE community forum :slight_smile:

I think you need certbot version 1.12 (or higher) to use --preferred-chain

3 Likes

Thank you, that bit information was missing from all proposed sollutions

Maybe it helps someone else, I did this:

sudo yum install python3
sudo mkdir /certbot
sudo python3 -m venv certbot/
sudo /certbot/bin/pip3 install --upgrade pip
sudo /certbot/bin/pip3 install certbot
# stop any services listening op ports 80/443
sudo /certbot/bin/certbot renew --force-renewal --preferred-chain "ISRG Root X1"
4 Likes

Notice that using pip as the installation method is not the recommended method for most Linux distributions. While it could have been the only way for @onmeac to update their certbot, please refer to https://certbot.eff.org/ to check the recommended installation method for your distribution/OS.

3 Likes

For sure, thank you. Definitely something we'll need to consider doing. I didn't realize our version was that far behind what's current. :frowning:

Also a tad bit hesitant to update as I didn't write the automation and scripting we have set up so we'll definitely need to do a bunch of testing to make sure it doesn't break our current setup. But this is probably something we should do regardless.

4 Likes

Do you run it on CentOS? I think this comment is related: Nginx with preferred-chain "ISRG Root X1" - still showing "DST Root CA X3" - #31 by FelixSchwarz

it seems there is some issue in CentOS certbot package and the fix is on the way.

I have encountered the same problems on CentOS machines.

3 Likes

Short version: Try certbot 1.11.0-2 from epel testing.

For more details please read:

7 Likes

Hello,

I have exactly the same issue than @onmeac :
I ran certbot renew --force-renewal --preferred-chain "ISRG Root X1",
in /etc/letsencrypt/renewal/<domain>.conf I have preferred_chain = ISRG Root X1 and version = 1.20.0, but despite this my cert.pem still depends on R3.
I'm checking the certificate by running the following command openssl s_client -servername mydomain.ca -showcerts -connect mydomain.ca:443, and I still have
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = mydomain.ca

Does someone have an idea of how to fix this ?

I tried the use of certbot installed by pip3 (after stopping any services listening op ports 80/443), as @onmeac did, but it didn't fix the issue.

Thanks in advance,

Jajoe

3 Likes

Hi @jajoe,

Your chain is the expected one if you specify `preferred_chain = ISRG Root X1`.

The R3 intermediate is always the issuer for all Let's Encrypt certificates. If you used the long chain, you would also be serving a path that refers to the expired DST Root CA X3. Since you specified the preferred chain, you are not serving that path and your server's suggested trust path chains up to ISRG Root X1, via R3, like all other current Let's Encrypt certificates.

Sorry, I mistook what you were saying about your openssl output.

Can you tell us your real domain? And what's in your fullchain.pem file? You didn't copy chain.pem elsewhere or hard-code a reference to a particular version of it or anything, right?

3 Likes

@jajoe
Can you show the lines of code in the web server that are using the cert files?

2 Likes

This is a regular output for OpenSSL chaining up to the ISRG Root X1 certificate. For example, if I connect to acme-v02.api.letsencrypt.org, whhich is also using the "short chain", my OpenSSL outputs:

CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = acme-v01.api.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN = acme-v01.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

So from that first output you can only see to which cert the chain is being build. But with the output just below that, you can see the chain send by the server.

4 Likes