Help thread for DST Root CA X3 expiration (September 2021)

[fleish]

@Nummer378

The chain(s) provided by LE generally do not contain root certificates (they're already included in the clients trust store and should never be sent by TLS servers). The current default chain does not include DST Root CA X3 itself (though it contains a certificate with this issuer ).

Maybe we're using the same terms for different things, but both chain.pem & fullchain.pem do include the root ISRG X1 certificate as the final certificate in each file. The problem for my use case was that certificate was not the self-signed version, but rather a version that was issued by DST CA X3 which caused vCenter to bail out because that is an expired certificate.

@Nummer378

But yes, you can request a chain from LE that does not include the final cross-sign up to DST Root CA X3. This is called the alternate chain . This chain must be explicitly requested by your ACME client - how to do this depends on your client, often it's a command line argument called preferred-chain (or similar).

From what I'm reading here, certbot (which is the client I'm using) is already defaulting to the "new" chain which still provides the cross-signed ISRG X1 root certificate.

[rg305]

@fleish

The problem for my use case was that certificate was not the self-signed version...

All self-signed versions of certs must be roots; and as such would never be found in a distributed chain.
Root certs are distributed via trust stores.

If you want to end your "chain" with the self signed root, then remove the cross-signed one from the chain.

[fleish]

@rg305

All self-signed versions of certs must be roots; and as such would never be found in a distributed chain.
Root certs are distributed via trust stores.

I'm not sure what to tell you, the fullchains I previously got back from LE/ACME via certbot contained 3 certs:

• The server cert being issued
• The intermediate cert which signed the server cert (currently R3)
• The root cert that signed the intermediate cert (currently ISRG Root X1)

Currently, the root cert provided is signed by the now-expired DST Root CA X3.

@rg305

If you want to end your "chain" with the self signed root, then remove the cross-signed one from the chain.

I'm not sure vCenter server would accept that without the ISRG Root X1 certificate added to its trust store.

I can also confirm that adding the --preferred-chain "ISRG Root X1" switch to certbot results in the fullchain only containing R3 & ISRG X1 currently & the chain only containing R3 currently.

At this point, I've got a working setup so I'm just documenting all of this here for others who may run into the same problem when they go to update their VMware vCenter server certificates with LE and find themselves in a bind.

[Osiris]

@fleish

I'm not sure what to tell you, the fullchains I previously got back from LE/ACME via certbot contained 3 certs:

• The server cert being issued
• The intermediate cert which signed the server cert (currently R3)
• The root cert that signed the intermediate cert (currently ISRG Root X1)

That was probably the ISRG Root X1-signed-by-DST Root CA X3 intermediate certificate.

Let's Encrypt doesn't and has never included the actual self signed root certificate in the certificate chain.

[rg305]

Yes, as we've both mentioned:

@fleish

The root cert that signed the intermediate cert (currently ISRG Root X1)

The provided "ISRG Root X1" NOT a root cert; as it is (cross-)signed by "DST Root CA X3".

Root certs show their "NAME" signed by (their same) "NAME".
See these two:

[blackberry_10]

BlackBerry 10 Devices (with the most recent and last OS update) have a work around for this expired certificate [ DST Root CA X3 ]. So the native BlackBerry Browser can still once again visit sites disabled by the expiry of this certificate (ie like Wikipedia).

On the BlackBerry phones in [Settings][Security and Privacy][Certificates]
Search for: [ DST Root CA X3 ]
Remove the CheckBox for Trusted (so it will now be untrusted).
In the BlackBerry case this caused an error that indicated the [ ISRG Root X1 ] might be the problem. However unchecking [ DST Root CA X3 ] now allows [ ISRG Root X1 ] to be the default certificate to check in the chain, and it is trusted already on BB10 phones. Now the Browser will no longer block access to sites that depend on this Security Certificate. In order to know that the expired [ DST Root CA X3 ] was the cause (in causing the block to Wikipedia etc) , you can optionally uncheck trusted for [ ISRG Root X1 ], just to see the[ DST Root CA X3] show up in the certificate details chain on the BlackBerry Web Browser site blocked certificate details page. Just make sure you check it back as Trusted after untrusting [ DST Root CA X3 ].

Easy Fix. Other solutions can be found on CrackBerry.com (the official unofficial resource community for BlackBerry related issues and users).

[blackberry_10]

So its not quite true that Blackberry 10 devices will no longer validate "Let's Encrypt certificates". All that a user needs to do to correct sites from being blocked, is remove Trusted from the expired certificate [DST Root CA X3], and the [ ISRG Root X1 ] will work as it is Trusted by default. The two certificates appear to conflict if both are marked as trusted and one is expired. On older Blackberry 10 OS versions <10.3.3, a user may need to import the ISRG certificate, if it is not included. Blackberry 10 phones can allow users to manually Trust or unTrust certificates that are installed, and so can continue to work after September 2021.

So perhaps this page needs to be altered:
Certificate Compatibility - Let's Encrypt

Platforms that trust DST Root CA X3 but not ISRG Root X1

These platforms would have worked up to September 2021 but will no longer validate Let’s Encrypt certificates.

• macOS < 10.12.1
• iOS < 10
• ...
• Blackberry >= 10.3.3 (version that added ISRG Root X1 unknown)

[Bruce5051]

@mmncs

I have tried acces on Opera and Chrome and I can see that it has removed

DST Root CA X3

But it still doesn't work on either. This is on my older mac.

Here is the overall rating:

SSL Server Test: www.beautonart.com (Powered by Qualys SSL Labs)

Overall rating: B

Protocols
TLS 1.3 No
TLS 1.2 Yes
TLS 1.1 Yes
TLS 1.0 Yes
SSL 3 No
SSL 2 No

Don't worry to much the "Overall rating:" it is just a grading system, worry more about the real security of your server. Yes the "Overall rating:" is helpful but not the end all be all.

My 23 year old son claims that I am "pedantically prolifically languagistly bombastic"
And SSL Labs gives my site SSL Server Test (Powered by Qualys SSL Labs)
but is actually secure in reality? I doubt it. Yet it is ridiculously restrictive, thus far from being widely compatible with older system.

[rg305]

@Bruce5051

My 23 year old son claims that I am " pedantically prolifically linguistically bombastic "

To that, I'd have to say...
Congrats on providing him with the opportunity of such a fine education!
Not only can he use "big words", he can make a complete sentence using only "big words" - you should be proud (I am; I don't even know him).
[although I think you might have misquoted "linguistically" as "languagistly" - but I'm no Harvard scholar so who knows]

[Bruce5051]

Thank you @rg305

[rg305]

I too have a "smart ass" 23 year old son - LOL

[Bruce5051]

Yeah, I used to tell my wife I was a "smart ass", she would reply no a "dumb ass".

[Nummer378]

@Bruce5051

And SSL Labs gives my site SSL Server Test: hp-67.com (Powered by Qualys SSL Labs)

Some would say that config is a bit over the top . But given that this really looks like you wanted to do it right, I would suggest that you turn off (TLSv1.2) session tickets too.

We need to talk about Session Tickets

HPKP is also not recommended anymore (sadly) and I would suggest some more AEAD ciphers on the TLSv1.2 side (the same as for 1.3 for example is fine). If you want to help out some non AES-NI accelerated devices, selecting ChaCha for them is also nice (OpenSSL has a flag for this - other libraries it depends).

[Bruce5051]

It is over the top; however it is just a toy for me. And it works on my Androids 4.4.4., 8.1, 9, and 11; Chrome, Firefox, Edge.

@Nummer378

HPKP

Yeah, some of the "testing" sites still ding for not having HPKP. Removing it would make the loading of the page a hair faster.

[Bruce5051]

@Nummer378

Some would say that config is a bit over the top . But given that this really looks like you wanted to do it right, I would suggest that you turn off (TLSv1.2) session tickets too.

Yeah, it is over the top in many senses. However it is also a simple configuration, which IMHO, has some advantages also.

I am not recommending that people (or their HTTPS: TLS/SSL connections) should be that tight.

I am surprised how many browsers and devices actually work these tight restrictions.

[ventura57]

Someone could explain to me how this still works !!! Below is the fullchain of my certificate in reverse order issued yesterday, Dec. 7th.

The "DST Root CA X3" is expired but still in use to renew certificates !!!

I understand "ISRG Root X1" is not expired but the issuer is expired. What am I missing here ?

Certificate 1 (top)

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

Certificate 2

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Validity
Not Before: Sep 4 00:00:00 2020 GMT
Not After : Sep 15 16:00:00 2025 GMT
Subject: C = US, O = Let's Encrypt, CN = R3

Certificate 3 (my certificate signed by lets encrypt)

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:fd:94:55:b2:59:85:92:40:26:24:15:bf:60:42:16:b1:5d
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Dec 7 16:16:34 2021 GMT
Not After : Mar 7 16:16:33 2022 GMT
Subject: CN =

[ventura57]

Someone could explain to me how this still works !!! Below is the fullchain of my certificate in reverse order issued yesterday, Dec. 7th.

The "DST Root CA X3" is expired but still in use to renew certificates !!!

I understand "ISRG Root X1" is not expired but the issuer is expired. What am I missing here ?

Certificate 1 (top)

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
40:01:77:21:37:d4:e9:42:b8:ee:76:aa:3c:64:0a:b7
Signature Algorithm: sha256WithRSAEncryption
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After : Sep 30 18:14:03 2024 GMT
Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

Certificate 2

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Validity
Not Before: Sep 4 00:00:00 2020 GMT
Not After : Sep 15 16:00:00 2025 GMT
Subject: C = US, O = Let's Encrypt, CN = R3

Certificate 3 (my certificate signed by lets encrypt)

Certificate:
Data:
Version: 3 (0x2)
Serial Number:
04:fd:94:55:b2:59:85:92:40:26:24:15:bf:60:42:16:b1:5d
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Dec 7 16:16:34 2021 GMT
Not After : Mar 7 16:16:33 2022 GMT
Subject: CN =

[ventura57]

Shouldn't "ISRG Root X1" be self signed ???
In my browser I have "R3" and "ISRG Root X1" but the last is self signed.

I am beginning to panic.

[Osiris]

@ventura57

I am beginning to panic.

Don't. This is intentional. Please read:

Extending Android Device Compatibility for Let's Encrypt Certificates - ...

[ventura57]

Thank you Osiris.
Before receive your message I was checking every certificate renewed using the command:

openssl s_client -connect the_url:443

and the fact all check was OK. You article explain everything.

Again, thank your very much.