Help thread for DST Root CA X3 expiration (September 2021)

amandadukor · October 18, 2021, 10:39pm

I have downloaded it. Thank you.
But the prob is, I can't just change the ca bundle. I have to reinstall everything.
I don't have the other ones

bruncsak · October 19, 2021, 6:46am

When you log in, sometimes you still have under your ZeroSSL account all the data (key, cert, bundle) available to download it again. Please do that, and do not forget about the "CA bundle". After that you upload them together to easyWP.

If the certificate with its private key is not in your ZeroSSL account any more, and you do not have any copy, only then you have to create a new certificate.

amandadukor · October 19, 2021, 9:42pm

Hello. I've tried to access it but I can't

bruncsak · October 20, 2021, 6:27am

So, it is time now to create a new certificate.

fleish · October 21, 2021, 12:31am

2 questions:

Is there a way to request the chain/fullchain files sent back from LE includes the self-signed ISRG X1 root vs. the one still cross-signed with the now-expired DST CA X3 root?
Is there a time-frame when the chain files will no longer include the cross-signed root?

I ask as I spent a good bit of time today troubleshooting why I could not update a VMware vCenter Server appliance and that turned out to be the reason. I had to manually remove the cross-signed ISRG X1 root out of the chain file & replace it with the self-signed version or else vCenter would fail to import and also wedge itself into a non-functional state in the process.

Nummer378 · October 21, 2021, 12:45am

The chain(s) provided by LE generally do not contain root certificates (they're already included in the clients trust store and should never be sent by TLS servers). The current default chain does not include DST Root CA X3 itself (though it contains a certificate with this issuer).

But yes, you can request a chain from LE that does not include the final cross-sign up to DST Root CA X3. This is called the alternate chain. This chain must be explicitly requested by your ACME client - how to do this depends on your client, often it's a command line argument called preferred-chain (or similar).

There is no explicit time-frame, but you can expect this chain to stay for some years, possibly until late 2024. The chain leading to the expired root is supposed to help with compatibility for some clients (mainly Android), so it's going to stay with us for a while. We're aware that some clients don't like the current default chain, which is why the alternate chain is available

The current and alternate chain served, as well as any upcoming planned changes can be found here:

fleish · October 21, 2021, 5:23am

Maybe we're using the same terms for different things, but both chain.pem & fullchain.pem do include the root ISRG X1 certificate as the final certificate in each file. The problem for my use case was that certificate was not the self-signed version, but rather a version that was issued by DST CA X3 which caused vCenter to bail out because that is an expired certificate.

From what I'm reading here, certbot (which is the client I'm using) is already defaulting to the "new" chain which still provides the cross-signed ISRG X1 root certificate.

rg305 · October 21, 2021, 5:30am

All self-signed versions of certs must be roots; and as such would never be found in a distributed chain.
Root certs are distributed via trust stores.

If you want to end your "chain" with the self signed root, then remove the cross-signed one from the chain.

fleish · October 21, 2021, 6:32pm

I'm not sure what to tell you, the fullchains I previously got back from LE/ACME via certbot contained 3 certs:

The server cert being issued
The intermediate cert which signed the server cert (currently R3)
The root cert that signed the intermediate cert (currently ISRG Root X1)

Currently, the root cert provided is signed by the now-expired DST Root CA X3.

I'm not sure vCenter server would accept that without the ISRG Root X1 certificate added to its trust store.

I can also confirm that adding the --preferred-chain "ISRG Root X1" switch to certbot results in the fullchain only containing R3 & ISRG X1 currently & the chain only containing R3 currently.

At this point, I've got a working setup so I'm just documenting all of this here for others who may run into the same problem when they go to update their VMware vCenter server certificates with LE and find themselves in a bind.

Osiris · October 21, 2021, 6:39pm

That was probably the ISRG Root X1-signed-by-DST Root CA X3 intermediate certificate.

Let's Encrypt doesn't and has never included the actual self signed root certificate in the certificate chain.

rg305 · October 21, 2021, 9:46pm

Yes, as we've both mentioned:

The provided "ISRG Root X1" NOT a root cert; as it is (cross-)signed by "DST Root CA X3".

Root certs show their "NAME" signed by (their same) "NAME".
See these two:

blackberry_10 · October 31, 2021, 10:19am

BlackBerry 10 Devices (with the most recent and last OS update) have a work around for this expired certificate [ DST Root CA X3 ]. So the native BlackBerry Browser can still once again visit sites disabled by the expiry of this certificate (ie like Wikipedia).

On the BlackBerry phones in [Settings][Security and Privacy][Certificates]
Search for: [ DST Root CA X3 ]
Remove the CheckBox for Trusted (so it will now be untrusted).
In the BlackBerry case this caused an error that indicated the [ ISRG Root X1 ] might be the problem. However unchecking [ DST Root CA X3 ] now allows [ ISRG Root X1 ] to be the default certificate to check in the chain, and it is trusted already on BB10 phones. Now the Browser will no longer block access to sites that depend on this Security Certificate. In order to know that the expired [ DST Root CA X3 ] was the cause (in causing the block to Wikipedia etc) , you can optionally uncheck trusted for [ ISRG Root X1 ], just to see the[ DST Root CA X3] show up in the certificate details chain on the BlackBerry Web Browser site blocked certificate details page. Just make sure you check it back as Trusted after untrusting [ DST Root CA X3 ].

Easy Fix. Other solutions can be found on CrackBerry.com (the official unofficial resource community for BlackBerry related issues and users).

blackberry_10 · October 31, 2021, 11:55am

So its not quite true that Blackberry 10 devices will no longer validate "Let's Encrypt certificates". All that a user needs to do to correct sites from being blocked, is remove Trusted from the expired certificate [DST Root CA X3], and the [ ISRG Root X1 ] will work as it is Trusted by default. The two certificates appear to conflict if both are marked as trusted and one is expired. On older Blackberry 10 OS versions <10.3.3, a user may need to import the ISRG certificate, if it is not included. Blackberry 10 phones can allow users to manually Trust or unTrust certificates that are installed, and so can continue to work after September 2021.

So perhaps this page needs to be altered:
Certificate Compatibility - Let's Encrypt

Platforms that trust DST Root CA X3 but not ISRG Root X1

These platforms would have worked up to September 2021 but will no longer validate Let’s Encrypt certificates.

macOS < 10.12.1
iOS < 10
...
Blackberry >= 10.3.3 (version that added ISRG Root X1 unknown)

Bruce5051 · November 24, 2021, 9:55pm

Don't worry to much the "Overall rating:" it is just a grading system, worry more about the real security of your server. Yes the "Overall rating:" is helpful but not the end all be all.

My 23 year old son claims that I am "pedantically prolifically languagistly bombastic"
And SSL Labs gives my site SSL Server Test: hp-67.com (Powered by Qualys SSL Labs)
but is actually secure in reality? I doubt it. Yet it is ridiculously restrictive, thus far from being widely compatible with older system.

rg305 · November 24, 2021, 10:07pm

To that, I'd have to say...
Congrats on providing him with the opportunity of such a fine education!
Not only can he use "big words", he can make a complete sentence using only "big words" - you should be proud (I am; I don't even know him).
[although I think you might have misquoted "linguistically" as "languagistly" - but I'm no Harvard scholar so who knows]

Bruce5051 · November 24, 2021, 10:08pm

Thank you @rg305

rg305 · November 24, 2021, 10:08pm

I too have a "smart ass" 23 year old son - LOL

Bruce5051 · November 24, 2021, 10:22pm

Yeah, I used to tell my wife I was a "smart ass", she would reply no a "dumb ass".

Nummer378 · November 24, 2021, 11:52pm

Some would say that config is a bit over the top . But given that this really looks like you wanted to do it right, I would suggest that you turn off (TLSv1.2) session tickets too.

HPKP is also not recommended anymore (sadly) and I would suggest some more AEAD ciphers on the TLSv1.2 side (the same as for 1.3 for example is fine). If you want to help out some non AES-NI accelerated devices, selecting ChaCha for them is also nice (OpenSSL has a flag for this - other libraries it depends).

Bruce5051 · November 25, 2021, 12:07am

It is over the top; however it is just a toy for me. And it works on my Androids 4.4.4., 8.1, 9, and 11; Chrome, Firefox, Edge.

Yeah, some of the "testing" sites still ding for not having HPKP. Removing it would make the loading of the page a hair faster.