Why is letsencrypt not fixing DST Root CA X3 issue

Apologies, not R3

This is causing issues as all the customer browsers is now stating that the sites are insecure.

We have tried to renew the certificates with new ones, however this does not solve the issue as it is still picking up this chain.

My issue is with why has a fix not been implemented to fix this on the certificates or even on certbot to cater for this.

Hi @machiel and welcome to the LE community forum :slight_smile:

I think you are going beyond a "rant" with:

When this is your first post about it and we are assisting YOU with YOUR problem.
Let me explain:
There is nothing technically wrong with the trust path being served by your web site.
See: [SSL Server Test: saicanews.co.za (Powered by Qualys SSL Labs)]
(SSL Server Test: saicanews.co.za (Powered by Qualys SSL Labs))
[OR use any other online certificate verification tool]
You can compare that trust path to the one this very community uses.
See: SSL Server Test: community.letsencrypt.org (Powered by Qualys SSL Labs)
They are exactly the same.

That said, there are (mostly older) clients that are unable to correctly validate that trust path.
Admittedly (IMHO), for more reasons than were anticipated.
But when you say:

You make it sound like there was an option where a certificate expiration date could simply be avoided and it only happened because it was the chosen path.
Nothing in the world can stop a certificate from expiring.
Once a certificate is issued, it has a start date and an end date (expiration).
That goes for every cert in the entire PKI system.

If you would care to read though their site (like within their announcement pages), you would see that there are two available trust paths.

  • the longer/default trust path which includes the now expired "DST Root CA X3" root.
    [note that's an X3 (not an R3) - there is no expired R3 in any of the current trust paths]
  • the shorter/alternate trust path which uses the proper self-signed "ISRG Root X1" root.
    [issued in 2015 and now in all major trust stores]

If you need help switching trust paths, you can either search this site (and find dozens of such examples or simply ask for help with that).

If you don't understand something, then you can also ask for clarification/explanation.
There is no conspiracy, no conscious decision was made to mess up the Internet PKI system.
None of that is happening.
What is happening (by and large) is that systems that have NOT updated their trust stores since 2015 are now having trust issues. And some systems (mainly Windows), that even though they do have updated trust stores fail to see (and trust) the "new" (since 2015) trusted root in the longer trust path.
All of this would be happening with every other CA, had their root cert expired sooner than this one did.
And this will likely continue to happen (also with other CAs) as long as older devices are still being used long long long after their manufacturers have stopped providing any support/updates for them or as long as people don't take any steps to keep their systems updated.

6 Likes

If you are to serve the shorter/alternate chain, you must now do one of these four things:

  • manually remove the last cert from the fullchain.pem (or cert.pem) file used.
    [not an automated solution - not recommended]
  • update your certbot 0.31.0 to a version that supports the parameter --preferred-chain
    [you would need certbot 1.12 or higher]
  • switch to another (newer) ACME client - like: acme.sh
    [which has support for the --preferred-chain parameter]
  • switch to another (FREE and ACME friendly) CA
    [with implies the use of another completely different trust path - without "DST Root CA X3" in it]

TIME CHECK...
Solution provided within 1 hour of your RANT - LOL

5 Likes

@rg305

Thank you for the solution path.

Just as an explanation, we did remove the last cert in the fullchain as well based on an article we found, however as stated this was not a proper fix as it was re-added after the new renewal days later.

I will try updating certbot again as this is the version it updated to during last attempt.

1 Like

If DST Root CA X3 is being added back in to your chain your ACME client is using the default chain served by Let's Encrypt. As stated by @rg305 you can use the "preferred issuer" feature of your acme client to build a chain to "ISRG Root X1" instead.

Your comment that all of Let's Encrypts mails and posts are only relevant to Windows servers is odd as I'd argue it's the exact opposite - are you sure it's a Let's Encrypt post you are talking about? Certbot (the default acme client) is mainly used on linux but certainly lots of the posts on the support forum (conducted by people who don't work for Let's Encrypt) talk about Windows as well.

5 Likes

I don't think that's an option available for:

3 Likes

No, as you noted earlier it needs 1.12 or later

But, if they did not want to update Certbot or use a diff client was --deploy-hook available back then - do you know?

If they are clever programmers they could use that to automatically update the chain like they did manually so it would survive the renewal. Could also copy it to a different folder for clarity and use that in their server.

I think it would be easier to update Certbot or even use another acme client. Just noting this given I do not know how challenging that would be in their environ.

3 Likes

They wouldn't have to go that far.
Something I just learned today [how Seinfeldish]:
Apache can still use the deprecated SSLCertificateChainFile.
So...
They could change (the default):

SSLCertificateFile      /etc/letsencrypt/live/EXAMPLE/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/EXAMPLE/privkey.pem

To:

SSLCertificateFile      /etc/letsencrypt/live/EXAMPLE/cert.pem
SSLCertificateChainFile /PATH/TO/EDITED/CHAIN.PEM       # one-time copy/edit
SSLCertificateKeyFile   /etc/letsencrypt/live/EXAMPLE/privkey.pem

And that would continue to work after renewals.
[until the chain is changed - not expected anytime soon]

4 Likes

@rg305 Very clever. I am going to add your post to my fave tricks collection :slight_smile:

4 Likes

That is precisely how I am doing on CentOS. The module mod_ssl ships the /etc/httpd/conf.d/ssl.conf file. I have only one modification in the file, I change:

#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

to

SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

I create that file to be just a softlink pointing to the real chain file.

At the installation time of the mod_ssl, the two other files referred in ssl.conf

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

are created (self signed certificate and its key) by the rpm's post installation snippet.
Then, I replace those files with symbolic links to point to the Let's Encrypt's certificate and to the associated key file.

Additional advantage that all those statements are inside the default virtualhost definition:

<VirtualHost _default_:443>
</VirtualHost>

There is no need to define at any place the servername. Apache will automatically recognize and acts properly on all names found in the SAN section of the certificate.

4 Likes

More information:

3 Likes

But the problems could have been prevented, if Letsencrypt had decided to start signing with a new intermediate CA that is not cross signed with the old expiring root CA. If they had done that no one would have noticed that there was an old root certificate expiring.

Jan Hugo Prins

Hi @jhaprins and welcome to the LE community forum :slight_smile:

There is no preventing a root cert expiry.
Intermediates are never explicitly entered into any global trust stores - they are trusted because they are signed by a trusted root.
When a trusted root expires, it will also take with it any intermediates that relied on it.
There does not exist another such intermediate that could have been used.

I think you need to rethink/relearn how the PKI system operates.

2 Likes

I know very well how certificates etc. work. What I say is the following:
If Letsencrypt had started signing new certificates, let's say half a year ago, with a new intermediate certificate that is not signed by the old root certificate, then all current certificates would not have the old root certificate in the trust chain and we wouldn't have seen all those problems.

Jan Hugo

Do you mean "ISRG Root X1" which was created in 2015?
If so, that is exactly how they are being signed today.

There are two trust chains served today by LE.
Both go through "ISRG Root X1".
One merely uses that same root cert as an intermediate (cross-signed by another root cert).
If you are having trouble with one trust path, you can use the other.
Both can't be set as default - only one can.
No matter which one was made default, the other side would have issue with that choice.

You can please:
Some of the people all of the time.
All of the people some of the time.
But never all of the people all of the time.

Which leads to trust path option #3: Use an alternate CA.

There is no silver bullet that can fix the current lack of updated systems being used on the Internet.

To be blunt: People smarter than you and I (put together) have reviewed this problem and haven't found such a solution to exist.

2 Likes

Ah, I see what the problem is. The new root is cross signed by the old root. And indeed, there is no way to remove that cross signature.

We removed the old root from the trust-store of all our systems, but indeed, that is a client side solution that doesn't scale.

I think the problem could have been prevented in the past if there would have been a second root in the global root trust store that was not signed by the old root. Then a new intermediate certificate could have been created that was signed by the new clean root certificate, and then this problem wouldn't exist at all.

The intermediate certificate R3 signed by DST Root CA X3 has been retired May this year already. So Let's Encrypt has already done what you're suggesting.

4 Likes

Sure, if one could travel back in time [even before LE even existed] and add an LE trusted root to all the now extremely outdated trust stores still in use, then viola!

2 Likes

The problem is how to support a vast number of old android devices that are not getting updates. Technology ... meet Reality :slight_smile:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.