Why is letsencrypt not fixing DST Root CA X3 issue

Help
1

My domain is: multiple (saicanews.co.za)

I ran this command: none

It produced this output: none

My web server is (include version): Apache 2.4.18 however various servers with different versions

The operating system my web server runs on is (include version): Ubuntu server 16.04 and later

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

Good day everyone.

I would like to know why Letsencrypt has expired the R3 CA , however they are not implementing a fix for removing that from the certificate chain.

We have been using letsencrypt and certbot for many years now, and have always been happy, however now they have expired the R3CA and it is causing issues all over the place.

All fixes found on the net is related to IIS which is not applicable to our environment, however trying to contact letsencrypt to confirm when a fix will be deployed, they simply shun you away stating they do not supply support.

So my question is , how do they expect people to keep using their service if they do not provide information or support when they decide to make a major change. This will result in a lot of people moving away to paid for SSL certificates.

Secondly I would like to know if anybody have a formal fix for linux servers in this regard as I have not been able to find any method or support on the net on how to set the letsencrypt certificates to not use the R3 CA.

Apologies for my rant but it is frustrating that a company we trusted for many years refuse to assist when they themselves made a major change and all of their mails and posts are only relevant to windows servers.

2

R3 hasn't expired.

3 Likes
3

Apologies, not R3

This is causing issues as all the customer browsers is now stating that the sites are insecure.

We have tried to renew the certificates with new ones, however this does not solve the issue as it is still picking up this chain.

My issue is with why has a fix not been implemented to fix this on the certificates or even on certbot to cater for this.

4

Hi @machiel and welcome to the LE community forum :slight_smile:

I think you are going beyond a "rant" with:

When this is your first post about it and we are assisting YOU with YOUR problem.
Let me explain:
There is nothing technically wrong with the trust path being served by your web site.
See: [SSL Server Test: saicanews.co.za (Powered by Qualys SSL Labs)]
(SSL Server Test: saicanews.co.za (Powered by Qualys SSL Labs))
[OR use any other online certificate verification tool]
You can compare that trust path to the one this very community uses.
See: SSL Server Test: community.letsencrypt.org (Powered by Qualys SSL Labs)
They are exactly the same.

That said, there are (mostly older) clients that are unable to correctly validate that trust path.
Admittedly (IMHO), for more reasons than were anticipated.
But when you say:

You make it sound like there was an option where a certificate expiration date could simply be avoided and it only happened because it was the chosen path.
Nothing in the world can stop a certificate from expiring.
Once a certificate is issued, it has a start date and an end date (expiration).
That goes for every cert in the entire PKI system.

If you would care to read though their site (like within their announcement pages), you would see that there are two available trust paths.

  • the longer/default trust path which includes the now expired "DST Root CA X3" root.
    [note that's an X3 (not an R3) - there is no expired R3 in any of the current trust paths]
  • the shorter/alternate trust path which uses the proper self-signed "ISRG Root X1" root.
    [issued in 2015 and now in all major trust stores]

If you need help switching trust paths, you can either search this site (and find dozens of such examples or simply ask for help with that).

If you don't understand something, then you can also ask for clarification/explanation.
There is no conspiracy, no conscious decision was made to mess up the Internet PKI system.
None of that is happening.
What is happening (by and large) is that systems that have NOT updated their trust stores since 2015 are now having trust issues. And some systems (mainly Windows), that even though they do have updated trust stores fail to see (and trust) the "new" (since 2015) trusted root in the longer trust path.
All of this would be happening with every other CA, had their root cert expired sooner than this one did.
And this will likely continue to happen (also with other CAs) as long as older devices are still being used long long long after their manufacturers have stopped providing any support/updates for them or as long as people don't take any steps to keep their systems updated.

6 Likes
5

If you are to serve the shorter/alternate chain, you must now do one of these four things:

  • manually remove the last cert from the fullchain.pem (or cert.pem) file used.
    [not an automated solution - not recommended]
  • update your certbot 0.31.0 to a version that supports the parameter --preferred-chain
    [you would need certbot 1.12 or higher]
  • switch to another (newer) ACME client - like: acme.sh
    [which has support for the --preferred-chain parameter]
  • switch to another (FREE and ACME friendly) CA
    [with implies the use of another completely different trust path - without "DST Root CA X3" in it]

TIME CHECK...
Solution provided within 1 hour of your RANT - LOL

5 Likes
6

@rg305

Thank you for the solution path.

Just as an explanation, we did remove the last cert in the fullchain as well based on an article we found, however as stated this was not a proper fix as it was re-added after the new renewal days later.

I will try updating certbot again as this is the version it updated to during last attempt.

1 Like
7

If DST Root CA X3 is being added back in to your chain your ACME client is using the default chain served by Let's Encrypt. As stated by @rg305 you can use the "preferred issuer" feature of your acme client to build a chain to "ISRG Root X1" instead.

Your comment that all of Let's Encrypts mails and posts are only relevant to Windows servers is odd as I'd argue it's the exact opposite - are you sure it's a Let's Encrypt post you are talking about? Certbot (the default acme client) is mainly used on linux but certainly lots of the posts on the support forum (conducted by people who don't work for Let's Encrypt) talk about Windows as well.

5 Likes
8

I don't think that's an option available for:

3 Likes
9

No, as you noted earlier it needs 1.12 or later

But, if they did not want to update Certbot or use a diff client was --deploy-hook available back then - do you know?

If they are clever programmers they could use that to automatically update the chain like they did manually so it would survive the renewal. Could also copy it to a different folder for clarity and use that in their server.

I think it would be easier to update Certbot or even use another acme client. Just noting this given I do not know how challenging that would be in their environ.

3 Likes
10

They wouldn't have to go that far.
Something I just learned today [how Seinfeldish]:
Apache can still use the deprecated SSLCertificateChainFile.
So...
They could change (the default):

SSLCertificateFile      /etc/letsencrypt/live/EXAMPLE/fullchain.pem
SSLCertificateKeyFile   /etc/letsencrypt/live/EXAMPLE/privkey.pem

To:

SSLCertificateFile      /etc/letsencrypt/live/EXAMPLE/cert.pem
SSLCertificateChainFile /PATH/TO/EDITED/CHAIN.PEM       # one-time copy/edit
SSLCertificateKeyFile   /etc/letsencrypt/live/EXAMPLE/privkey.pem

And that would continue to work after renewals.
[until the chain is changed - not expected anytime soon]

4 Likes
11

@rg305 Very clever. I am going to add your post to my fave tricks collection :slight_smile:

4 Likes
12

That is precisely how I am doing on CentOS. The module mod_ssl ships the /etc/httpd/conf.d/ssl.conf file. I have only one modification in the file, I change:

#SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

to

SSLCertificateChainFile /etc/pki/tls/certs/server-chain.crt

I create that file to be just a softlink pointing to the real chain file.

At the installation time of the mod_ssl, the two other files referred in ssl.conf

SSLCertificateFile /etc/pki/tls/certs/localhost.crt
SSLCertificateKeyFile /etc/pki/tls/private/localhost.key

are created (self signed certificate and its key) by the rpm's post installation snippet.
Then, I replace those files with symbolic links to point to the Let's Encrypt's certificate and to the associated key file.

Additional advantage that all those statements are inside the default virtualhost definition:

<VirtualHost _default_:443>
</VirtualHost>

There is no need to define at any place the servername. Apache will automatically recognize and acts properly on all names found in the SAN section of the certificate.

4 Likes
13

More information:

3 Likes
14

But the problems could have been prevented, if Letsencrypt had decided to start signing with a new intermediate CA that is not cross signed with the old expiring root CA. If they had done that no one would have noticed that there was an old root certificate expiring.

Jan Hugo Prins

15

Hi @jhaprins and welcome to the LE community forum :slight_smile:

There is no preventing a root cert expiry.
Intermediates are never explicitly entered into any global trust stores - they are trusted because they are signed by a trusted root.
When a trusted root expires, it will also take with it any intermediates that relied on it.
There does not exist another such intermediate that could have been used.

I think you need to rethink/relearn how the PKI system operates.

2 Likes
16

I know very well how certificates etc. work. What I say is the following:
If Letsencrypt had started signing new certificates, let's say half a year ago, with a new intermediate certificate that is not signed by the old root certificate, then all current certificates would not have the old root certificate in the trust chain and we wouldn't have seen all those problems.

Jan Hugo

17

Do you mean "ISRG Root X1" which was created in 2015?
If so, that is exactly how they are being signed today.

There are two trust chains served today by LE.
Both go through "ISRG Root X1".
One merely uses that same root cert as an intermediate (cross-signed by another root cert).
If you are having trouble with one trust path, you can use the other.
Both can't be set as default - only one can.
No matter which one was made default, the other side would have issue with that choice.

You can please:
Some of the people all of the time.
All of the people some of the time.
But never all of the people all of the time.

Which leads to trust path option #3: Use an alternate CA.

There is no silver bullet that can fix the current lack of updated systems being used on the Internet.

To be blunt: People smarter than you and I (put together) have reviewed this problem and haven't found such a solution to exist.

2 Likes
18

Ah, I see what the problem is. The new root is cross signed by the old root. And indeed, there is no way to remove that cross signature.

We removed the old root from the trust-store of all our systems, but indeed, that is a client side solution that doesn't scale.

I think the problem could have been prevented in the past if there would have been a second root in the global root trust store that was not signed by the old root. Then a new intermediate certificate could have been created that was signed by the new clean root certificate, and then this problem wouldn't exist at all.

19

The intermediate certificate R3 signed by DST Root CA X3 has been retired May this year already. So Let's Encrypt has already done what you're suggesting.

4 Likes
20

Sure, if one could travel back in time [even before LE even existed] and add an LE trusted root to all the now extremely outdated trust stores still in use, then viola!

2 Likes