When this is your first post about it and we are assisting YOU with YOUR problem.
Let me explain:
There is nothing technically wrong with the trust path being served by your web site.
See: [SSL Server Test: saicanews.co.za (Powered by Qualys SSL Labs)]
(SSL Server Test: saicanews.co.za (Powered by Qualys SSL Labs))
[OR use any other online certificate verification tool]
You can compare that trust path to the one this very community uses.
See: SSL Server Test: community.letsencrypt.org (Powered by Qualys SSL Labs)
They are exactly the same.
That said, there are (mostly older) clients that are unable to correctly validate that trust path.
Admittedly (IMHO), for more reasons than were anticipated.
But when you say:
You make it sound like there was an option where a certificate expiration date could simply be avoided and it only happened because it was the chosen path. Nothing in the world can stop a certificate from expiring.
Once a certificate is issued, it has a start date and an end date (expiration).
That goes for every cert in the entire PKI system.
If you would care to read though their site (like within their announcement pages), you would see that there are two available trust paths.
the longer/default trust path which includes the now expired "DST Root CA X3" root.
[note that's an X3 (not an R3) - there is no expired R3 in any of the current trust paths]
the shorter/alternate trust path which uses the proper self-signed "ISRG Root X1" root.
[issued in 2015 and now in all major trust stores]
If you need help switching trust paths, you can either search this site (and find dozens of such examples or simply ask for help with that).
If you don't understand something, then you can also ask for clarification/explanation.
There is no conspiracy, no conscious decision was made to mess up the Internet PKI system.
None of that is happening.
What is happening (by and large) is that systems that have NOT updated their trust stores since 2015 are now having trust issues. And some systems (mainly Windows), that even though they do have updated trust stores fail to see (and trust) the "new" (since 2015) trusted root in the longer trust path.
All of this would be happening with every other CA, had their root cert expired sooner than this one did.
And this will likely continue to happen (also with other CAs) as long as older devices are still being used long long long after their manufacturers have stopped providing any support/updates for them or as long as people don't take any steps to keep their systems updated.
Just as an explanation, we did remove the last cert in the fullchain as well based on an article we found, however as stated this was not a proper fix as it was re-added after the new renewal days later.
I will try updating certbot again as this is the version it updated to during last attempt.
If DST Root CA X3 is being added back in to your chain your ACME client is using the default chain served by Let's Encrypt. As stated by @rg305 you can use the "preferred issuer" feature of your acme client to build a chain to "ISRG Root X1" instead.
Your comment that all of Let's Encrypts mails and posts are only relevant to Windows servers is odd as I'd argue it's the exact opposite - are you sure it's a Let's Encrypt post you are talking about? Certbot (the default acme client) is mainly used on linux but certainly lots of the posts on the support forum (conducted by people who don't work for Let's Encrypt) talk about Windows as well.
But, if they did not want to update Certbot or use a diff client was --deploy-hook available back then - do you know?
If they are clever programmers they could use that to automatically update the chain like they did manually so it would survive the renewal. Could also copy it to a different folder for clarity and use that in their server.
I think it would be easier to update Certbot or even use another acme client. Just noting this given I do not know how challenging that would be in their environ.
They wouldn't have to go that far.
Something I just learned today [how Seinfeldish]: Apache can still use the deprecated SSLCertificateChainFile.
So...
They could change (the default):
That is precisely how I am doing on CentOS. The module mod_ssl ships the /etc/httpd/conf.d/ssl.conf file. I have only one modification in the file, I change:
are created (self signed certificate and its key) by the rpm's post installation snippet.
Then, I replace those files with symbolic links to point to the Let's Encrypt's certificate and to the associated key file.
Additional advantage that all those statements are inside the default virtualhost definition:
<VirtualHost _default_:443>
</VirtualHost>
There is no need to define at any place the servername. Apache will automatically recognize and acts properly on all names found in the SAN section of the certificate.
But the problems could have been prevented, if Letsencrypt had decided to start signing with a new intermediate CA that is not cross signed with the old expiring root CA. If they had done that no one would have noticed that there was an old root certificate expiring.
Hi @jhaprins and welcome to the LE community forum
There is no preventing a root cert expiry.
Intermediates are never explicitly entered into any global trust stores - they are trusted because they are signed by a trusted root.
When a trusted root expires, it will also take with it any intermediates that relied on it.
There does not exist another such intermediate that could have been used.
I think you need to rethink/relearn how the PKI system operates.
I know very well how certificates etc. work. What I say is the following:
If Letsencrypt had started signing new certificates, let's say half a year ago, with a new intermediate certificate that is not signed by the old root certificate, then all current certificates would not have the old root certificate in the trust chain and we wouldn't have seen all those problems.
Do you mean "ISRG Root X1" which was created in 2015?
If so, that is exactly how they are being signed today.
There are two trust chains served today by LE.
Both go through "ISRG Root X1".
One merely uses that same root cert as an intermediate (cross-signed by another root cert).
If you are having trouble with one trust path, you can use the other.
Both can't be set as default - only one can.
No matter which one was made default, the other side would have issue with that choice.
You can please: Some of the people all of the time. All of the people some of the time. But never all of the people all of the time.
Which leads to trust path option #3: Use an alternate CA.
There is no silver bullet that can fix the current lack of updated systems being used on the Internet.
To be blunt: People smarter than you and I (put together) have reviewed this problem and haven't found such a solution to exist.
Ah, I see what the problem is. The new root is cross signed by the old root. And indeed, there is no way to remove that cross signature.
We removed the old root from the trust-store of all our systems, but indeed, that is a client side solution that doesn't scale.
I think the problem could have been prevented in the past if there would have been a second root in the global root trust store that was not signed by the old root. Then a new intermediate certificate could have been created that was signed by the new clean root certificate, and then this problem wouldn't exist at all.
The intermediate certificate R3 signed by DST Root CA X3 has been retired May this year already. So Let's Encrypt has already done what you're suggesting.
Sure, if one could travel back in time [even before LE even existed] and add an LE trusted root to all the now extremely outdated trust stores still in use, then viola!