Help thread for DST Root CA X3 expiration (September 2021)

rg305 · September 29, 2021, 11:28pm

Just edit the file mentioned above:

To use the newly created file with the contents provided.
[did I fail to mention that you should just make a new file with those certs? - in case you want to revert]

MagnusW · September 29, 2021, 11:30pm

Yes, this seems to be seen only on some devices. I had to remove the OLD Let's Encrypt intermediate certificates via certmgr.msc / Intermediate Certification Authorities / Certificates. Be sure to let the NEW cert be (expiry 2025/09/15), and remove the old certs (expiry dates 2021/09/29 and 2021/03/something here, there may be even more LE certs there).

On one system, restarting IIS after this fixed the issue, while a different system had to be restarted altogether.

rg305 · September 29, 2021, 11:30pm

And we probably have less of an idea here...
In this case, I would consider the "In case of emergency, break glass!" approach and switch to another FREE CA [until this gets resolved].

rg305 · September 29, 2021, 11:31pm

Hi @MagnusW, welcome to the LE community forum

Yeah, Windows is a strange beast!

MagnusW · September 29, 2021, 11:42pm

Thanks!

Yeah, prioritizing old intermediates over new, if that is what's happening, sure looks like a bug to me.

rg305 · September 29, 2021, 11:45pm

It seems to take first from order: alphabetically, Not Before Date, etc.
[or something even sillier than that]

rmbolger · September 29, 2021, 11:51pm

Pretty sure we determined that Windows was prioritizing Not Before (the issue date) instead of Not After (the expiration date). And the expiring R3 was unfortunately created after the non-expiring R3. Though this seems to have been changed on Windows Server 2022.

PaulWiggy · September 30, 2021, 12:37am

I've managed to fix this - as mentioned elsewhere in this chain. I deleted the old certs that were incorrectly being picked up by IIS in the chain - In MMC I removed Intermediate Cert Auth - R3 (I had 2 old ones) and Trusted Root - DST Root CA X3. I then had to re-boot my server (restarting IIS didn't fix it), now working good. I use a CDN so also had to re-export the SSL's and install of the CDN.

Hope this helps others

rpestl · September 30, 2021, 3:22am

Unbelievable that expired certificates are being found by some clients when valid ones exist. Another option I found to solve this problem on windows was to set a "PreferredIssuer": "ISRG Root X1" in my win-acme client. Then forced and each SSL certificate to renew. This uses the ISRG Root X1 intermediate instead of the expired DST Root. This required NO reboot of IIS or the server.

webprofusion · September 30, 2021, 3:27am

The act of re-binding your certs will have forced the change (because the old R3 is now expired windows won't use it to build new chains but may be hanging on to old cached ones). The Preferred Issuer preference in win-acme was not the root cause of the fix. Windows builds it own chains and ignores the ones you give it.

nulltrope · September 30, 2021, 3:46am

We have our own home-rolled ACME client written in Java that makes requests to the LE acme v2 API and started seeing Java SSL validation errors a few hours ago for all API requests with the following in the stack trace:

javax.net.ssl.SSLHandshakeException: sun.security.validator.ValidatorException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

Turns out Java 1.8 does not continue to respect the the DST root after it expired, and the truststore does not have the ISRG root by default (at least, ours did not). The fix was to manually import the new ISRG root into the Java truststore as follows:

keytool -import -alias isrgrootx1 -keystore $JAVA_HOME/jre/lib/security/cacerts -trustcacerts -file isrgrootx1.cer

Just wanted to drop a note here in case anyone else faces a similar issue! And good luck to everyone else staying up late (or waking up early) to keep their sites online

rpestl · September 30, 2021, 3:50am

Totally makes sense thanks. I wish windows had figured that out when building the previous chain... Would have saved me some stress this evening.

MagnusW · September 30, 2021, 9:26am

THANKS! I can confirm that IIS/Exchange only needs re-binding, with minimal downtime. Do it like this:

Edit Bindings for the Exchange site (Default Web Site)
Edit one of the 443 bindings
Choose a different cert from the drop-down, anything goes. Do NOT click OK. Choose the correct (latest) cert again. Click OK. Windows will now re-bind with the correct intermediate.
Do the same with the other 443 binding.
Optionally, if you are using the LE cert for Exchange Back End, edit its bindings and do the same there (there will be only one binding).

Do note that I had already removed the old intermediates from the cert store.

tsolg2001 · September 30, 2021, 9:28am

Here is the fix for Opnsense.
ios devices working again,

Go to OpnSense machine in CLI and select shell option
Execute opnsense-patch -c plugins ae69739 9220a41 247408e 774374a4 => All patches have been applied successfully. Have a nice day.
Go to Services: ACME Client: Certificates and click on new button Re-import on my FQDN row
Restart HAProxy
Try curl call, same result : curl: (60) SSL certificate problem: unable to get local issuer certificate
Execute openssl s_client -showcerts -connect FQDN:443 : Same result, missing X1
Go check System: Certificats: FQDN issuer => R3 (Let's Encrypt)

bllackpython · September 30, 2021, 9:30am

Hi All,
Our proxies have suddenly started blocking a bunch of websites (as it's blocking on day it seems).
An example is:
https://www.dailyecho.co.uk/

I see the DST root ca in the chain, is that supposed to be right?

kahootali · September 30, 2021, 9:32am

Hi,

We had our certs generated from vancluever/acme and running on nginx using Nginx Ingress Controller on Kubernetes and it failed today, after DST -> R3 cert expired, and now I have recreated the cert to use ISRG Root X1 as preferred chain and applied it, the issue is on chrome, website is still showing cert failed, if i see the chain it is DST Root -> R3 > cert, but if I open in Firefox, it gives ISRG -> R3 -> cert which works fine, also I tried the cert on openssl,

openssl s_server -accept 8080 -www -cert tls.crt -key tls.key -CAfile ca.crt
it shows correct chain, what am i missing?
its not working on Chrome for most of the users but working fine for some?

mts · September 30, 2021, 9:52am

Yes, thank you! It's working now.

youfu · September 30, 2021, 9:53am

After switching the preferred chain from DST Root CA X3 to ISRG Root X1, I can no longer receive WebHook (specificaly, DingTalk bot outgoing webhook) from Alibaba.
They deployed their proprietary Alibaba JDK 8 which does not include ISRG Root X1, whilst OpenJDK includes ISRG Root X1 since 2017-07-18.

This would trouble more people when they try to switch the chain for OpenSSL < 1.1.0 compatibility.

mildsar · September 30, 2021, 10:12am

I see that you re-issued your certificate today. Did it resolve problem?

Osiris · September 30, 2021, 10:59am

Slight supplement/warning to the above command: never ever just copy/paste and run such commands if you don't know what they are doing! The listed patches are:

I urge every OpnSense user to review these patches before blindly applying them. Although luckily I assume the opnsense-patch utility will only use commits from the official repository..