Microsoft Windows Root Certificate Lazy-Loading

[orangepizza]

TL;DR: do one of those, anything will work
open https://valid-isrgrootx1.letsencrypt.org from IE, edge, or chrome (firefox uses own store so doesn't work here) Or
Run this one liner command to run in windows clinet to fetch the ISRG root from MS
powershell invoke-webrequest -usebasicparsing https://valid-isrgrootx1.letsencrypt.org
it will spit out result of webrequest but you can discard it.

then restart your problematic program.

powershell invoke-webrequest uses MS's verification stack so it can fetch new root certificate from MS
-usebasicparser is because otherwise you need initialized internet explorer to run, which I can't assume

Problem is windows doesn't have all trusted root certificate by default, but download from MS server when they first need to verification against that.(Lazy-loading) but if a program load trust store from local file run their own verification stack, ISRG root cert isn't in local file so it's untrusted, and it won't ask MS server to load new root certificate.

so to fix this you should connect something that uses ISRG root certificate from program that use windows internal verification process. for TLDR I pick powershell invoke-webrequest, but you can use anything you are easy to work with.

for long term solution as programmer that write the client, you should either use windows' cert verification stack, or bring your own trust store with your own trust anchor (like from Mozilla's NSS) loading (system) CAs from windows than using your own verification functions is not a valid option. (if you use latter you probably should also load user-trusted enterprise CAs added in windows too)

---

wrote some to use as point-to thread from problems with "windows don't have Isrg in local yet beacause lazyloading" problem.

any improvement will be welcome and need better name for this thread to draw attention

[rmbolger]

The PowerShell example can be reduced to this if you're already in a PowerShell terminal:

$null=iwr https://valid-isrgrootx1.letsencrypt.org -useb

Or this if you're running it from legacy cmd.exe or the Run dialog:

powershell -C "$null=iwr https://valid-isrgrootx1.letsencrypt.org -useb"

[s0ftice]

thanks works great. is this permanent, or does it have to be re-run (after reboot/certain time)?

[orangepizza]

should be permanent in that release (not sure if it survive upgrade like 20H2 to 21H1 or like though)

[Nummer378]

On the topic of root lazy-loading, I did observe the following behaviour when trying to load ISRG Root X2:

If the system already knows the cross-sign of ISRG Root X2 (signed by ISRG Root X1), it will use that for path building and will not load ISRG Root X2 self-signed into the trusted root store. This even applies when visting sites like valid-isrgrootx2.letsencrypt.org, which do not send the cross-sign themselves.

On such a system, in order to get the system to lazy load ISRG Root X2, you need to remove the cross-sign first. In order to do that, I had to open certmgr as administrator (I did this by opening it from an elevated command prompt), then looking at Intermediates and either deleting ISRG Root X2-signed-by-ISRG Root X1, or by moving it to untrusted. Note that certmgr shows different certificates depending on if you call it elevated or not and it can also differ between user accounts.

Then simply visting valid-isrgrootx2.letsencrypt.org from a client that uses the Windows platform verifier caused Windows to lazy load the self-signed version of ISRG Root X2 immediatly.

Note that compatibility-wise it makes no difference whether the system builds a path up to ISRG Root X1 or ISRG Root X2 - they're both trusted and their EKU's are the same. Their only difference is the expiry date, and I hope that Windows will lazy-load a "better" version of ISRG Root X2, once the cross-sign expires (in about 4 years).

This is just interesting to know in case you wonder why your system might not load ISRG Root X2 - Windows can be very lazy.

[Ted]

This isn't just a windows thing I observed the same "lazy" behavior from an older Thunderbird version.

[Nummer378]

Mozilla products in general do not do lazy loading of root certificates - their trust store is fixed and shipped with the client, so any 2016+ version should just trust ISRG Root X1. ISRG Root X2 is not included in Mozilla's Root program yet.

[Ted]

I had manually added the root and intermediate certs to the Thunderbird cert store. And to the mailserver. Still did not work. Then, 15-20 minutes later I looked in the Tbird cert stores one more time - made no changs -and then it started working.

Obviously I did something that flushed a cache or rebuilt a chain or something. At the time I wasn't attempting to troubleshoot Thunderbird. I was attempting to troubleshoot Lets Encrypt. Otherwise I would have documented everything more carefully and come up with a step by step.

There are still to this day display bugs in Thunderbird that have been there for 10 years at least that I see pop up from time to time that are not easily reproducible. One of the most annoying is the message pane just turning itself on in the layout under certain situations. But it's par for the course for that, Thunderbird strives to be very annoying software.