The real issue here is a 2 parter:
Combined, this is currently a deal breaker come Sept 2021, at least for our use case. The first one can be a problem for everyone running W10 with no ISRG root, come Sept 2021.
Not if they use IE/Edge (at least once - to any site with such a root)
Something RasDial(Microsoft?) needs to think about (and fix).
You are implying then that no change should ever be made - but the Internet is built on change.
At some point all the current roots will have expired... then what?
@yegor If that Lazy loading is a general thing, subject to all certs and not only the ISRG root, why would the DST Root CA X3 root certificate be any different compared to the ISRG root?
DST root has been around since 2000 (XP era). ISRG was not packaged into Windows until mid 2018, and as evidence shows, it's not present in all cases, even on post mid-2018 Windows releases.
@rg305 I'm not suggesting change is bad, I'm simply pointing out an issue which is a problem for our use case, and our use case is not unique.
The "fix" has to come from the programs using the root store.
There is a method in place to update and keep that root store updated.
Why RASDIAL fails to implement that process is unknown - but that is where the problem starts.
I don't follow. I'm confused. Earlier @aarongable told us about the lazy loading of root certs in Windows.. But that isn't applicable to all root certificates? Is the lazy loading story only applicable to some, i.e. more recent root certificates?
I found the following in http://woshub.com/updating-trusted-root-certificates-in-windows-10/:
certutil.exe -generateSSTFromWU roots.sst
Not sure if the .sst file is the same as your linked .stl, but it seems to be some kind of Windows Update (WU) root certificate list downloader. Not sure if it uses http:// or https:// though, can't test it from my Linux workstation 
This MS document also refers to some other certutil options: An update is available that enables administrators to update trusted and disallowed CTLs in disconnected environments in Windows - Microsoft Support
CertUtil [Options] -syncWithWU DestinationDir
Which will also get you Authrootstl.cab which can be extracted to get the .stl file. Also not sure if that would download it through HTTPS.
.sst = Serialized Certificate Store File
.stl = Should have been CTL [Certificate Trust List] (but that extension was already used)
They seem to do/provide the same thing.
hmm...
Most of those actions require multiple steps and administrative user access.
Going to a website using IE/Edge requires none of that.
I've done some more digging. The issue is as follows, based on my understanding and a bit more fiddling around:
That being said, non-browser OS subsystems (like rasdial, maybe others) do NOT trigger this flow and the chain cannot be verified.
So the MS local root store consists partly of locally hardcoded root certs and partly of a list of possible to-be downloaded root certs from Windows Update? Very strange...
Hello,
I would like to chime in here. I too can confirm the general sentiment of this thread.
My current conclusions are very loosely:
Overall this seems to imply lazy-loading of root certificates in Windows 10 is BROKEN for at the very least IKEv2/IPsec connections which is bad. This means that VPN users are at the mercy of some other mechanism in windows to trigger the loading the ISRG Root X1 prior to connecting to the VPN and hence the intermittent/inconsistent reports and/or lack of reports in general. I do not see how this is at all resolvable without intervention from Microsoft. Even then, lets say Microsoft pushes an update. If nobody takes the update, the problem still exists. This is a major problem for LetsEncrypt users relying on moving to ISRG Root X1 after DST Root CA X3 expires.
I would love to hear from anyone more familiar with the inner workings of Microsoft Windows and it's trusted root store about how this can be addressed. There are undoubtedly people using LetsEncrypt for VPN connections and this is a huge problem that nobody seems to be talking about!
@hft @yegor, does the lazy-loading trigger successfully if you just load an image in the browser via <img src> or something?
@yegor is there any way to get your users to run native code, or browse somewhere, that will result in a browser triggering the lazy-load process? Could they, for example, visit your own web site in a browser to trigger it? (If an image load would trigger it, you could inline load an image from a site that presents a chain to X1 even if you don't use such a chain on your main site.)
Overall I agree with @hft's thought that you should probably get Microsoft to comment on whether there's an intended official solution for non-browser Windows software that may need a root that's not yet present locally.
While that will probably work schoen (and I have seen that simply going to a URL of any kind that contains the CA will add it) it is not a solution for my case unfortunately, only a workaround for those who have the luxury of building their own software around this issue. I believe this is why this has largely gone unnoticed.
The be clear the example where this happens is when you are configuring a native/core component of windows (that is IKEv2 connectivity which relies on valid server TLS) on its own and it can't trust the root. That is, you have no supporting software to pre-trigger the fetching of the root. To get a rough idea of how common this may be, simply search for "IKEv2 LetsEncrypt" on google.
When I first came across this I immediately started looking for any other references to the issue, and then started looking at if there were any reported ISRG Root X1 issues with Windows 10 in general (didnt really see any...) While it's clearly been added to their trust root program, it seems as though it works for everybody under most use cases. What I am puzzled by is a lack of clear messaging saying "Windows 10 is supported by LetsEncrypt roots!" For example the only mention of Windows is not directly suggesting Windows 10 support exists here https://letsencrypt.org/docs/certificate-compatibility/. This lead to some second guessing of these findings and wondering if it was even supposed to work in the first place 
Yes, it would definitely be a better solution if Microsoft would proactively ship the ISRG root and/or make non-browser uses of TLS trigger the fetch.
Did some more fiddling around, noticed the following:
Assuming this was legacy Windows PowerShell 5.1 or earlier rather than the newer PowerShell Core 6 or 7, you likely weren't actually running the real wget. You were running PowerShell's native Invoke-WebRequest function which has a default alias of wget.
PS C:\> get-alias wget
CommandType Name Version Source
----------- ---- ------- ------
Alias wget -> Invoke-WebRequest
Subsequently, Invoke-WebRequest is just using .NET Framework http client libraries under the hood which are themselves using the same lower level Win32 API http client libraries that things like Internet Explorer do as well which I would guess is why the fetch happens. Things that don't trigger the fetch are likely using even lower level raw socket libraries.
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.