Old R3 Intermediate Cert will impact the ISRG Root X1 Transition

[jmboby]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

Production-Grade Container Orchestration

cert-manager

https://community.letsencrypt.org
...any domain with LE issued certs

I ran this command:
Browsing via Chrome/Firefox/Opera on Win10 local machine

It produced this output:
When checking the TLS cert and issuer chain I noticed it was the old chain and did not include ISRG at all!. I then spotted the old R3 Intermediate cert is in the certificates store on my Win10 machine and also within Firefox, and this is also true for my colleagues, we are running a typical enterprise Win10 image with regular patching.
The old R3 points shows the issuer as 'DST Root CA X3' with NO mention of ISRG Root X1. This R3 cert expires 30th Sept 2021.

So my question is won't this break the transistion for many clients in the next couple of days as both old R3 and DST Root CA X3 expire. How widespread is this? We are running a typical enterprise Win10 image with regular patching.

I manually fixed by downloading the newer R3 https://letsencrypt.org/certs/lets-encrypt-r3.der and deleting the old from my local cert store, this then reflected the correct chain with ISRG being part a part of it.

My web server is (include version):
Chrome: 93.0.4577.63
Firefox: 92.0.1
Opera: 79.0.4143.66

The operating system my web server runs on is (include version):
Win10

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
N/A

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
We are using Cert-manager to issue LE certs on K8, however this is not relevant for this issue. The relevant issue is old R3 intermediate certs still hanging around in many clients it seems!

[rmbolger]

The chain you see in the browser is not necessarily the chain being served by the web server. You'd need to use something like ssllabs.com or openssl s_client to check what is actually being served.

If the R3 being served by the webserver is actually the old expiring R3 and it doesn't get replaced after it expires, then yes. That site will cause clients to throw errors. But that's not something you can really do anything about unless you also operate the site in question. It could also cause client errors if the clients remove the expiring DST root before it actually expires.

But in many cases, the site is either serving the newer R3 or no chain at all (which is its own problem). And it's just your browser/OS that is choosing to currently validate via the old R3 chain...which will change as soon as that chain is no longer valid.

[jmboby]

@rmbolger

erved by the webserver is actually the old expiring R3 and it doesn't get replaced after it expires, then yes. That site will cause clients to throw errors. But that's not something you can really do anything about unless you also operate the site in question. It could also cause client errors if the clients remove the expiring DST root before it actually expires.

But in many cases, the site is either se

Yes I agree, this is client side issue rather than server. The problem is that the new R3 needs to be present in the clients certificate store otherwise when the chain is no longer valid (in the next couple of days) the client browser will not trust the chain at all right?

[rmbolger]

@jmboby

The problem is that the new R3 needs to be present in the clients certificate store otherwise when the chain is no longer valid (in the next couple of days) the client browser will not trust the chain at all right?

Not necessarily. Intermediate certs like R3 are typically loaded on demand or just use the copy provided by the server. What matters is whether the root is trusted/valid.

And on Windows specifically, even the Root certificates can be loaded on demand. Freshly installed Windows OSes don't necessarily trust either the DST Root or ISRG Root. But the first time you visit a site that chains to them, they'll get downloaded to the Trusted Roots cert store.

[jmboby]

@rmbolger

Intermediate certs like R3 are typically loaded on demand or just use the copy provided by the server. What matters is whether the root is trusted/valid.

And on Windows specifically, even the Root certificates can be loaded on demand. Freshly installed Windows OSes don't necessarily trust either the DST Root or ISRG Root. But the first time you visit a site that chains to them, they'll get downloaded to the Trusted Roots cert store.

Ah I see! well that's an easy fix then. I didn't realise clients would download-on-demand.

Interesting to know the Windows also downloads root certs on demand, I just presumed they had to be pre-loaded. Thanks for your comments around openssl s_client and seeing what is 'actually' served by the webserver........I will go and check what our relevant webservers are serving up.

[jmboby]

I tried deleting my local R3 Intermediate cert and then browsing to the relevant website, just to test out the download-on-demand. I didn't see any evidence of it downloading the R3 but as you say - what matters is whether the root cert is trusted or not..........and in this case the ISRG Root X1 is in my Trusted CA store so all is good, and full chain was trusted by my browser.

Thanks @rmbolger, much appreciated!

[rmbolger]

@jmboby

I tried deleting my local R3 Intermediate cert and then browsing to the relevant website, just to test out the download-on-demand. I didn't see any evidence of it downloading the R3

If you were looking in the Local Computer's Intermediate store, try checking the Current User's Intermediate store instead. For example, compare the outputs of these two PowerShell commands.

# "CA" is the low level name for Intermediate Certifications Authorities
Get-ChildItem Cert:\LocalMachine\CA
Get-ChildItem Cert:\CurrentUser\CA

On my own Win10 machine, there's only 5 certs in LocalMachine\CA (none relating to Let's Encrypt) and 51 in CurrentUser\CA which includes a bunch of LE related stuff. It even has expired stuff like the old Let's Encrypt Authority X3 intermediate. It also varies by browser. IE, Edge, and Chrome (for the time being) use the Windows stores, but Firefox uses its own.

I believe the lazy loading is dependent on generally having outbound Internet access as well. Not sure what happens on systems that can't reach the greater Internet or have strict proxies in place.