Potential problem with R3 intermediates on Windows servers

Client dev
1

Hi Folks,

So I'll start by saying I hope I'm wrong. Please disprove whatever I say next :slight_smile: - I'll jot down my findings here anyway.

I believe there is a potential problem with the way Windows (server and desktop) performs chain building when using certificates from its local machine certificate store, which will result in certain services presenting expired chains after the R3 expiry on Sept 30th.

Specifically I believe Windows has a tendency to choose the soon-to-be expiring R3 (chained to DST Root X3) in preference to the R3 > ISRG Root X1 > DST Root X3 chain. This is regardless of your original certificate (I am definitely storing a PFX with leaf > R3 > ISRG Root X1) and I believe it happens because windows looks for "issuer=C = US, O = Let's Encrypt, CN = R3" and uses the first one it finds in the SYSTEM users certificates store (Intermediate Certification Authorities).

The end result is that services using the local machine certificate store server an expiring chain (IIS, MS Exchange, Remote Desktop services etc), rather than those using raw cert files (as .pem, .key etc). Deleting the expiring R3 intermediate from this store and leaving the newer R3 > ISRG Root X1 intermediate fixes the problem. I don't yet know if there is any process which will cause windows to automatically repopulate with the wrong R3. Generally I think the intermediates get populated upon importing a certificate that uses them, but windows does have some auto population for some events.

For a simulated client connection in the future I'm using the following on ubuntu:

Passing (presents correct chain):
faketime '2021-10-01' openssl s_client -servername beta01.projectbids.co.uk -connect beta01.projectbids.co.uk:443

Failing (presents wrong chain):
faketime '2021-10-01' openssl s_client -servername api-01.openchargemap.io -connect api-01.openchargemap.io:443

Access to view the SYSTEM users certificate store is via psexec -i -s cmd.exe and running certmgr.

I'm far from being an expert on certificate chains, or what windows is thinking. We can probably pull someone from Microsoft onto the conversation if it proves to be a big issue. If on the other hand it's easily disproven as just my mistake, then it's nothing to worry about. I thought I'd share this here in case it becomes apparent that it's a big problem.

If you have a windows based service that correctly serves it's TLS using the machine certificate store (not nginx/apache etc) and SSL Server Test (Powered by Qualys SSL Labs) or openssl reports the R3 > ISRG Root X1 chain without any extra effort then I'd like to know if the old R3 intermediate is present or not to prove if this only affects older/some installations.

I believe this also affects the client in some ways but it may only be relevant to the windows certmgr UI (which often shows R3 > DST Root X3, when other tools report R3 > ISRG Root X1). I've not yet tested from windows with the date set in the future.

1 Like
2

TL;DR

My theory is that if a Windows server used the old R3 > DST Root X3 in the past it will fail to use the newer R3 > ISRG Root X1, except where the webserver is apache/nginx etc.

1 Like
3

As an aside does anyone know why http://r3.i.lencr.org/ (the from the certificate Authority Information Access) needs to return the expiring R3 instead of the new one?

1 Like
4

It doesn't? Unless I read SSLLabs incorrectly, it uses the R3 intermediate cert signed by ISRG Root X1 and is valid until Mon, 15 Sep 2025 16:00:00 UTC?

5

That would be https://r3.i.lencr.org/ :slight_smile:

plain old http://r3.i.lencr.org/ is the information supplied in the leaf certificate so systems can download the intermediate that issued it (to my knowledge). Visiting that downloads the old R3 cert.

1 Like
6

similar issue here.

I even removed the old R3 from Identrust from the "computer account"'s cert store and the IIS panel when double clicking the cert shows the trust path to ISRG.

however when looking at it with chrome it shows to the IdenTrust root.

1 Like
7

I'm getting some expert help on this so I'll revert back here once I know either way.

It sounds like when the R3 really expires the other path will become the active one, so I'll do some testing with a server from the future :slight_smile:

1 Like
8

Ah, I misunderstood, sorry.

2 Likes
9

I think the FAKETIME only simulates an adjustment on the clients clock.
What you need to test is on the server; To see what it will do in October.

2 Likes
10

Thanks, yeah that's what I'm thinking. It's a little bit 'fingers crossed' though :slight_smile:

2 Likes
11

Update: No luck. Setting the (windows) client and (windows) server forward to the same date in October doesn't seem to make a difference, plus various other things in the OS break due to other failed https connections caused by having the wrong date/time.

I was hoping that the the server would start to favour the newer R3 and for a brief period it did seem to work, then it broke again. I'm hopefully just testing it wrong.

2 Likes
12

Thanks for trying :slight_smile:
Getting Y2K flashbacks!

2 Likes
13

Yeah, that is weird. Can we have somebody who made the decision explain it or point to a document that explains it? It would be a shame if servers that don't serve any chain, plus clients that attempt AIA chasing = failure on 29 September 2021.

@Osiris do we need to ask the team about this? I don't like to prod people who've got better things to do with their time, but if it's a mistake it'd be better to fix it before it blows up.

3 Likes
14

This was already asked in the DST Root CA help thread, but we're still waiting for an official staff response.

I would guess that switchting the R3 AIA cert is a change LE is planning to make, but hasn't done this yet.

4 Likes
15

Well, I'm not sure why it's still sending the older R3 cert to be honest. Since 4 May 2021 this certificate isn't even in active use, only if someone would build their own chain manually. I guess it's worth asking @lestaff about it.

2 Likes
16

Update:

TL;DR : Wait and see, if problem occurs try turning it off and then back on again..

Problem:
- When both the old and new Let's Encrypt R3 intermediate certificates are present windows will favour the old (expiring) R3.

Resolution:

  • The new chain will only be served when the old R3 has expired and a restart may be required to force the changeover (or maybe not, you could wait and see). To force Windows Server to serve using the newer chain, ensure the newer R3 intermediate is in the machine intermediate certification authorities store and restart the machine.
  • The certificate UI on a windows machine (client or server) will show the leaf > R3 > DST Root X3 chain even if the served chain was leaf > R3 > ISRG Root X1, until you delete the old R3 intermediate from the local machine certificate store (intermediate certification authorities).

Theory
When windows encounters an R3 for the first time after a reboot, it doesn't trust it's local intermediate store (which may contain the newer R3) and instead follows the leaf certificate AIA url [I have no idea if this is actually the case or not] and uses that linked intermediate (which is currently the old R3). This means that for the most part windows will prefer to build/serve a chain with the old R3 until that expires. Removing the old R3 from the local intermediate store does not help.

Testing methodology

Server in the future:

  • Windows server 2019, Date initially set to current day.

Clients in the future:

  • Windows server 2019, Date set to october 2021.
  • Ubuntu VM, set to October 2021

Server Setup

  • Windows Server 2019 with IIS installed. TCP Port 80, 443 open and 3389 for RDP.
  • Install the old Let's Encrypt R3 > DST Root X3 intermediate (https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.der) into Local Machine "Intermediate Certification Authorities" store.
  • Setup an http website (with http hostname binding) and get certificate from Let's Encrypt using https://certifytheweb.com (any acme tool that stores to the certificate store will do). e.g. https://test02.future.projectbids.co.uk
    • New Certificate > Select IIS Site. Click Test to validate http connectivity then Request Certificate (to automatically fetch cert, store via PFX then bind to IIS (SNI)).
    • Path to original PFX before storing in certificate store is shown in Certificate > Advanced > Actions, this is useful to open the PFX and see the component intermediates etc originally used are correct.
  • The certificate will be using the chain leaf > R3 > ISRG Root X1 > DST Root X3,

Testing
Server and Client set to Current Day
Testing with openssl from WSL (ubuntu) returns cert chain leaf >R3> DST Root CA X3 :

		openssl s_client -servername test02.future.projectbids.co.uk -connect test02.future.projectbids.co.uk:443
		CONNECTED(00000003)
		depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
		verify return:1
		depth=1 C = US, O = Let's Encrypt, CN = R3
		verify return:1
		depth=0 CN = test02.future.projectbids.co.uk
		verify return:1

Server and Client set to October

Testing client from the future (both client and server VMs set to future time Oct 2021, server not restarted):

openssl s_client -servername test02.future.projectbids.co.uk -connect test02.future.projectbids.co.uk:443
	CONNECTED(00000003)
	depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
	verify error:num=10:certificate has expired
	notAfter=Sep 30 14:01:15 2021 GMT
	verify return:1
	depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
	notAfter=Sep 30 14:01:15 2021 GMT
	verify return:1
	depth=1 C = US, O = Let's Encrypt, CN = R3
	verify error:num=10:certificate has expired
	notAfter=Sep 29 19:21:40 2021 GMT
	verify return:1
	depth=1 C = US, O = Let's Encrypt, CN = R3
	notAfter=Sep 29 19:21:40 2021 GMT
	verify return:1
	depth=0 CN = test02.future.projectbids.co.uk
	notAfter=Nov  5 05:58:44 2021 GMT
	verify return:1
	---
	Certificate chain
	 0 s:CN = test02.future.projectbids.co.uk
	   i:C = US, O = Let's Encrypt, CN = R3
	 1 s:C = US, O = Let's Encrypt, CN = R3
	   i:O = Digital Signature Trust Co., CN = DST Root CA X3

After Server Restarted (and after old R3 expiry)
Testing client from the future (client vm and server vm is set to future time Oct 2021):

openssl s_client -servername test02.future.projectbids.co.uk -connect test02.future.projectbids.co.uk:443
CONNECTED(00000005)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = test02.future.projectbids.co.uk
verify return:1
---
Certificate chain
 0 s:CN = test02.future.projectbids.co.uk
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

  • Server restarted and tests repeated. ** ISRG Root X1 returned OK **

  • Removed old R3 from all intermediate stored, restarted server. Old R3 chain returned : fails

  • Removed both R3 certs from all intermediate stored, restarted server. Old R3 chain returned : fails

  • Manually re-imported the same test02.projectbids.co.uk certificate. New R3 intermediate re-appeared in store. Attempted net stop http as way to flush certs but service didn't stop. Restarted.

  • Repeated tests, ** ISRG Root X1 returned OK **

Set server time back to automatic (current day), restarted.

  • Repeated tests, Old R3 chain returned
4 Likes
17

Based on Certificate Path Validation in Bridge CA and Cross-Certification Environments - Microsoft Tech Community

I think Windows prefers the expiring R3 because it's notBefore date is actually newer than the non-expiring R3 (it was issued more recently, despite expiring first).

5 Likes
18

Even though it doesn't look like this has anything to do with how Windows Server chooses to build its chains, we have updated http://r3.i.lencr.org to return the version of the R3 certificate issued by ISRG Root X1, rather than the version issued by DST Root CA X3.

9 Likes
19

Thanks, I had a TODO item to remind me to follow up on this in September and now I don't need to. I am not certain whether doing this actually makes the world definitely better, but it might and NOT doing it certainly can't make the world better so thank you.

4 Likes
20

Thanks @aarongable! If i get a chance to re-test I might give it a go to see if it does make a difference anywhere. It seems like a good idea anyway.

2 Likes