Potential problem with R3 intermediates on Windows servers

TL;DR

My theory is that if a Windows server used the old R3 > DST Root X3 in the past it will fail to use the newer R3 > ISRG Root X1, except where the webserver is apache/nginx etc.

1 Like

As an aside does anyone know why http://r3.i.lencr.org/ (the from the certificate Authority Information Access) needs to return the expiring R3 instead of the new one?

1 Like

It doesn't? Unless I read SSLLabs incorrectly, it uses the R3 intermediate cert signed by ISRG Root X1 and is valid until Mon, 15 Sep 2025 16:00:00 UTC?

That would be https://r3.i.lencr.org/ :slight_smile:

plain old http://r3.i.lencr.org/ is the information supplied in the leaf certificate so systems can download the intermediate that issued it (to my knowledge). Visiting that downloads the old R3 cert.

1 Like

similar issue here.

I even removed the old R3 from Identrust from the "computer account"'s cert store and the IIS panel when double clicking the cert shows the trust path to ISRG.

however when looking at it with chrome it shows to the IdenTrust root.

1 Like

I'm getting some expert help on this so I'll revert back here once I know either way.

It sounds like when the R3 really expires the other path will become the active one, so I'll do some testing with a server from the future :slight_smile:

1 Like

Ah, I misunderstood, sorry.

2 Likes

I think the FAKETIME only simulates an adjustment on the clients clock.
What you need to test is on the server; To see what it will do in October.

2 Likes

Thanks, yeah that's what I'm thinking. It's a little bit 'fingers crossed' though :slight_smile:

2 Likes

Update: No luck. Setting the (windows) client and (windows) server forward to the same date in October doesn't seem to make a difference, plus various other things in the OS break due to other failed https connections caused by having the wrong date/time.

I was hoping that the the server would start to favour the newer R3 and for a brief period it did seem to work, then it broke again. I'm hopefully just testing it wrong.

2 Likes

Thanks for trying :slight_smile:
Getting Y2K flashbacks!

2 Likes

Yeah, that is weird. Can we have somebody who made the decision explain it or point to a document that explains it? It would be a shame if servers that don't serve any chain, plus clients that attempt AIA chasing = failure on 29 September 2021.

@Osiris do we need to ask the team about this? I don't like to prod people who've got better things to do with their time, but if it's a mistake it'd be better to fix it before it blows up.

3 Likes

This was already asked in the DST Root CA help thread, but we're still waiting for an official staff response.

I would guess that switchting the R3 AIA cert is a change LE is planning to make, but hasn't done this yet.

4 Likes

Well, I'm not sure why it's still sending the older R3 cert to be honest. Since 4 May 2021 this certificate isn't even in active use, only if someone would build their own chain manually. I guess it's worth asking @lestaff about it.

2 Likes

Update:

TL;DR : Wait and see, if problem occurs try turning it off and then back on again..

Problem:
- When both the old and new Let's Encrypt R3 intermediate certificates are present windows will favour the old (expiring) R3.

Resolution:

  • The new chain will only be served when the old R3 has expired and a restart may be required to force the changeover (or maybe not, you could wait and see). To force Windows Server to serve using the newer chain, ensure the newer R3 intermediate is in the machine intermediate certification authorities store and restart the machine.
  • The certificate UI on a windows machine (client or server) will show the leaf > R3 > DST Root X3 chain even if the served chain was leaf > R3 > ISRG Root X1, until you delete the old R3 intermediate from the local machine certificate store (intermediate certification authorities).

Theory
When windows encounters an R3 for the first time after a reboot, it doesn't trust it's local intermediate store (which may contain the newer R3) and instead follows the leaf certificate AIA url [I have no idea if this is actually the case or not] and uses that linked intermediate (which is currently the old R3). This means that for the most part windows will prefer to build/serve a chain with the old R3 until that expires. Removing the old R3 from the local intermediate store does not help.

Testing methodology

Server in the future:

  • Windows server 2019, Date initially set to current day.

Clients in the future:

  • Windows server 2019, Date set to october 2021.
  • Ubuntu VM, set to October 2021

Server Setup

  • Windows Server 2019 with IIS installed. TCP Port 80, 443 open and 3389 for RDP.
  • Install the old Let's Encrypt R3 > DST Root X3 intermediate (https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.der) into Local Machine "Intermediate Certification Authorities" store.
  • Setup an http website (with http hostname binding) and get certificate from Let's Encrypt using https://certifytheweb.com (any acme tool that stores to the certificate store will do). e.g. https://test02.future.projectbids.co.uk
    • New Certificate > Select IIS Site. Click Test to validate http connectivity then Request Certificate (to automatically fetch cert, store via PFX then bind to IIS (SNI)).
    • Path to original PFX before storing in certificate store is shown in Certificate > Advanced > Actions, this is useful to open the PFX and see the component intermediates etc originally used are correct.
  • The certificate will be using the chain leaf > R3 > ISRG Root X1 > DST Root X3,

Testing
Server and Client set to Current Day
Testing with openssl from WSL (ubuntu) returns cert chain leaf >R3> DST Root CA X3 :

		openssl s_client -servername test02.future.projectbids.co.uk -connect test02.future.projectbids.co.uk:443
		CONNECTED(00000003)
		depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
		verify return:1
		depth=1 C = US, O = Let's Encrypt, CN = R3
		verify return:1
		depth=0 CN = test02.future.projectbids.co.uk
		verify return:1

Server and Client set to October

Testing client from the future (both client and server VMs set to future time Oct 2021, server not restarted):

openssl s_client -servername test02.future.projectbids.co.uk -connect test02.future.projectbids.co.uk:443
	CONNECTED(00000003)
	depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
	verify error:num=10:certificate has expired
	notAfter=Sep 30 14:01:15 2021 GMT
	verify return:1
	depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
	notAfter=Sep 30 14:01:15 2021 GMT
	verify return:1
	depth=1 C = US, O = Let's Encrypt, CN = R3
	verify error:num=10:certificate has expired
	notAfter=Sep 29 19:21:40 2021 GMT
	verify return:1
	depth=1 C = US, O = Let's Encrypt, CN = R3
	notAfter=Sep 29 19:21:40 2021 GMT
	verify return:1
	depth=0 CN = test02.future.projectbids.co.uk
	notAfter=Nov  5 05:58:44 2021 GMT
	verify return:1
	---
	Certificate chain
	 0 s:CN = test02.future.projectbids.co.uk
	   i:C = US, O = Let's Encrypt, CN = R3
	 1 s:C = US, O = Let's Encrypt, CN = R3
	   i:O = Digital Signature Trust Co., CN = DST Root CA X3

After Server Restarted (and after old R3 expiry)
Testing client from the future (client vm and server vm is set to future time Oct 2021):

openssl s_client -servername test02.future.projectbids.co.uk -connect test02.future.projectbids.co.uk:443
CONNECTED(00000005)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = test02.future.projectbids.co.uk
verify return:1
---
Certificate chain
 0 s:CN = test02.future.projectbids.co.uk
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
  • Server restarted and tests repeated. ** ISRG Root X1 returned OK **

  • Removed old R3 from all intermediate stored, restarted server. Old R3 chain returned : fails

  • Removed both R3 certs from all intermediate stored, restarted server. Old R3 chain returned : fails

  • Manually re-imported the same test02.projectbids.co.uk certificate. New R3 intermediate re-appeared in store. Attempted net stop http as way to flush certs but service didn't stop. Restarted.

  • Repeated tests, ** ISRG Root X1 returned OK **

Set server time back to automatic (current day), restarted.

  • Repeated tests, Old R3 chain returned
4 Likes

Based on Certificate Path Validation in Bridge CA and Cross-Certification Environments - Microsoft Tech Community

I think Windows prefers the expiring R3 because it's notBefore date is actually newer than the non-expiring R3 (it was issued more recently, despite expiring first).

5 Likes

Even though it doesn't look like this has anything to do with how Windows Server chooses to build its chains, we have updated http://r3.i.lencr.org to return the version of the R3 certificate issued by ISRG Root X1, rather than the version issued by DST Root CA X3.

9 Likes

Thanks, I had a TODO item to remind me to follow up on this in September and now I don't need to. I am not certain whether doing this actually makes the world definitely better, but it might and NOT doing it certainly can't make the world better so thank you.

4 Likes

Thanks @aarongable! If i get a chance to re-test I might give it a go to see if it does make a difference anywhere. It seems like a good idea anyway.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.