IIS 8.5 building incorrect chain with Lets Encrypt Authority X3

_xentia · April 11, 2016, 9:06am

@Rouzax, this was the first thing i do

Excluding logFiles - i check it there manually - there is no such file in my server. Creepy…

Rouzax · April 11, 2016, 10:16am

Are you able to find that other cer file?

_xentia · April 11, 2016, 10:40am

Strange, but it’s a fact - other cer file for X3
DAC9024F54D8F6DF94935FB1732638CA6AD77C13.cer
was not found too. So this lead me to more confusion now

Rouzax · April 11, 2016, 11:10am

I can confirm that a new Windows install for the same sites generates working certificates.

yon · April 11, 2016, 11:24am

me too . i had removed x1 cert in windows 2012R2 IIS8.5. but it is still not normal work.

enter link description here

_xentia · April 11, 2016, 11:42am

Oh man, i can’t do that. I have more than 10 educational web system running and configuration here is not so simple. Praying for some solution appear soon

forcedfx · April 12, 2016, 2:54pm

Adding myself to the list. I’ve tried all the suggestions here and am still seeing this issue on my Android device in Dolphin Browser and Chrome, and in SSL labs. On desktop everything works fine. Hoping for a solution.

jsha · April 12, 2016, 5:47pm

Just wanted to chime in and say we’re looking into this and trying to come up with a good answer. Your work on this thread is very helpful too, so please do keep trying things and let us know what you find!

jsha · April 13, 2016, 7:36am

I was able to reproduce this issue in an AWS instance running Windows Server 2012 R2 and IIS 8.5, and I believe I may have a fix.

Steps to reproduce:

Configure IIS and install an old certificate issued from the X1 intermediate.
Run on another machine (replacing the IP address with the IP address of your IIS server):

$ openssl s_client -connect 50.18.147.177:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'
 0 s:/CN=ada679.le-test.hoffman-andrews.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Note that the issuer field of the end-entity certificate (the first one) matches the subject field of the issuer certificate (the second one).

Remove the old certificate, and install a new one issued from the X3 intermediate.
On another computer, run (again):

$ openssl s_client -connect 50.18.147.177:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'`
 0 s:/CN=ada679.le-test.hoffman-andrews.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Note that the new end-entity certificate has an Issuer field pointing to the X3 intermediate, but the certificate chain provided in the handshake offers only the X1 intermediate. This is a misconfiguration, and will cause the site to generate errors in some browsers (browsers with the X3 intermediate already cache may successfully load the site).

I tried running certutil -urlcache * delete in PowerShell on the Windows machine. As others have reported, this was insufficient to fix the misconfiguration.

Then I added a Certificates console to the Microsoft Management Console (MMC) following these instructions. When prompted to choose an account, I chose “My user account.” I then navigated within that console to “Console Root > Certificates (Local Computer) > Intermediate Certification Authorities > Certificates” (shown below).

You can see in the screenshot that “Let’s Encrypt Authority X1” was present there. I removed it. After that, runnning the OpenSSL command once more produced correct results:

$ openssl s_client -connect 50.18.147.177:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'
 0 s:/CN=ada679.le-test.hoffman-andrews.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Note that the issuer field of the end-entity certificate now matches the subject field of the intermediate certificate provided in the chain.

Please let me know if this fixes, or fails to fix, your problem. Note that since browsers may cache intermediates themselves, the most reliable test for the fix is an openssl command like the one above, or using SSL Server Test.

SanderAtSnakeware · April 13, 2016, 8:44am

Okay, I followed your procedure and did find a stale X1 certificate in de personal store for intermediate certificates. (Windows 2012r2 with IIS 8.5).

However, removing it did not produce any results: the stale X1 certificate is still served (even after a reboot) and external validation by SSLlabs and openSSL do not show a change.

An interesting problem…

palloquin · April 13, 2016, 9:22am

jsha, thank you for your help!

I’m trying to follow your flow, but not getting it to work.

The x1 certificate that was once installed I had already removed days ago. I installed and removed it again, installed and removed the x3 one to… nothing seems to help yet.

palloquin · April 13, 2016, 9:41am

Has anybody who has access to MS support tried contacting them?
https://support.microsoft.com/en-us/assistedsupportproducts

We do not have software assurance so support is a bit harder to get.

any of these will do:

Software Assurance
MSDN/Technet subscription
or other MS support contract

alternative will cost $499,- for one incident (prob. refunded if issue ms fault)

_xentia · April 13, 2016, 11:52am

@jsha - on me also x1 cerificate was removed past week. I’ve double check, also remove x3 and install it again - no change - fail to fix the problem

Rouzax · April 13, 2016, 12:53pm

I don’t have any intermediate X1 certificate anymore on my server, removed those a while back when I tried to wipe everything from Let’s Encrypt to start again with fresh certificates.

Knagis · April 13, 2016, 2:41pm

After spending most of the day on this exact same issue I hope I will make most of you very happy as I think I have found the solution…

As with everyone else, the X1 certificate was nowhere to be found yet IIS managed to serve it in the chain.

After a lot of head banging I finally found the one place no one had looked before. The user certificate store for the Local System account. That’s right - there is such a thing and it is not the same as the computer store.

To get to it, you need to download PsTools from SysInternals and run psexec -i -s mmc.exe, go to File -> Add-Remove Snap-in, choose Certificates and My user account. Now go into Intermediate Certificate Authorities and you should find that elusive X1 certificate hiding there.

It might be enough to remove the X1 and then restart IIS, but I ended up adding X3 certificate here just to be sure (right click on the certificate list - click All Tasks -> Import and choose the X3 file).

After this you need to “touch” the bindings in IIS (for example, change the certificate and then back again or delete/add the binding) and after IIS is then restarted it will finally start to serve the correct chain.

RvdHout · April 13, 2016, 2:58pm

@Knagis…you’re my hero!!!

I had to redo the bindings after deleting the X1 Cert in Local System account, but after that everything else is working fine!

some-guy-23 · April 13, 2016, 3:00pm

Thanks so much for this fix! Sure enough, I opened up that cert store, and there was the X1 certificate!

Removed, and all seems to be fine now! (Do you have a bitcoin address?)

Knagis · April 13, 2016, 3:03pm

Nope, just paypal, but I think that in this case the feeling that I found the solution that so many were looking for will be my prize. Solving the impossible has always been fun

_xentia · April 13, 2016, 3:09pm

WOW - work without problem! Man - great job!
That tool did the magic!

Rouzax · April 13, 2016, 3:15pm

This is an awesome find.