IIS 8.5 building incorrect chain with Lets Encrypt Authority X3

Adding myself to the list. I’ve tried all the suggestions here and am still seeing this issue on my Android device in Dolphin Browser and Chrome, and in SSL labs. On desktop everything works fine. Hoping for a solution.

Just wanted to chime in and say we’re looking into this and trying to come up with a good answer. Your work on this thread is very helpful too, so please do keep trying things and let us know what you find!

1 Like

I was able to reproduce this issue in an AWS instance running Windows Server 2012 R2 and IIS 8.5, and I believe I may have a fix.

Steps to reproduce:

  • Configure IIS and install an old certificate issued from the X1 intermediate.
  • Run on another machine (replacing the IP address with the IP address of your IIS server):
$ openssl s_client -connect 50.18.147.177:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'
 0 s:/CN=ada679.le-test.hoffman-andrews.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Note that the issuer field of the end-entity certificate (the first one) matches the subject field of the issuer certificate (the second one).

  • Remove the old certificate, and install a new one issued from the X3 intermediate.
  • On another computer, run (again):
$ openssl s_client -connect 50.18.147.177:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'`
 0 s:/CN=ada679.le-test.hoffman-andrews.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Note that the new end-entity certificate has an Issuer field pointing to the X3 intermediate, but the certificate chain provided in the handshake offers only the X1 intermediate. This is a misconfiguration, and will cause the site to generate errors in some browsers (browsers with the X3 intermediate already cache may successfully load the site).

I tried running certutil -urlcache * delete in PowerShell on the Windows machine. As others have reported, this was insufficient to fix the misconfiguration.

Then I added a Certificates console to the Microsoft Management Console (MMC) following these instructions. When prompted to choose an account, I chose “My user account.” I then navigated within that console to “Console Root > Certificates (Local Computer) > Intermediate Certification Authorities > Certificates” (shown below).

You can see in the screenshot that “Let’s Encrypt Authority X1” was present there. I removed it. After that, runnning the OpenSSL command once more produced correct results:

$ openssl s_client -connect 50.18.147.177:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'
 0 s:/CN=ada679.le-test.hoffman-andrews.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

Note that the issuer field of the end-entity certificate now matches the subject field of the intermediate certificate provided in the chain.

Please let me know if this fixes, or fails to fix, your problem. Note that since browsers may cache intermediates themselves, the most reliable test for the fix is an openssl command like the one above, or using SSL Server Test.

1 Like

Okay, I followed your procedure and did find a stale X1 certificate in de personal store for intermediate certificates. (Windows 2012r2 with IIS 8.5).

However, removing it did not produce any results: the stale X1 certificate is still served (even after a reboot) and external validation by SSLlabs and openSSL do not show a change.

An interesting problem…

jsha, thank you for your help!

I’m trying to follow your flow, but not getting it to work.

The x1 certificate that was once installed I had already removed days ago. I installed and removed it again, installed and removed the x3 one to… nothing seems to help yet.

Has anybody who has access to MS support tried contacting them?
https://support.microsoft.com/en-us/assistedsupportproducts

We do not have software assurance so support is a bit harder to get.

any of these will do:

  • Software Assurance
  • MSDN/Technet subscription
  • or other MS support contract

alternative will cost $499,- for one incident (prob. refunded if issue ms fault)

@jsha - on me also x1 cerificate was removed past week. I’ve double check, also remove x3 and install it again - no change - fail to fix the problem :frowning:

I don’t have any intermediate X1 certificate anymore on my server, removed those a while back when I tried to wipe everything from Let’s Encrypt to start again with fresh certificates.

After spending most of the day on this exact same issue I hope I will make most of you very happy as I think I have found the solution…

As with everyone else, the X1 certificate was nowhere to be found yet IIS managed to serve it in the chain.

After a lot of head banging I finally found the one place no one had looked before. The user certificate store for the Local System account. That’s right - there is such a thing and it is not the same as the computer store.

To get to it, you need to download PsTools from SysInternals and run psexec -i -s mmc.exe, go to File -> Add-Remove Snap-in, choose Certificates and My user account. Now go into Intermediate Certificate Authorities and you should find that elusive X1 certificate hiding there.

It might be enough to remove the X1 and then restart IIS, but I ended up adding X3 certificate here just to be sure (right click on the certificate list - click All Tasks -> Import and choose the X3 file).

After this you need to “touch” the bindings in IIS (for example, change the certificate and then back again or delete/add the binding) and after IIS is then restarted it will finally start to serve the correct chain.

20 Likes

@Knagis…you’re my hero!!!

I had to redo the bindings after deleting the X1 Cert in Local System account, but after that everything else is working fine!

1 Like

Thanks so much for this fix! Sure enough, I opened up that cert store, and there was the X1 certificate!

Removed, and all seems to be fine now! (Do you have a bitcoin address?) :smiley:

1 Like

Nope, just paypal, but I think that in this case the feeling that I found the solution that so many were looking for will be my prize. Solving the impossible has always been fun :sunglasses:

4 Likes

WOW - work without problem! Man - great job!
That tool did the magic!

This is an awesome find. :tada:

YES! I can confirm this works.

I used the glorious psexec -i -s mmc.exe, removed the X1 certificate from the intermediate store, did an IISreset, removed the existing bindings and reapplied them. And now it works!

@Knagis you are a great man, thank you!

There is an important catch!

You will have to renew all certs hosted in the IIS from X1 to X3 before applying this solution.

If you don’t do that, your old X1 certs will stop working correctly and their SSL Labs grade will be capped to B.

That’s exactly what happened in ours webservers, which hosts hundreds of LE domain certs.

Am I right?

2 Likes

Yesssssss! Super Knagis!

Problem solved

good way. that is right… :smiley:

@Knagis, thank you for the clear directions! I’m glad folks are finding this fixes their problems. @actyler1001, does this work for you?

@_xentia, @Rouzax, @SanderAtSnakeware, is there something different between @Knagis’ instructions and mine that fixed your problem? I think what @Knagis describes is very similar to what I said, and I want to make sure I capture the distinction so I can explain it well to others. Is the difference that PsUtil allows you to run MMC on a remote machine, and previously you were running MMC on a local machine?

Thanks,
Jacob

The difference is using psexec.exe to launch the management console while impersonating the system account. So that when you choose “My user account” it actually loads not your but the store of the Local System account.

2 Likes