IIS 8.5 building incorrect chain with Lets Encrypt Authority X3

Okay, I followed your procedure and did find a stale X1 certificate in de personal store for intermediate certificates. (Windows 2012r2 with IIS 8.5).

However, removing it did not produce any results: the stale X1 certificate is still served (even after a reboot) and external validation by SSLlabs and openSSL do not show a change.

An interesting problem…

jsha, thank you for your help!

I’m trying to follow your flow, but not getting it to work.

The x1 certificate that was once installed I had already removed days ago. I installed and removed it again, installed and removed the x3 one to… nothing seems to help yet.

Has anybody who has access to MS support tried contacting them?
https://support.microsoft.com/en-us/assistedsupportproducts

We do not have software assurance so support is a bit harder to get.

any of these will do:

  • Software Assurance
  • MSDN/Technet subscription
  • or other MS support contract

alternative will cost $499,- for one incident (prob. refunded if issue ms fault)

@jsha - on me also x1 cerificate was removed past week. I’ve double check, also remove x3 and install it again - no change - fail to fix the problem :frowning:

I don’t have any intermediate X1 certificate anymore on my server, removed those a while back when I tried to wipe everything from Let’s Encrypt to start again with fresh certificates.

After spending most of the day on this exact same issue I hope I will make most of you very happy as I think I have found the solution…

As with everyone else, the X1 certificate was nowhere to be found yet IIS managed to serve it in the chain.

After a lot of head banging I finally found the one place no one had looked before. The user certificate store for the Local System account. That’s right - there is such a thing and it is not the same as the computer store.

To get to it, you need to download PsTools from SysInternals and run psexec -i -s mmc.exe, go to File -> Add-Remove Snap-in, choose Certificates and My user account. Now go into Intermediate Certificate Authorities and you should find that elusive X1 certificate hiding there.

It might be enough to remove the X1 and then restart IIS, but I ended up adding X3 certificate here just to be sure (right click on the certificate list - click All Tasks -> Import and choose the X3 file).

After this you need to “touch” the bindings in IIS (for example, change the certificate and then back again or delete/add the binding) and after IIS is then restarted it will finally start to serve the correct chain.

20 Likes

@Knagis…you’re my hero!!!

I had to redo the bindings after deleting the X1 Cert in Local System account, but after that everything else is working fine!

1 Like

Thanks so much for this fix! Sure enough, I opened up that cert store, and there was the X1 certificate!

Removed, and all seems to be fine now! (Do you have a bitcoin address?) :smiley:

1 Like

Nope, just paypal, but I think that in this case the feeling that I found the solution that so many were looking for will be my prize. Solving the impossible has always been fun :sunglasses:

4 Likes

WOW - work without problem! Man - great job!
That tool did the magic!

This is an awesome find. :tada:

YES! I can confirm this works.

I used the glorious psexec -i -s mmc.exe, removed the X1 certificate from the intermediate store, did an IISreset, removed the existing bindings and reapplied them. And now it works!

@Knagis you are a great man, thank you!

There is an important catch!

You will have to renew all certs hosted in the IIS from X1 to X3 before applying this solution.

If you don’t do that, your old X1 certs will stop working correctly and their SSL Labs grade will be capped to B.

That’s exactly what happened in ours webservers, which hosts hundreds of LE domain certs.

Am I right?

2 Likes

Yesssssss! Super Knagis!

Problem solved

good way. that is right… :smiley:

@Knagis, thank you for the clear directions! I’m glad folks are finding this fixes their problems. @actyler1001, does this work for you?

@_xentia, @Rouzax, @SanderAtSnakeware, is there something different between @Knagis’ instructions and mine that fixed your problem? I think what @Knagis describes is very similar to what I said, and I want to make sure I capture the distinction so I can explain it well to others. Is the difference that PsUtil allows you to run MMC on a remote machine, and previously you were running MMC on a local machine?

Thanks,
Jacob

The difference is using psexec.exe to launch the management console while impersonating the system account. So that when you choose “My user account” it actually loads not your but the store of the Local System account.

2 Likes

@jsha - your instructions was overlap @Knagis solution at almost 99%. Main difference is that tool - psexec - which allow to run mmc for local user (!). I’m in deep research now - what can be a simpliest way to reach that store - without additional tools - to reach that cert store. Still can’t believe that in Windows server i’ll need a add tool to comply that task

1 Like

Great job! Thank you very much for sharing! :slight_smile:
I was checking the personal cert store of all user accounts on the server and double and tripple checked cert caches but I wouldn’t come up with the idea to run certmgr with psexec…
:white_check_mark:

SysInternals is a Microsoft Toolset, is it not? :slight_smile: