I can confirm that a new Windows install for the same sites generates working certificates.
me too . i had removed x1 cert in windows 2012R2 IIS8.5. but it is still not normal work.
Oh man, i can’t do that. I have more than 10 educational web system running and configuration here is not so simple. Praying for some solution appear soon
Adding myself to the list. I’ve tried all the suggestions here and am still seeing this issue on my Android device in Dolphin Browser and Chrome, and in SSL labs. On desktop everything works fine. Hoping for a solution.
Just wanted to chime in and say we’re looking into this and trying to come up with a good answer. Your work on this thread is very helpful too, so please do keep trying things and let us know what you find!
I was able to reproduce this issue in an AWS instance running Windows Server 2012 R2 and IIS 8.5, and I believe I may have a fix.
Steps to reproduce:
- Configure IIS and install an old certificate issued from the X1 intermediate.
- Run on another machine (replacing the IP address with the IP address of your IIS server):
$ openssl s_client -connect 220.127.116.11:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:' 0 s:/CN=ada679.le-test.hoffman-andrews.com i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1 i:/O=Digital Signature Trust Co./CN=DST Root CA X3
Note that the issuer field of the end-entity certificate (the first one) matches the subject field of the issuer certificate (the second one).
- Remove the old certificate, and install a new one issued from the X3 intermediate.
- On another computer, run (again):
$ openssl s_client -connect 18.104.22.168:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:'` 0 s:/CN=ada679.le-test.hoffman-andrews.com i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X1 i:/O=Digital Signature Trust Co./CN=DST Root CA X3
Note that the new end-entity certificate has an Issuer field pointing to the X3 intermediate, but the certificate chain provided in the handshake offers only the X1 intermediate. This is a misconfiguration, and will cause the site to generate errors in some browsers (browsers with the X3 intermediate already cache may successfully load the site).
I tried running
certutil -urlcache * delete in PowerShell on the Windows machine. As others have reported, this was insufficient to fix the misconfiguration.
Then I added a Certificates console to the Microsoft Management Console (MMC) following these instructions. When prompted to choose an account, I chose “My user account.” I then navigated within that console to “Console Root > Certificates (Local Computer) > Intermediate Certification Authorities > Certificates” (shown below).
You can see in the screenshot that “Let’s Encrypt Authority X1” was present there. I removed it. After that, runnning the OpenSSL command once more produced correct results:
$ openssl s_client -connect 22.214.171.124:443 -showcerts </dev/null 2>/dev/null | grep ' [si]:' 0 s:/CN=ada679.le-test.hoffman-andrews.com i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3 i:/O=Digital Signature Trust Co./CN=DST Root CA X3
Note that the issuer field of the end-entity certificate now matches the subject field of the intermediate certificate provided in the chain.
Please let me know if this fixes, or fails to fix, your problem. Note that since browsers may cache intermediates themselves, the most reliable test for the fix is an openssl command like the one above, or using SSL Server Test.
Okay, I followed your procedure and did find a stale X1 certificate in de personal store for intermediate certificates. (Windows 2012r2 with IIS 8.5).
However, removing it did not produce any results: the stale X1 certificate is still served (even after a reboot) and external validation by SSLlabs and openSSL do not show a change.
An interesting problem…
jsha, thank you for your help!
I’m trying to follow your flow, but not getting it to work.
The x1 certificate that was once installed I had already removed days ago. I installed and removed it again, installed and removed the x3 one to… nothing seems to help yet.
Has anybody who has access to MS support tried contacting them?
We do not have software assurance so support is a bit harder to get.
any of these will do:
- Software Assurance
- MSDN/Technet subscription
- or other MS support contract
alternative will cost $499,- for one incident (prob. refunded if issue ms fault)
@jsha - on me also x1 cerificate was removed past week. I’ve double check, also remove x3 and install it again - no change - fail to fix the problem
I don’t have any intermediate X1 certificate anymore on my server, removed those a while back when I tried to wipe everything from Let’s Encrypt to start again with fresh certificates.
After spending most of the day on this exact same issue I hope I will make most of you very happy as I think I have found the solution…
As with everyone else, the X1 certificate was nowhere to be found yet IIS managed to serve it in the chain.
After a lot of head banging I finally found the one place no one had looked before. The user certificate store for the Local System account. That’s right - there is such a thing and it is not the same as the computer store.
To get to it, you need to download PsTools from SysInternals and run
psexec -i -s mmc.exe, go to
File -> Add-Remove Snap-in, choose
My user account. Now go into
Intermediate Certificate Authorities and you should find that elusive X1 certificate hiding there.
It might be enough to remove the X1 and then restart IIS, but I ended up adding X3 certificate here just to be sure (right click on the certificate list - click
All Tasks -> Import and choose the X3 file).
After this you need to “touch” the bindings in IIS (for example, change the certificate and then back again or delete/add the binding) and after IIS is then restarted it will finally start to serve the correct chain.
@Knagis…you’re my hero!!!
I had to redo the bindings after deleting the X1 Cert in Local System account, but after that everything else is working fine!
Thanks so much for this fix! Sure enough, I opened up that cert store, and there was the X1 certificate!
Removed, and all seems to be fine now! (Do you have a bitcoin address?)
Nope, just paypal, but I think that in this case the feeling that I found the solution that so many were looking for will be my prize. Solving the impossible has always been fun
WOW - work without problem! Man - great job!
That tool did the magic!
This is an awesome find.
YES! I can confirm this works.
I used the glorious psexec -i -s mmc.exe, removed the X1 certificate from the intermediate store, did an IISreset, removed the existing bindings and reapplied them. And now it works!
@Knagis you are a great man, thank you!
There is an important catch!
You will have to renew all certs hosted in the IIS from X1 to X3 before applying this solution.
If you don’t do that, your old X1 certs will stop working correctly and their SSL Labs grade will be capped to B.
That’s exactly what happened in ours webservers, which hosts hundreds of LE domain certs.
Am I right?
Yesssssss! Super Knagis!