Sorry, sometimes I’m blind!
Yes, Let’s Encrypt used the same key pair for two different machines!
You know what I would like to write here now 
Please fix that immediately!
1 Like
Yes, Let’s Encrypt did change the intermediate they use for signing new certificates last week. Here’s the post announcing the change.
Previously issued certs used the “Let’s Encrypt Authority X1”, so certificate chains would need to be constructed with that. But now new certs are signed using the “Let’s Encrypt Authority X3”. This is a good thing, as we now get Windows XP support. In my testing, I’ve found Chromium on Linux to accept a certificate chain where the cert is signed by the X3 intermediary but erroneously includes the X1; whereas Firefox, curl and SSL Labs all complain that the chain is invalid. Your results may vary on other OS’es and browsers; any automated scripts creating chains need to be updated to use the X3 for any new certs.
Some of the language on https://letsencrypt.org/certificates/ is now out of date; toward the bottom it still describes LE as using the X1 intermediary, and shows this in the diagram. Someone at LE should fix this!
1 Like
Thanks for pointing out the bugs in https://letsencrypt.org/certificates/. Will fix them.
One thing to keep in mind regarding your test results: Most likely your Chromium instance on Linux had a cached copy of the X3 intermediate, which is why it validated a chain even when that chain didn’t include the right intermediate. The cached intermediates are not stored as part of your Chrome profile. I believe moving aside the “~/.pki” directory should have the effect of clearing your intermediate cache.
Hello… I am quite new to this and i seem to be having a similar problem… read through different articles but none seems to apply to my case… I am running on Azure using the Azure Let’s encrypt extension… However i keep getting certificate not trusted errors on mobile browsers especially… Here is my ssl labs result Thank you
Are you running IIS? Try to make sure all your certificates are new certificates and then remove the X1 intermediate from your trust store. Seems like IIS is confused, because both intermediates use the same key and thus have the same key identifier, which IIS seems to use to serve the intermediate. If there are two certificates matching that identifier, it just takes the first one.
Unfortunately that is easier said than done on Azure. - I’m the author of the site extension, and MS is trying to help with a way forward, but it is gonna take a little while (most likely a few weeks). So if your site is a production site where it is a must to have a fully working SSL you will have to go buy a certificate. I recommend that you follow along in this issue in the site extension github repo. https://github.com/sjkp/letsencrypt-siteextension/issues/42
Hi everyone… I am experiencing this same problem on a Windows Server 2012 R2 system (on premise) I went to renew my Let’s Encrypt certificate and found after installing that there were issues with the chain. Not trusted in Firefox or on an iPhone. Chrome and IE seem ok. SSL Labs reports bad certificate chain… I renewed/re-issued the cert about three times and attempted to install the x3 intermediate certificate, but still no joy… If I revert back to the original x1 Let’s Encrypt Cert, all is well… But it expires in a couple of weeks… 
I tried deleting all instances of the X1 intermediate from the store to force IIS to serve the correct one, but it didn’t seem to make any difference in my case…
Ps… Why did we care about supporting xp again?
Take a look at this ServerFault answer, which seems to cover the same issue.
This IIS behaviour certainly makes things a bit more complicated. IIRC the key was not rotated to avoid having to wait for the next key signing ceremony and to avoid breaking things for HPKP users who pinned to the intermediate. This might have to be revisited for future intermediate changes. Hopefully, there won’t be much of a reason to rotate the intermediates anytime soon. 
So I read through this post and just wanted to clarify… It’s telling me to delete ALL of the intermediate certifications from my web server’s store? Not just the old X1 cert, but everything?
Regards,
Adam Tyler
I don’t think removing intermediates that are not related to Let’s Encrypt will do anything useful. It might just be phrased that way so you avoid missing some other related intermediate.
I’d try just removing X1 and proceed with the other steps. If that doesn’t work, remove X1 and X3 and see if that does it. Don’t forget to re-add X3, though, and make sure you don’t have any older certificates for other domains that might still rely on X1!
Thanks… I just followed these steps, but I am getting the same result unfortunately… Chain is broke.
I tried using this site to check the chain:
https://www.sslshopper.com/ssl-checker.html
From the scan result it still appears like the x1 intermediate cert is getting served. But it’s been deleted and the certutil cache clear command was run… WTF IIS?
I have exactly the same problem. My setup: Windows Server 2012 R2 + IIS 8.5
What I have tried so far (in different variations):
- removed all bindings in IIS
- export certificate (not including chain)
- remove all certificates from store (including LE Authority X1, LE Authority X3 intermediates and DST Root CA X3)
- clear cache (certutil -urlcache * delete)
- reboot
- import DST Root CA X3 + LE Authority X3
- import my certificate (alternatively request a new one from LE)
- Configure binding in IIS to use my certificate
Result is always the same:
- Certification path in certstore shows correct path: myCert -> LE Authority X3 -> DST Root CA X3
- LE Authority X1 NOT in certificate store (cannot find it anywhere on the machine)
- ssllabs.com report invalid served chain: myCert -> LE Authority X1 -> DST Root CA X3
- When I open the site in IE on the server itself I can see the chain: myCert -> Le Authority X1 -> DST Root CA C3 as well (note: When I show the chain on the same machine via certificate store I get the correct chain)
My conclusion so far:
IIS has a “secret” cache where it stores the “wrong” LE Authority X1 certificate and serves it to the client. I have no ideas left how to get rid of this? Any suggestions? Or has anyone even solved this problem???
Dude, get out of my head! My concluding thought exactly. To confirm I am running Windows 2012 R2 as well.
Oh great goddess of IIS 8.5, please instruct us in the secret art of clearing your intermediate certificate cache. We the community swear to use the knowledge for good, not evil…
I just threw up a quick and dirty technet forum post about it: https://social.technet.microsoft.com/Forums/en-US/9cd5d6c8-25e5-494d-acf2-d94230e68a9f/iis-85-on-windows-2012-r2-returns-the-incorrect-intermediate-certificate?forum=winserver8gen
And another post on the IIS forum:
https://forums.iis.net/p/1230532/2117835.aspx?p=True&t=635953125359182511
I just renewed a cert using the X3 intermediate on a Linux/Apache server and it worked just fine… Guess that’s why they call the Linux web server LAMP and the Windows web server WIMP… lol
LAMP: Linux, Apache, MySQL, PHP
WIMP: Windows, IIS, Microsoft SQL, Powershell
For those of you tracking the IIS 8.5 specific issue we've discussed, there has been some activity on the Technet forum. no solution yet, but an interesting discussion around searching the local certificate stores using PowerShell in an effort to find this hidden X1 cert that keeps getting served...
https://social.technet.microsoft.com/Forums/en-US/9cd5d6c8-25e5-494d-acf2-d94230e68a9f/iis-85-on-windows-2012-r2-returns-the-incorrect-intermediate-certificate?forum=winserver8gen&prof=required
~~~~~~~~~~~~~~~~Update 4/6/2016
It seems I am limited to only 3 posts for one thread so I am posting my reply to "T917820981726". His post from 4/6/16 11:20 AM PST.
So I had this same experience on 2012 R2, however restarting the server and the IIS service didn't seem to make any difference. I am going to try again though to be sure...This thread seems to have gotten a little off the main topic, but just wanted to chime in and saying I’m also experiencing the exact same issues with Windows Server 2012 R2 and IIS 8.5
I’ve tried just about everything, but Chrome and OpenSSL still show X1 as the intermediate cert, not X3. The only thing I’ve seen change so far is that Firefox now, for whatever reason, is showing the correct cert chain (with X3), even though originally it did not.
Mainly trying to figure this out as CloudFlare strict SSL is not working because it also believes the cert is invalid (due to the intermediate cert pointing at X1 still).
Same here, IIS8.5 and Windows2012r2 having persistent X1 certificate issues. The X1 intermediate is no longer to be found in the certificate store and if I check the certificate in IIS it shows the correct certification path. However, IE on the local machine shows the X1 intermediate.
I use Firefox locally and it shows the correct X3 intermediate, but Chrome and IE don’t. I tried the common steps listed above but the X1 certificate seems to be pretty persistent…
Hi,
note that the problem may also be on the Client side, especially with IE/Edge and Chrome since AFAIK they both use SChannel to verify the certificate chain. The client may have buffered the Let’s Encrypt X1 certificate, and they might first use the buffered certificates instead of the intermediate sent by the server to verify the chain, and since IIS (wich also uses SChannel) is able to build the wrong chain with the X1 instead of X3 certificate, the same case might happen on the client side with IE and Chrome even if the server sends the correct X3 intermediate (provided that the user previously visited a website on that computer which uses the old X1 intermediate).
For example, I once had a problem with an StartSSL certificate where the intermediate certificate used SHA-1 and expired after Jan 2016 (Chrome shows a warning in that case). Later, StartSSL published a new Intermediate that used SHA-256, but you could use that new intermediate with the same leaf certificate you already had. So, I removed the old SHA-1 intermediate and added the new SHA-256 intermediate on the server, and according to SSLLabs, the server sent the correct SHA-256 intermediate. However, on most PCs (where I previously visited that page) browsers like IE and Chrome still showed the SHA-1 intermediate with the warning, whereas on a freshly installed Windows VM they showed the new SHA-256 intermediate.
Note that this is only speculation, I have not yet tried to install a new Let’s Encrypt certificate with the X3 intermediate on one of my servers.
Just to put the theory that this is a client-side issue to rest, I went ahead and created a new VM on a completely different network, that has never accessed my site.
Ran this OpenSSL check there -
openssl s_client -connect {server.com}:443 -servername {domain.server.com}
It still displayed X1 for the intermediate certificate. So, definitely appears to be a server-side issue! (at least in my case)
How did you generate the SSL cert and intermediate cert and installation ? using the official client ? or a different client ?
The solution would be to check which intermediate cert is on the server, and if incorrect then download and install the correct intermediate.
How / why it has the wrong one ( and from what you say it certainly seems to be the wrong one in your case ) I’m not sure without more info about how you generated it.
1 Like