Which version of Chrome?
1 Like
I've already stated this in the very beginning of this thread, but to reiterate:
Your issue is that your IIS server is not sending the long chain (but the short chain). The short chain is known to break Android < 7.1.1.
IIS servers always send the short chain (because the long chain involves an expired certificate IIS tries to avoid). Exceptions are when the system does not trust ISRG Root X1, or is unaware of the expiry of DST Root CA X3.
You can try applying workarounds (e.g distrusting ISRG Root X1) in order to get it to build the long chain - I've already posted that workaround above - but that may come with side effects, such as no longer being able to connect to the Let's Encrypt API (this depends on some factors though). It is in general not recommended.
Otherwise you could consider putting a reverse proxy in front of your IIS, that is capable of sending the long chain.
It looks to me like you have installed ISRG Root X1 on the server. That won't do anything, you would have to install that root on every single Android device connecting to you. Also note that its not always possible to reliably change the trust store on Android manually (some apps/versions will only ever use the system trust store).
3 Likes
Its is Chrome browser 96.0.4664.45 on Android 7.0
1 Like
Hello Nummer378,
For a moment let us assume your argument is right that IIS servers always send the short chain then it should happen to all certificates issued by other issuers which I do not see it happening. This is happening only to Lets' Encrypt issued certificates. Why to point finger at IIS server?
Secondly, as I mentioned, the workaround (e.g distrusting ISRG Root X1) has not lasted longer and the problem is back again.
Thirdly, reverse proxy in front of your IIS is not a viable and good option.
Fourthly, installing a root on every single Android device connecting to website is not practical.
I think problem is in the certificate itself or components related to its validation. This was not happening before Sep 2021 and it does not happen on certificates issued by other issuer and this behavior is appearing only with LET'S Encrypt certificate in and after Sep 2021.
I am very much fond of Let's Encrypt certificates and hopefully, some patches or fixes on the validation system would be released or applied.
1 Like
I can browse to https://www.ashiro.ca/
Using:
-
Android 8.0.0:
Chrome 92.0.4515.115
Firefox 90.1.3 -
Android 4.4.2:
Chrome 81.0.4044.138
Opera 58.4.2878.56737
Unfortunately, I don't have an Android 7
1 Like
I know Android 7.0 might not be available to you and you won't be able to replicate the warning message. This is strange behavior. Therefore, I posted pictures of "Invalid Certificate" warning displayed on Android 7.0.
There is silence, so we presume there is basic problem in SSL certificate itself.
1 Like
That is a bad presumption.
More than 230M sites are secured with such certificates.
I doubt you have found a problem with the cert that no one else can find.
2 Likes
You have been given the answer repeatedly by different people in various ways. You have not been able to understand it. Not further silence, there is no further point to make.
3 Likes
As per Nummer378, the IIS servers always send the short chain then can anyone give me the answer to following two questions.
-
Why the certificate is validated by the browsers on client device with Android versions higher than 7.0 especially when same IIS server is feeding ssl certificate to all devices or browsers.
-
Why the certificate is validated by safari browser of all iPhone devices especially when same IIS server is feeding ssl certificate to all devices or browsers.
1 Like
Because the short chain uses the "ISRG Root X1" cert that was introduced into trust stores in 2015.
Any system that hasn't been updated since 2015 will NOT contain that root cert in their trust stores and must be provided with a longer chain [that includes a trusted root anchor].
3 Likes
Hi rg305,
Once again, the problem started in Sep 2019. Certificate was validated till then and after Sep 2019 it is failing to be valid on Android 7.0 and older operating systems. By the way, it is happening only on devices running on Android 7.0 or older versions.
If short chain that uses the "ISRG Root X1" was introduced into trust stores in 2015 then between year 2015 and Sep 2021, how come certificates was validated by Android 7.0 and older OS.
1 Like
Please go read on the actual timeline of events.
Although "ISRG Root X1" was introduced into mainstream trust stores in 2015, it wasn't the primary root used by LE to issue certs until many years later (when their DST cross-signed certs expired).
1 Like
All of you are focusing on one issue "short chain". If "short chain" and IIS are the culprit then the problem should have been evident between year 2015 and Sep 2021. Which is not the case here. I think there could be other problem which we have not discussed it so far.
1 Like
Okay, this is my final attempt trying to explain the situation. This time I will try to cover everything, starting from the basics.
In the current PKI infrastructure, trust is derived from so-called roots (or trust anchors). Those are self-signed certificates that are installed into the trust stores of clients. Clients trust (almost) all certificates signed by them. ISRG Root X1 and DST Root CA X3 are both such root certificates. There are hundreds of other root certificates, but those two are of importance for Let's Encrypt.
A major difference between the mentioned roots is their age. DST Root CA X3 is a root certificate dating to a year <= 2006 (exact timeline is not known to me). When a new root certificate is created, it needs to reach the trust store of all clients via software updates. Given that DST Root CA X3 is older than 15 years, it had lots of time to do so. (Very) old Androids therefore have this certificate in their trust store, even if they did not have a system update in over 10 years.
ISRG Root X1 however is much more recent. It was accepted into the Mozilla root store in 2016 (similar times for other root programs). That means it "only" had 5 years to propagate into client trust stores via software updates. A major problem point for updates is Android. Android only receives updates to its trust store via major OS upgrades. Specifically, ISRG Root X1 is only included in Android versions 7.1.1 and higher. For other operating systems this depends. You can have a look at the certificate compatibility page. For example, ISRG Root X1 is included in iOS >= 10, Windows >= XP SP3 (if the update mechanism worked), macOS >= 10.12.1, Firefox >= 50...
When Let's Encrypt first launched (in 2015), it was known that it would take some time before ISRG Root X1 would reach most trust stores. Hence Let's Encrypt took the decision to get what is called a cross-sign. A different CA - IdenTrust - used their root certificate DST Root CA X3 to sign Let's Encrypts intermediate certificates. This meant that every client trusting DST Root CA X3 would trust Let's Encrypts certificates, no matter if it knew about ISRG Root X1 or not. Because DST Root CA X3 is so much older, this meant that it was present in pretty much every active internet device.
Let's Encrypt continued this intermediate cross-sign practice until May 2021. In May 2021 Let's Encrypt switched to a different cross-signing method: Instead of cross-signing the intermediate, they cross-signed the root: ISRG Root X1 got a different version that was signed by DST Root CA X3. Let's Encrypt started sending out this new "long" chain on May 4, 2021.
Initially, no one (including IIS) had a problem with that chain - DST Root CA X3 had not yet expired, and the chain now included both ISRG Root X1 and DST Root CA X3. Clients that trusted ISRG Root X1 (the majority) could use that, and other (such as Android < 7.1.1) would use DST Root CA X3 for trust. Everything was still fine.
Then, on Sep 30 14:01:15 2021 GMT, DST Root CA X3 expired. This expiry date was set when the certificate was first created - roughly 15-21 years prior.
Most clients no longer trust an expired root certificate. Hence most clients now had to use ISRG Root X1 for trust - but what if their trust store did not yet include it? Well, that's the main problem. DST Root CA X3 is no longer trusted because it's expired and ISRG Root X1 hasn't reached the client via software update. That results in certificates no longer being trusted on that client.
The situation on Android is special however: Android does not care about expired root certificates - it continues to trust them. So you can theoretically fix Android compatibility by sending a chain ending in the expired root certificate, and everything will be fine on Android. Because the "long" chain engineered by Let's Encrypt also includes ISRG Root X1, most non-Android clients will use that for their verification - so everyone should be happy in theory.
IIS servers however started disliking the long chain the second DST Root CA X3 expired, because IIS cares about the expiry - it tries to be "smart" and "fix" the issue with the expired root, without realizing that sending an expired root actually helps with Android < 7.1.1 compatibility. Hence your issue.
Other certificate authorities will use different root certificates, that have different expiration dates and have been present in trust stores for longer. They will eventually have the same problem, when their roots expire - for example in 5, 10, 15 or 20 years. Every client that does not receive updates to its trust store will eventually start failing trust validation - its just that the timeline varies between CAs. For Let's Encrypt the day was Sep 30 2021.
So the TL;DR solution for you is: Use a different certificate authority that has different roots and therefore different compatibility.
4 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.