Users of older Android and Windows 7 not able to access website

Dear Community,

I'm facing the same issue for my this website, https://www.onlinevidyalaya.net/. Older Android devices and Windows 7 users are not able to access the website.

To solve this issue, I followed this guideline,

  • Imported isrg-root-x1-cross-signed.der into trusted store
  • moved isrgrootx1.der (self signed certificate) to untrusted store
  • Reissued let's encrypt certificate again (I'm using plesk panel)
  • rebooted the server (using windows server 2012 r2) 3-4 times.
  • manually tried to bind certificate from IIS manager also.

Still facing the issue. Can anybody please help me on this issue?

I see these website owners have resolved the issue ( opensea.io and guardiandigital.com). Can you please help me with the same? what steps do I need to follow?

SSL Server Test: opensea.io (Powered by Qualys SSL Labs) (this is not my website, they have resolved the issue)
SSL Server Test: opensea.io (Powered by Qualys SSL Labs)
showing 3 certificates under the additional certificates section.

SSL Server Test: onlinevidyalaya.net (Powered by Qualys SSL Labs) showing only 2 certificates under the additional certificates section.

@rg305 @schoen and other experts please guide.

Have a good day!

Thanks!

3 Likes

Hi @parth.patel welcome to the LE community forum :slight_smile:

The site is serving the recently expired R3 cert:

---
Certificate chain
 0 s:/CN=onlinevidyalaya.net
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---

This is quite unexpected; as it is not the preferred trust path (and hasn't been for many months).
Which ACME client are you using?
How did you get the cert into IIS?

2 Likes

Hey @parth.patel. As @rg305 mentioned, your site is still serving an old expired copy of the R3 intermediate certificate. This means you likely have a copy of it stuck somewhere on your system that needs to be removed.

It is most likely in either the Local Computer's Intermediate trust store, or the SYSTEM user's Intermediate trust store (or both). You can view the Local Computer's stores by running certlm.msc as admin. Viewing the SYSTEM user's stores is a bit more tricky and requires the PsExec utility. Once you have it you can run psexec.exe -i -s certmgr.msc as admin.

Once you've remove any errant copies. Reboot and you should be fixed. But use something like the Namecheap SSL Checker to be sure.

As a side note, I'm working on a PowerShell script that will seek out these rogue copies of the expired R3 on Windows and remove them.

5 Likes

If you want to give it a shot, I've got a preliminary version finished. You can download it here:

Download and Save it as Find-ExpiredR3.ps1, open PowerShell as administrator, cd to the folder where the file is saved, and run it like this:

.\Find-ExpiredR3.ps1

If it shows any found copies, you can run it again with the -Remediate option to have it delete them for you.

.\Find-ExpiredR3.ps1 -Remediate
4 Likes

I don't know enough PowerShell to read this, but it's really awesome you made this tool. Thanks!

3 Likes

I'm debating making a more comprehensive script that would generally fix/cleanup the Windows cert stores as they relate to Let's Encrypt, but there are a lot of variations of what could be considered "right" depending on whether you're a client, server, or both.

2 Likes

Hello @rmbolger, @rg305

Before it was showing like this without doing any changes on Server. PFA

Currently status is,

  • ISRG Root X1 (self-signed) (which will expire in 2035) is moved to Untrusted Store.
  • ISRG Root X1 (cross signed) (which will expire in 2024) is stored in trusted root store and Intermediate chain store.
  • DST Root X3 (expired on 30th September, 2021) is still in Trusted root store so, do you want me to delete that?

I hope you have understood my concern, let's encrypt certificate should work on older Android devices and windows 7 without asking clients to make changes to their side. Changes should be done from Server only.

I gave two examples of someone's website,
opensea.io & guardiandigital.com, it's working completely fine with older Android versions.

I want to achieve same like above websites.

Thank you!

Hi @rg305

I'm using Let's encrypt.

Using plesk panel, I can install, reissue let's encrypt certificate.

I've mentioned you in my other reply, please check that.

Thanks!

1 Like

If you are going to server old Android devices then you will need to serve that trusted path.
The one being served it NOT that one.
[there are three trust paths - that one is the worst of them]

The cross-signed ISRG Root X1 does not need to be in Trusted Roots, only Intermediate. The DST Root should remain in Trusted Roots. It is also important to distinguish whether you made these changes in the Local Computer stores or the SYSTEM user's stores. Untrusting the self-signed ISRG Root X1 will do less damage if it is only untrusted by the SYSTEM user and not the whole computer. Here's an article I wrote on the specifics:

But you also still need to look for any remove any existing copies of the expired R3 certificate that likely still exist and are causing problems getting the new chains (either of them) to be served. You can either do that manually or using the script I linked earlier in the thread.

3 Likes

The need for system administrators to intervene to force the default chain in windows environment is frustrating. The default chain served by the ACME client via API really makes no difference if you are hosting a site with IIS, since it continues to serve only the alternative chain.
Until hours ago my Server with Windows Server was serving the long chain without requiring any intervention. But after he automatically obtained the ISRG Root X1 certificate via Windows Update, it went on to serve only the short chain.

1 Like

When moving the self-signed certificate "ISRG Root X1" to "Untrusted Certificates" will not cause the ACME client to be unable to establish SSL connection to the Let's Encrypt API during the renewal or issuance of a new certificate?
The Let's Encrypt API uses the self-signed ISRG Root X1 certified short chain.

image

1 Like

It is frustrating, especially when it requires this sort of hacky solution. I feel you. But it's the double-edged sword of using an OS that tries to help you out with low level stuff like this. You've never even had to think about certificate chains until now because Windows just handled it. But now that you do, you realize you don't have the level of control you'd like. Arguably, you've still been better off than the Linux folks who have unknowingly been serving no chain for years causing issues with who knows how many clients.

You are correct that the API currently serves the short chain (which makes sense because there aren't a lot of ACME clients running on old versions of Android). But in my very limited testing on a single instance of Windows Server 2019, untrusting the self-signed ISRG Root X1 cert on the SYSTEM user does not fully break LE validation for the SYSTEM user. It only breaks validation against sites using the long chain (such as the main Let's Encrypt website). Here's a test I ran on my server that is currently serving the long chain using my method following a reboot:

I've launched PowerShell as the SYSTEM user and attempt to connect to both the Prod API endpoint and the main Let's Encrypt website. The API endpoint works, while the website throws an SSL error.

And like I mentioned in the article I wrote, it doesn't break any validation for other users on the same system because they all see the self-signed ISRG Root X1 as trusted. So even if on some other combination of OS/patch level it does break LE validation entirely for the SYSTEM user, you can just change your renewal task to run as a different user.

3 Likes

Excellent work. Thank you so much for the clarifications.

1 Like

Thanks a ton! @rmbolger

It helped a lot, the issue has been resolved and working perfectly on older devices.

Let's Encrypt community is very supportive and thanks a lot to experts like you who are working hard day and night to support this community.

Everyone, please follow this link those who want to support let's encrypt certificate on older devices, suggested by @rmbolger

You can follow the below steps, I've tried to make it simple.
Note: These changes I've made on Windows Server 2012 R2.

  • Do not remove the DST Root CA X3 certificate (Expired on 30/09/2021) from Trusted Root Certification Authorities.

  • ISRG Root X1 certification self-signed and cross-signed both should not be in Trusted Root Certification Authorities.

  • ISRG Root X1 certification self-signed (Expires on 04/06/2035) should be in Untrusted Certificates.

  • ISRG Root X1 certification cross-signed (Expires on 30/09/2024) should be in Intermediate Certification Authorities.

  • Remove old R3 certificate from Intermediate Certification Authorities.

  • Add current R3 certificate (Expires on 15/09/2025) to Intermediate Certification Authorities.

  • After completing the above steps server reboot is mandatory to reflect the changes.

  • That's all! Your issue is resolved 100%

You can use Namechep SSL Checker to verify the chain being served.

We've resolved the issue for this website www.onlinevidyalaya.net
Our clients are happy after this solution. :smiley:

Have a great day! Again thanks to @rmbolger and the Let's Encrypt community.

4 Likes

Glad I was able to help. Cheers!

2 Likes

I asked my users that have this issue to test your website, and it does not work at all. They still have an issue with their certificates. They have windows 7 if it can help you.

Edit: I think that what misled you was maybe believing that opensea.io works for users with windows 7, but it's not so. All the users I communicate with still have an invalid certificate for opensea included. It bothers me deeply because in the last few years I've written a lot of scripts that rely on Let's Encrypt among others. But if you want to keep your site usable by all your previous users, i.e., like 2 days ago, you have no choice but to use another CA I think.

I notice that there is an error on the site: ACME CA Comparison - Posh-ACME zeroSSL does not offer free wildcards (I think). In fact LE was the only one to offer them and that's why it's a huge problem for me.

A post was split to a new topic: Ubuntu Android problem

This solution could be useful for those who can't search for their own certificates on their own servers, well I can't really imagine how it's possible to not know how to use a search tool by file extension but let's move on.

On the other hand this solution is simply impossible for the client side. Only a very small minority of people will find this script, and even fewer will be able to use it.

Not to mention that cloudflare still delivers the expired DST Root CA X3 for at least 25% of their users (for the free version) if I understand correctly.

I still have a lot of users with old devices, one could imagine that the most dedicated would click on the scary red message to use the website. But as soon as your application is a bit more complex and uses subdomains with XHR requests or uses WebRTC or WebSocket, it completely destroys your application for these users.

We are talking about 100m to 200m users in the world who still have windows 7? It touches a public obviously less rich by the way. It's pretty sad that Let's Encrypt doesn't care about them anymore (I hope I'm wrong).

I still have hope to find a workaround server side (only), but after a day of intense research I personally had to resign myself to use a paid CA to generate my wildcard certificates.

Thanks for your reactivity though.

1 Like

Cloudflare has their own CA from digicert (but probably in name only, probablly still managed by digicert people), not impacted by this. you actually need business+ tier to use your on certificate