PLEASE cross-sign with another older CA before the 9/2021 deadline

I love LE, for the automation and security. But I still have some visitors to my sites (and users of my web applications!) which are using older versions of MacOS and iOS. Definitely Android, but I know that issue is "fixed" with a workaround for Android for another few years.

I am here to BEG you to please cross-sign with another older CA, to extend compatibility for older devices for another 3 - 5 years. 2016 is just too recent, especially for visitors working in less developed countries, or in modern companies that have strict rules against upgrading devices.

Thanks for your consideration,
Richard

3 Likes

Welcome to the Let's Encrypt Community, Richard :slightly_smiling_face:

Wish granted? :genie:

IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3. The new cross-sign will be somewhat novel because it extends beyond the expiration of DST Root CA X3.

2 Likes

The newer cross-sign, though, is only for devices that don't care that the root in their trust store is expired, as is apparently the case for older Android devices. I think what's being asked for here, though, is support for (If I understand the Certificate Compatibility list correctly) iOS before 10, and MacOS before 10.12.1. Perhaps those systems do care about the expiration of the roots in their default trust store? In which case, I'm not sure what actions can be taken, as if a system doesn't get security updates (including to its trust store) it's really not safe to use on the Internet. (I'd say the same thing about old Android, which apparently does have this bizarre workaround though, so I must admit I don't quite understand the dynamics here.)

3 Likes

To further clarify, this is a cross-signing of the ISRG Root X1 root certificate and not a cross-signing of the R3 intermediate certificate.

Previously, the RSA issuance chain was as follows:

  • Your certificate (leaf, included)
  • Let’s Encrypt Authority X3 (intermediate, included)
  • DST Root CA X3 (root, in trust store)

Now, the RSA issuance chain will be as follows:

  • Your certificate (leaf, included)
  • R3 (intermediate, included)
  • ISRG Root X1 (intermediate, included)
  • DST Root CA X3 (root, in trust store)

1 Like

Yes, that's what I'm asking for-- I understand about Android compatibility being extended through a work-around. What I'm asking for is that LE please cross-sign with another CA so that folks on older Linux, Mac, and iPhones won't be interrupted and see scary "This site isn't safe" messages.

Sadly, I'm having to go through and purchase traditional SSL certs (positiveSSL) just to preserve compatibility with older visitors and users of my software. I was just hoping this last-ditch plea would have any effect on the powers-that-be :confused:

2 Likes

Will these devices not work with the chain I presented above? Are they sensitive to the root expiration?

1 Like

As I understand it, that's correct. After September 2021, those devices will no longer trust certificates issued by Let's Encrypt, according to the compatibility page. For example, I had over 50 visitors from MacOS 10.12 last week. All such visitors will start receiving certificate security safety warnings if I don't switch to a different SSL cert provider before the deadline. I know it isn't much, but I don't want to miss a single potential client or user of my software.

2 Likes

That's unfortunate. :pensive:

Wish DST Root CA X3 could simply be "reissued" (with the same public key) and included in the chain. It would mean sending 4 certificates, but at least the full, unexpired chain would be present.

1 Like

Heh, yes it is. Which is why I am begging LE to just cross-sign with another CA (DST Root CA X3 expires on Sept 30, 2021) before then, and there should be no interruption.

Instead, they are relying exclusively on their own CA (ISRG Root X1), which was only just created in 2016. Meaning, devices or OSes made before 2016 won't trust it, unless they've been updated to. And sadly, that leaves out a lot of Mac users.

Even the Android users (with the workaround) need to be at version 7.1 or above by 2025. Meanwhile, there are plenty of Android phones out there running something before 7.1, especially in the developing world.

And all of this can be remedied if LE just got cross-signed by another CA that extended compatibility for, say, another 5 years. 2016 is just too soon!

2 Likes

There are at least 2 other free ACME CAs (ZeroSSL and BuyPass) you could use instead of Let's Encrypt and instead of purchasing certs. I have no idea how widely distributed each of their roots are or what devices may or may not be compatible. ZeroSSL uses a Sectigo root that doesn't expire until 2038 and BuyPass has their own root that doesn't expire until 2040. Both were issued originally in 2010 which I would think means they have decent distribution across older devices. The only caveat with BuyPass is that you can't get wildcard certs with them.

8 Likes

I don't see anything on the compatibility page indicating which devices will check the expiration of roots. Do anyone know if iOS and macOS does this?

3 Likes

That really is the question, whether they (1) only care about the public keys in the store and nothing else (in which case the "Android workaround" will work for them), (2) only care about the public keys but are fine with any dates presented by a real root certificate (in which case the workaround @griffin proposed of asking DST Root CA X3 to reissue itself with a longer date would work if they were willing to do so), or (3) actually check against the dates in their own built-in trust store (in which case the trust store needs to be updated somehow or a different CA that's already in there with a longer date needs to be used instead).

I feel like somebody must have done some testing, and it may even be linked to elsewhere on the forum, and that's what led to the Android case being the biggest issue that they focused on fixing?

With how widespread Let's Encrypt certificates are, a lot of the Internet will "break" for these older devices anyway once DST Root CA X3 expires, so presumably your site won't be alone. I know there was some talk quite some time ago about somebody might want to distribute some kind of "app" that had as its entire purpose adding ISRG Root X1 to the device's trust store to help deal with the expiration on Android devices; perhaps something like that could exist for these iOS and MacOS users that you're concerned about here too? If the users want their devices to keep working on the Internet, some sort of third-party "trust store update app" might be popularized for them?

2 Likes

My only concern with my own recommendation is that I'm not sure if there's a precedent for including a true root certificate in the issuance. The only way to verify it would be to match its public key with the public key of a trusted (but expired) root in the trust store. Is it legitimate to download the entire chain and just trust it?

1 Like

It's "legitimate" in the sense of "compliant with the current specs" as I understand them*. A server can send any number of "chain" certificates to try to assist the client in trusting their leaf. A client could use those, see that there's a longer-lived-and-not-yet-expired chain available including a public key in its trust store, and make a trust decision based on that. I don't know as any client actually does so in practice, though.

* My understanding could of course be wrong, though.

2 Likes

Thank you for this information. From the compatibility list, ZeroSSL looks like the best (ZeroSSL Compatibility List – ZeroSSL), and using their "zerossl bot" wrapper for certbot, the syntax is basically the same as certbot. See the github for it here: GitHub - zerossl/zerossl-bot: The repository for the ZeroSSL certbot wrapper

I think I'll experiment and probably end up going with ZeroSSL. I just wish I didn't have to switch from LE at all :frowning:

4 Likes

This would not change anything at all, as the clients which are affected by the soon-to-be-expired root CA are precisely the clients that don't understand how trust anchors work and will always verify against the root they have stored. (Most clients that handle chains the smart way won't be affected in the first place, unless you have a very modern TLS implementation but a very old trust store)

IdenTrust started replacing their DST Root CA X3 back in 2014, nowadays they have stopped using DST Root CA X3 and have switched to their newer roots - IdenTrust Commercial Root CA 1 and IdenTrust Public Sector Root CA 1.

The agreement Let's Encrypt/ISRG and IdenTrust have obviously doesn't include the usage of their newer root CA's, otherwise they would have long switched - instead they renewed a signature beyond the lifetime of the old root, which will certainly cause disruptions for users. I can only speculate as to why their contract did not allow a better solution, but as it is now we will have to live with it - either throw old Android overboard, or throw all of your older & non-browser (scripts, bots etc) overboard. Choose one.

The recommendation for a different CA is a good one, as it gives the best compatibility. ZeroSSL chains up to Sectigo, they use a root certificate from 2004 (in most trust stores since 2005). If one wants to stay with Let's Encrypt - which is understandable - the compatibility hit is going to happen.

Maybe this is going to help the ecosystem in the long run, as not updating system trust stores for 5+ years is really not a good practice. Maybe this will wake up some large integrators. I have a Smart TV in the home from 2015 which has a trust store that dates approximately to ~2012 - that's just horrible.

PS: While re-reading this I realized that I didn't specify that their are two distinct issues I'm talking about:

  1. There are clients with modern trust store (ISRG Root X1 included), but bad chain verification that will not work with Let's Encrypts future chain (for example GnuTLS & LibreSSL up to a version released in mid-2020, will take years until that is deployed widely)
  2. Clients that simply have a too old trust store.

The OP likely only refers to 2), yet both are going to cause disruptions to different clients.

4 Likes

@swampopus

I'm sorry but we won't be able to do this. We hear your pain and wish we could provide broader compatibility for longer, but we concluded it wasn't feasible for us. To be clear, this problem only affects browsers and apps that rely on the OS's root store. Any browser or app that ships their own root store, like Firefox, won't be affected. You might recommend your site visitors install and use of Firefox, when/if possible.

As an aside, there will be a wave of root expirations in the next few years and this will be a widespread problem that each Certificate Authority will need to solve.

12 Likes

I do not thing it is very safe using older versions of OS X or iOS unless the device is obsolete. Windows can run on older Macs which is an alternative. Old phones are another problem.

Sure, it isn't safe, but I don't have any control over what hardware/software my visitors are using. I can't flash a message that says "You must buy a new iPhone to visit my website" or "You must only use Firefox", etc.

BTW-- After trying to get ZeroSSL and Buypass actually working, I had the most luck with Buypass & Certbot, following this page's instructions: SSL | Buypass Go SSL – Technical information | Buypass CA

2 Likes

There are not many safety issues with older OSX releases. Apple aggressively obsoletes laptop/desktop hardware though - the max Operating System is defined by their software installers, not physical device constraints. Several recent OSX releases can be patched to install - and work perfectly - on older machines.

In terms of iOS devices, in recent years they have been more upgradeable (early iOS devices could not necessarily upgrade much after release). Apple released their last security updates for iOS9 and iOS10 in 2019, so any hardware going back to around 2011 can easily run something reasonably secure.

The real problem with legacy devices when it comes to SSL Certificates, is that vendors will release Security Fixes to handle Vulnerabilities -- but something like a "Trusted Root" is considered a Feature, and features improve the utility of devices and dissuade people from upgrading, so they're not released.

2 Likes