For context, Let’s Encrypt announced this morning that certs issued after July 9th will come with an intermediate cert issued by their own ISRG root, instead of the Identrust cross-signed intermediate: https://letsencrypt.org/2019/04/15/transitioning-to-isrg-root.html
The ISRG root has been added to every major trust store, so anyone running an up-to-date OS will be fine. But unfortunately, the Android ecosystem is infamous for short support cycles. Over 50% of users are running a version of Android older than Nougat, meaning they don’t have the ISRG root installed: https://developer.android.com/about/dashboards
Due to Let’s Encrypt’s enormous success, a huge portion of the web will break for these users if you start issuing certs chained to the ISRG root instead of the current DST root. Worse still, this is going to disproportionately impact lower-income users who can’t afford to upgrade.
The blog post mentions the option of replacing the ACME-provided intermediate with the cross-signed one, but realistically, most people just go with the default settings. (And because techies are more likely to have newer phones, they might not notice any problems on their own.) Plus, you might not have that option if your cert is handled by your web host, or by a SAAS provider. I’d be surprised if more than 10% of Let’s Encrypt domains stick with the Identrust root after the July cutover.
I implore you guys to reconsider the timing of this. Maybe wait a year or so to see if the Android situation gets better? Or even get a new cross-signature from a different root? (Not sure how feasible that is, I’m sure cross-signatures from highly-compatible roots aren’t cheap.)
Sorry but I don't get how the web will 'break'. Users of these old phones will still get updates for their installed apps from the Google store since it does not use LE certificates. When they will access https sites using let's encrypt, they will get warnings and the nice people on the internet will advice them to install Firefox.
I tested the Let'encrypt new root test site and it displays green on my old 5.1 banger using Firefox 66.
What is the huge new problem exactly ? Honestly I can't see it.
Not if they're using the built-in Android browser. In that case they rely on the manufacturer to update their system roots, which is never gonna happen for those old devices.
"You can visit our website if you download a new browser" would be an awful reality.
Edit: I misinterpreted your post, my first paragraph makes no sense ..
I must be dumb but I don’t see how the google browser would have problem accessing any site that does NOT use Let’s Encrypt certificates. And the Google store itself does not use LE certificates, does it ?
Edit: I replied before seeing your edit, sorry for the noise
Firefox is a workaround, but only for web browsing. There are lots of apps that use Let’s Encrypt for their APIs. Also think about apps that connect to third-party servers - things like Podcast apps and RSS readers.
Just an example, we use Mattermost (open source Slack clone) at my office, and it has an app where you enter your server’s URL. Our server uses Let’s Encrypt, so if any of our users are using Android < 7 (which I’m sure many are), the app will break for them sometime after July 9th. (Hopefully certbot adds an easy way to use the cross-signed intermediate before then, otherwise we’ll have to hack together a script to overwrite the intermediate, or just pay $10 for a commercial cert.)
Thanks for making it clearer, @bensjoberg.
Yes, I have no idea of the typical use of Android by people having very old versions like me. Since Krackattacks I’m aware that my relic is not usable for any serious matter and I am just using it like a … well, phone for mundane stuff and some bland browsing of general sites while I travel. Certainly no banking or confidential exchanges.
I know people having old phones (even older) and they are not using apps either, and people using apps but they have shining new things, well at least supported phones.
My experience is hardly a serious market study .
I agree that your post is a good warning for LE, have they thought out everything ?
Lots of older Mac systems that don’t get official support by Apple anymore (I doubt Apple will push new root certificates to those machines via some hidden system updates) will run in trouble too, f.ex. Safari under Yosemite already complains when loading up the Let’s Encrypt site for testing the new root certificate stuff:
Let’s Encrypt’s compatibility list doesn’t list this as a problematic setup. I consider “widely trusted by browsers” something different, than just running w/o error with the 2 or 3 latest releases.
This is in some cases like 50% of my website’s visitors that come via such outdated setups. IMHO Let’s Encrypt should adopt a slightly more conservative tactic in this case. Progress is all nice and dandy, but in the end going on to aggressive will result in the opposite effect (Joe Average users losing trust/ confidence in the security of sites and get confused by those things.)
Also…
what would be the proper and secure way to manually import the root certificate into Mac OS X’s system keychain? Is there a “proper steps” recommendation for this? Or just double click the randomly downloaded pem file and hope for the best?
Only the default chain served via ACME will switch to using ISRG’s root directly in July. Using the default chain is not required, the cross-signature will remain available until September 2021.
We are aware of issues with Android compatibility and will follow the situation. In the mean time if this is a concern for you please continue using the cross-signed intermediate and everything will continue working the way it does today.
Using cross signed intermediate will still be an option.I am more concerned about there will be no cross signed intermediate after Sep 2021.Maybe Android 4.x and iOS 9 devices will disappear at that time,but we still need to handle Android 5-7.0 and some other old clients,so I think I have to find a replacement in the future.
Everybody planning to implement the manual-chain workaround, don’t ignore the little detail that the current version of the cross-signed intermediate actually expires Mar 17 2021, 6 months ahead of the Sept deadline being mentioned here. Josh’s blog post indicates that they plan to obtain a renewal at some point “within the next year” that will be valid until Sept.
Anybody utilizing certbot can likely ignore this nuance, as I’m sure those guys will implement a proper generic solution. But for anybody like us implementing this workaround manually, keep this in mind before you hard-code the current intermediate cert. Once the renewal intermediate is published, you’ll need to revisit this and start using that instead.
I think the point of starting the transition early is that while there is a path to getting the current cross-sign extended into September 2021, the actual root doing the cross-sign will expire in 2021.
To get a well established cross-sign that goes out further than 2021, Let’s Encrypt will have to persuade Identrust (or another CA) to issue the necessary cross-signs from a broadly included root with a later expiration. That may range from expensive to unachievable.
Setting this in motion gives 2 years (during which it’s possible to manually use the old chain) for site admins, app developers, users, etc, to work out their mitigations for the long haul.
Which is to say these Android clients who would potentially break by changing the intermediate will definitely break in 2021 regardless of what Let's Encrypt does, right?
Just a note, you can still buy new tablets with Android 4.4 installed. (Which might allow upgrading, or maybe not.)
I also think that it’s too early to switch to the new root, especially without having a good overview what actually supports it and what not. (https://letsencrypt.org/docs/certificate-compatibility/ still shows what’s compatible with the cross-signed intermediate.)
Yes, unless extraordinary (and likely costly) efforts are taken.
Which is why it does make sense to go ahead and change out the default chain so that things that will eventually break can break now (and get noticed) while there’s still a lengthy period of time during which it’s trivial to use the alternate chain) to work out whatever long term solution is required.
(For example, in an Android app, you can have your own trust store and delegate trust decisions to that rather than the system trust store.)
Due to concerns about insufficient ISRG root propagation on Android devices we have decided to move the date on which we will start serving a chain to our own root from July 8, 2019, to July 8, 2020.
I’m so happy to hear this! Android’s version distribution has already moved a lot since I started this tread. An extra year will make a huge difference.
Thank you ISRG for listening to feedback and prioritizing end users.
In addition to running browsers without security updates being questional, wouldn’t those already start running into SSL protocol and/or cypher suite incompatibilities with webservers, so wouldn’t those browsers start “malfunctioning” and their users switching to other browsers or devices because of that to get around that even with the root switch delay?
Android as far back as 5.1 has TLS 1.2 with AEAD modes, so pretty modern and unlikely to cause errors any time soon. Chrome on Android is updated outside of updates to the OS, but the OS trust store is used to determine what TLS certs are trusted.
.
