@lestaff will LE Staging be configured to return leaf + 2 intermediate certs (as described in "What exactly is changing?" section of https://letsencrypt.org/2020/12/21/extending-android-compatibility.html) so we can test before this happens in LE Production? Thanks.
4 Likes
Yes, that's the plan. We hope to have new certificates in Staging, including a longer chain as the default, sometime in January.
6 Likes
This is an interesting idea, getting a Root-Cross-Sign valid for longer than the Root itself - I didn't knew that Android wasn't validating expiry dates on trust anchors, but it makes sense from a compatibility point of view.
I only have one concern: Does this work with other devices, other than Android?
Specifically, from what I believe to know (not exactly 100% sure though), old versions of OpenSSL (and GnuTLS) always validate up to the highest certificate in the chain, even if they should know that an intermediate is a valid trusted root. I think they don't have an "early return" like modern browsers, where validation is simply stopped as soon as a trust anchor is reached.
There are blog posts (or things discussed on various bugtrackers) that sound like trouble could happen with OpenSSL and GnuTLS as soon as DST Root CA X3 has expired. The cross-sign signature on ISRG Root X1 will still be valid, but if a client validates up to DST Root CA X3 after it has expired AND checks the expiry date (which isn't done by Android, but maybe OpenSSL/GnuTLS does?) validation will fail. This could apply EVEN if ISRG Root X1 is in the trust store, so validation without the cross-sign will work normally, but with the compatibility extension these devices will fail handshakes starting in late 2021+?
Is this concern valid, or is my assumption about the behavior of old versions of OpenSSL/GnuTLS incorrect?
4 Likes
My gut instinct is that the "short" chain with just the intermediate is what should be returned by the ACME server, with the alternate being the "long" chain with both the intermediate and the cross-signed ISRG Root X1. It seems to me that the longer chain (purely designed for compatibility with platforms that aren't getting security updates) should only get served if explicitly requested, so that handshakes everywhere are only longer if they need to be.
But I'm sure other people around who are primarily targeting Android might disagree on what the default should be, though. 
2 Likes
@petercooperjr The webserver doesn't have any information other than SNI before establishing the TLS handshake unfortunately.
3 Likes
Yep, that's a very valid concern, and one of the things we plan to address in a more technical post, as mentioned at the end of the current post. There are other posts which also tackle this topic, such as Ryan Sleevi's Path Building Showdown. Your analysis is good.
This is one of the reasons that we will still be supplying the EE <-- R3 <-- ISRG Root X1 chain as an alternate, and working with ACME client developers to ensure that their clients can be configured to select that alternate. Subscribers who value compatibility with older versions of OpenSSL and GnuTLS over compatibility with older versions of Android will be able to use the alternate chain to get the compatibility they need.
We believe that compatibility with older Android devices is the higher priority here. It matches up with our mission to support end users: not just subscribers/sites, but users of the Internet who may have little to no control over what device they use to browse. Also, maintainers of sites whose clients are using OpenSSL and GnuTLS are more likely to have the technical know-how to configure alternate certificate chains. We thought about this choice for a long time, and while we love the idea of serving the shorter more efficient chain by default, we don't currently think that's the right call.
We'll be thinking -- and talking publicly -- about this more in the future. Thanks to this cross-sign, we'll be able to provide chains that work for both of these use-cases. If our prioritization turns out to be incorrect, it will be easy for us to switch which chain we provide by default.
7 Likes
and when we have ECDSA CAs it would be DST-> ISRG x1 -> ISRG X2 -> E1 -> leaf? what's really long chain 
3 Likes
Just adding to this thought, I really think you also need to give client developers at least 2 months lead time from having the fake certs on the staging environment before doing a shift like this, as most Clients are not prepared in the slightest.
A lot of clients don't support alternate chains; most who do, do not support them well. Why? These are impossible to test on the public staging environment, and more complex to set-up or know-about if using private test environments.
If using Pebble to test or as a reference, developers must consciously enable this (IMHO, this should have been the default option). The barrier-to-entry for testing these is ridiculously high - requiring a deeper understanding of Pebble or Boulder than one should need.
4 Likes
Considering that one chooses compatibility with Older Android Devices, how do below clients behave after DST Root CA X3 expires - do they check the whole chain including expiry date or stop even if they find an intermediate cert as trusted on the platform?
- Most of the browsers - Chrome, Firefox, IE, Edge
- iOS
- .Net HTTP Libraries
- Python request libraries
If someone has already done research on this, then it may help a lot of developers.
Thanks
4 Likes
it's what happended at addtrust external ca root expired (https://crt.sh/?id=4860286) but it that case intermediate for it expired too, only thing that was effected by that was old version of openssl and GnuTLS so in effect they minesweeped the field already for us.
3 Likes
What is the best way to find out when you are pushing the changes to Staging and to Production? When you change the intermediate in Production several weeks ago, it broke a lot of my systems, so I need to plan changes at the same time that you make yours.
1 Like
Hello @csatola-od,
News are announced on API Announcements category.
For more info take a look to the pinned post About the API Announcements category.
Cheers,
sahsanu
4 Likes
Changes in the intermediate should never ever result in broken systems. Please reconsider your current methods.
4 Likes
Yep. Agreed. Maybe I'm missing something, but the issue I've found is that many android devices don't have the intermediates, so if I don't supply it manually, they whine about broken chain.
1 Like
In a well functioning web server, the intermediates should be served by the web server in addition to the end entity (EE) certificate. While some web browsers will fetch the intermediate certs on demand when only the EE is provided, some like your android devices will not and you get the errors you see. How to serve the intermediate varies depending on the web server. Some just need the chain appended to the EE cert. Others have a separate directive for configuring the chain.
4 Likes
You should indeed serve an intermediate, but you should never ever hardcode that intermediate. The Let's Encrypt servers have a method to signal the ACME client which intermediate is being used to sign the certificate and the ACME client should use that information to have the appropriate intermediate certificate available next to the certificate. And any service using the certificate should also use that intermediate which comes with the certificate.
Two things can go wrong in the above picture:
- The ACME client has a hardcoded intermediate and doesn't use the intermediate provided by the Let's Encrypt server, or
- The service using the certificate has the intermediate hardcoded, where it should use the intermediate provided by a well designed ACME client.
7 Likes
Thanks for calling me out on this! Part of my automation for LE certs was recycled from other team members. Prior to learning more about the process yesterday, (thanks to you challenging my assumptions) I didn't realize that the cert request returned the full chain for me. I'm working on updating the rest of my process to include the intermediate included in the ACME response instead of a hardcoded intermediate.
4 Likes
@csatola-od Did you build your own ACME client?
2 Likes
@Osiris No, using Posh-ACME
1 Like
Ah, that's a Windows client.. I don't have any experience with it, but I assume it fetches the intermediate properly.
2 Likes