See our announcement here. If you have any questions, feel free to ask them in this thread.
9 Likes
Has this shorter chain of trust been available for a while now? Looking at our current certificate, I do not see "DST Root CA X3" in our chain or anything that expires in September 2024.
1 Like
The "short chain" or "alternate chain" has been available since May 4th, 2021: Production Chain Changes
As of today, you need to manually request this chain via your ACME client (it's typically called the "preferred chain" feature). The details depend on your ACME client.
There isn't anything special about the certificates. You can serve every Let's Encrypt certificate with the long or short chain. It's just a matter of how your webserver and/or ACME client is configured.
Also see Long (default) and Short (alternate) Certificate Chains Explained
The webserver at www.worthingtonlibraries.org appears to have been (manually) configured to serve the short chain. This is not the current default.
10 Likes
Awesome. That's what I thought we did. I guess I forgot that the longer one was still available. So these upcoming 2024 changes should not affect us, since we are using the short chain already.
8 Likes
You got it!!
7 Likes
Less of a question, more of two comments.
The API Announcement might be more clear with a bonus paragraph that just states something like:
Starting 2024-02-08, the "Short Chain" will become the new default chain, and the "Long Chain" will become an alternate chain.
This wasn't clear until I read through the blog post.
The stuff about maintaining support for Android also seems a bit confusing, and might benefit from some "hints" on what people should look for in documentation:
Not all ACME clients support Alternate Chains, but many of the most popular ones do. Subscribers who need to continue support for legacy Android devices can configure the chain preference with a compatible client. For example, with Certbot this can be done with the
preferred-chainoption
Perhaps listing what the option is for a handful of the most popular clients would be beneficial there too.
6 Likes
Yay, I'm glad the tiny post here got you to read the whole real thing!
We have seven months until the first change; we'll be using that time to prep additional documentation.
6 Likes
In particular, the command to run for certbot (and probably other popular clients) to switch to the DST Root CA X3 chain for people needing old Android support should be listed somewhere prominently.
I also feel like something somewhere should address the situation on Windows-based servers, where I recall that many configurations wouldn't serve the cross-sign as part of the chain at all (and thus really nothing is changing for them, I think), but I'm not sure if that's still the case.
8 Likes
For Windows clients and servers (not using cross platform things like apache/nginx) the chain the (Windows) client will build and the chain the server will serve is to ISRG Root X1 (self signed) unless root certificate downloads/updates have been disabled. It's really hard to force Windows to serve the DST Root CA X3 chain because it will try really hard to avoid using the expired root. Most of these issues have been flushed out over the last year or two.
If you are using a cert with apache/nginx etc on Windows however the chain served will be to DST Root CA X3 if that's what your ACME client is defaulting to, but Windows clients will still build their own path once they get to R3 because they (should) know about ISRG Root X1 (self signed) and will prefer that.
I believe desktop Google Chrome and MS Edge both now provide their own browser specific root stores, but I think all mobile versions still rely on the Android/iOS stores. On iOS all browsers use Safaris Webkit under the covers, so it's dependent on their OS version.
Where I expect we will see client issues:
- Older android devices
- Older MacOS, iOS
- Linux servers that still don't know ISRG Root X1 (self signed), including those using curl to fetch stuff from elsewhere, and the elsewhere is currently using the DST Root CA X3 chain.
5 Likes
any estimate when this change will be active in LE Staging environment? apologies if i missed that bit. thanks!!
4 Likes
We are currently planning to make these changes in Staging about two days before we make the changes in Prod, as per our normal deployment process. That can be changed if people would find it useful to have the short chain as the default in Staging significantly sooner, but in general we try to minimize long-term diffs between Staging and Prod.
6 Likes
I understand needing to strike a balance between those using staging to diagnose something "now" vs. those using staging to test how their client would work in the future, but I think it might be beneficial for the lead time in this case to be on the order of a week or two, rather than a day or two.
It would also be helpful if in addition to having good documentation on how to select a chain in production, there was good documentation on how to select a chain in staging. The current staging environment documentation doesn't even mention "Doctored Durian Root CA X3" at all.
8 Likes
I run an API with a user base that has clients on older hardware, I suspect some of those will run into issues, I could capture HTTP User-Agent for a while to attempt to identify older clients.
Does anybody have say a list of User-Agent specs that are likely going to run into issues when the chain shortens? Something like that would be ideal so I can match incoming requests against such list and notify corresponding users to update.
Has anyone compiled such overview?
If you need an A/B test you could get another cert with the shorter chain and apply for an hour per day to see who squeals.
I don't believe you can reliably determine that from the user-agent. Each client will be using some form of TLS library (like openssl) to communicate but in turn that TLS library will reference a CA certificate store for the list of root certificates it trusts, and that store can be at a shared OS level or even specific to each distribution of the client software. So, two users could be using the same software but have CA trust stores in very different states.
I think in your case you could start to compile the list of user-agents you see and how frequently used they are (you could export a log to a CSV and look at in PowerBI etc to aggregate) and make a guess as to which ones are most likely affected by how old googling says they are.
Another strategy to consider is varying the CA you use, for instance BuyPass GO currently uses a root from 2010, which is quite likely to be present in stores. The benefit of testing this live (and accepting the possibility of outages for some clients) is that intermittent testing can be switched off and on, so if you're going to be affected at a significant scale it's good to know in advance.
5 Likes
The closest "official"-ish list of working clients is the Certificate Compatibility documentation page. Clients listed under "Platforms that trust ISRG Root X1" will continue to work, and clients listed under "Platforms that trust DST Root CA X3 but not ISRG Root X1" need the cross-sign to be served. As @webprofusion says, though, those platforms probably don't map easily to the user agent strings. You might be able to do some sorts of analysis, though. Would certainly be good if someone ran through the methods and shared on the community, though.
The main system that the cross-sign is for is older Android, so if your API users aren't via Android applications then it probably won't impact things. Hard to really test, though.
One of the challenges of doing A/B testing, of trying to switch chains and see what breaks, is that many clients (primarily regular web browsers, but maybe API uses too that use some of the same libraries) can cache cross-signs they've seen and use them to build a chain (as browsers work very hard to work around misconfigured web servers), so even once you stop serving the cross-sign in your TLS handshakes, browsers that have ever seen the cross-sign (even from another site serving it) could in theory still be using it. While the actual expiration of the cross-sign should be a non-event (especially given that the root it's signed from is already expired), it may be that there's some oddball system out there which is still using it even once servers stop sending it, and would break once it expires. It's really hard to know given the wide range of client software out there. (It was shocking to me in the first place years ago when I learned that the cross-sign would be useful, since that means that these Android devices are regularly connecting to the Internet but no longer get security updates.)
8 Likes
Hi,
In the blog post it says "On Thursday, Feb 8th, 2024 , we will stop providing the cross-sign by default in requests made to our /acme/certificate API endpoint". How can we request the longer chain? Is there a query parameter that we need to include? or does our ACME account need to be allow-listed to be able to request the longer chain?
Thank you!
2 Likes
You would need to configure your client. (And as said above, I'm hoping this is the kind of documentation that they'll be publishing between now and then.) For instance, for certbot, I think you'd need something like
certbot reconfigure --cert-name example.com --preferred-chain "DST Root CA X3"
(But that's from memory and I haven't tried it.) You should be able to configure it now, actually, and if you've specifically configured it then you'll continue getting the one you've configured. All that's changing is which chain a client will get if they just get the default and haven't picked a specific one.
8 Likes
Thank you for the response! To be sure, can someone from Let's Encrypt confirm that this will work after February 8th?
1 Like
Yes, between Feb 8 and June 6, it will still be possible for clients to request the longer, more-compatible-with-older-Android-devices chain which goes all the way up to DST Root CA X3.
From an ACME / RFC8555 perspective, the way to do so is:
- Extract the certificate URL from the finalized Order object
- GET this url, which will provide both the default chain and a set of
Link: rel="alternate"headers - GET the urls specified by all of those headers, which will provide alternate chains for the same cert
- Parse all of the chains, and use whatever heuristics you like to select the chain to install on the web server
For example, if you visit https://acme-v02.api.letsencrypt.org/acme/cert/0426e21be0d8cd31cfe4407e0ece8bac9e0b you'll download the current lencr.org certificate with the long chain. If you inspect the response headers, you'll see Link: <https://acme-v02.api.letsencrypt.org/acme/cert/0426e21be0d8cd31cfe4407e0ece8bac9e0b/1>;rel="alternate". And if you visit that URL, you'll download the same cert but with the shorter chain.
Between Feb 8 and June 6, this will still work exactly the same, except that the initial request will return the shorter chain and the Link: rel="alternate" header will point to a URL where you can download the longer chain.
Some clients, such as certbot mentioned above, provide functionality to do this for you. Certbot in particular allows the operator to configure a name like "ISRG Root X1" or "DST Root CA X3", and it will select the chain where that name appears closest to the root. Other clients allow the operator to configure chain selection in other ways. If you use your own custom client, obviously you can make it behave however you like.
8 Likes