We've just posted an announcement regarding the upcoming OpenSSL Client Compatibility Changes for Let's Encrypt Certificates:
If you have any questions or concerns, feel free to put them here!
It's helpful to list in your post the versions of other common libraries (such as LibreSSL and GnuTLS) which are impacted, but it will likely make sense - given the tremendous number of users of Let's Encrypt - to collect compatibility data from community members, not only in the lead up to the event but also afterwards to help diagnose the inevitable deluge of support questions. There are doubtless dozens, or even hundreds, of distinct systems that may or may not have the preferred behaviour as Ryan described, some of them quietly based on OpenSSL (or other popular libraries) and some not. Roughly like with the compatibility information collected when Let's Encrypt was new (around trust of the Identrust root whose expiry is now our concern).
I also feel like the summary table (with the ticks and crosses) isn't doing a great job of communicating what it intends to, but I don't have a clear proposal to improve it, so I know this observation isn't very helpful.
4 Likes
In the other thread, you wrote the following:
By default, we'll recommend the certificate chain that does the most good for the most people -- the longer, Android-compatibility chain.
IMHO, it would be really great if LE offered a very simple small machine-and-human file that is really just an ordered listing of the current preferred chains. Perhaps dates might be in there too.
These threads with long detailed explanations are wonderful -- when I have time to read them. I'd really love to just look at a short text file and say "ok, we're using this root now" or "we should switch to another root on a specific date".
5 Likes
Currently, you offer two chains:
R3 (signed by ISRG Root X1) -> Subscriber certificate [default]
R3 (signed by DST Root CA X3) -> Subscriber certificate
and you mention that soon you will be moving to
ISRG Root X1 (signed by DST Root CA X3) -> R3 (signed by ISRG Root X1) -> Subscriber certificate [default]
R3 (signed by ISRG Root X1) -> Subscriber certificate
Two questions about how this transition is going to be managed:
- Are you going to start offering the longer chain as an alternate before you switch it over to be default, so that we have time to test with it?
- Once you do switch over to the longer chain as the default, will you still offer R3 (signed by DST Root CA X3) as an alternate?
We currently have code that selects the R3 signed by DST Root CA X3 chain using the alternate mechanism (done to prepare for the abandoned switch to ISRG Root X1 plan) and would like to know if we need to roll that back. It'd also give us an opportunity to see if there are any unexpected bumps in the road with the longer chain.
4 Likes
This is not currently part of our plans. The long chain contains an intermediate cross-signed by DST Root CA X3, a Root that is valid until September of this year. Outside of testing environments where you can control the root store and the system clock, you won't be able to test behavior after expiry. Luckily, thanks to the Sectigo expiry last year we already have a good idea which clients/versions have trouble with expired roots.
Come June 2021 we plan to offer two chains:
Long (default):
ISRG Root X1 (signed by DST Root CA X3) -> R3 (signed by ISRG Root X1) -> Subscriber certificate
Short (alternate):
R3 (signed by ISRG Root X1) -> Subscriber certificate
5 Likes
The (much) bigger concern for some of us is not a worry over the roots or OpenSSL cients, but over ACME Clients correctly handling:
- Long Chains
- Alternate Chains
Spinning up these features in a test environment using Boulder or Pebble is not a trivial task for people familiar with those systems; many clients in-the-wild were built without those tools against staging. The Staging system does not offer longer chains yet; I'm not even sure if offers alternate chains either. There are a lot of ACME clients and client-versions in production that may not support either of these concepts at all, or correctly.
IMHO, LetsEncrypt should have deployed these features to Staging at least a year in advance of the changes.
7 Likes
Staging does offer alternate chains (chaining back to Fake LE Root X1, Fake LE Root X2) but currently no chains with 2 intermediates. aarongable indicated in this thread on 20 Dec 2020 - Questions re: Extending Android Device Compatibility - "We hope to have new certificates in Staging, including a longer chain as the default, sometime in January."
5 Likes
Welcome to the Let's Encrypt Community @mikemelo, and thanks for posting!
We're still working hard to get a chain with two intermediates out to our staging environment. 2021 is already shaping up to be a very busy year for the Let's Encrypt SRE team. They've been hard at work adding further automation to our staging environment. Our SRE team is predicting that a staging chain with two intermediates will land in early to mid February.
3 Likes
From an ACME client standpoint we have some really awesome data on how most clients handled our last intermediate change. For any given day of issuance since our last intermediate change less than 0.23% of issuance resulted in an incorrect intermediate being served as part of the chain. This tells us that the overwhelming majority of ACME clients are serving the chain downloaded from the certificate URL described in RFC8555 Section 7.4.2 instead of hard-coding an expected chain. Subscribers using the default chain with most ACME clients should not see issues.
3 Likes
I agree with @jvanasco that the default chain response going from a single cert to multiple certs is likely to highlight more bugs with client software than the recent intermediate change. I'm also worried about bugs related to alternate chain selection since "ISRG Root X1" will exist in both chains.
In any case, the sooner people can test against something close to what will eventually be in production, the better. Looking forward to the changes in staging.
7 Likes
Pebble has been supporting longer chains since end of August 2020. One more reason to integrate Pebble into your tests if you're a ACME client author 
5 Likes
I want to know that which chain will https://acme-v02.api.letsencrypt.org/directory used?
Android-compatible chain or openssl 1.0.x compatible chain?
2 Likes
May I ask how this data has been acquired? Making a HTTPS to the hostnames in a certificate after issuance?
4 Likes
Yes, you nailed it! Within the first 48 hours of issuance we attempt a single TLS handshake and inspect the chain for intermediate mismatches. If your server doesn't respond on the first attempt we do not retry. While we fully expected that some clients may hard-code the intermediate, we were super jazzed to see that the overwhelming majority of Let's Encrypt subscribers are using clients that do not.
3 Likes
We will be serving the Android-compatible chain.
3 Likes
Thanks for clarifying things for us, beautifulentropy. I might write-up some kind of calendar/collection guidance in one place at some point in the near future as there are a lot of correlated information tidbits spread here and there.
2 Likes
After some discussions with my peers we have decided that the API will serve the shorter chain.
3 Likes
That's quite a change, right? Probably warrents a change in OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates and also invalidates the whole huge blog post Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt - Free SSL/TLS Certificates ? Or I'm interpreting the question incorrectly: @WillyYang might indeed literally talk about the chain used by the API endpoint itself. I interpreted it as "which chain will be send by default when issuing a certificate".
3 Likes
I am referencing the chain used by the API endpoint itself. Apologies for any confusion there!
4 Likes
Just so I'm understanding fully, we're talking about:
- the API endpoint serving the shorter chain (rooted at self-signed ISRG Root X1) to ACME clients for purposes of securing the ACME transactions
- the production primary chain becoming the longer chain (rooted at ISRG Root X1 signed by DST Root CA X3)
- the production alternate chain remaining the shorter chain (rooted at self-signed ISRG Root X1)
Thus the only change to the upcoming production primary chain is this:
leaf <- R3 <- DST Root CA X3
becomes this:
leaf <- R3 <- ISRG Root X1 <- DST Root CA X3
3 Likes