I'm not sure if it's misunderstanding or you simply stopped reading. The text gives an explanation why this works.
3 Likes
My apologies. My answer is there!
"Android has intentionally chosen not to use the notAfter field of trust anchors."
Thank you very much.
5 Likes
You can test what happens with a chain that only leads to ISRG Root X1 today, your ACME client might support an option to have it sort this out for you, or you can manually construct the chain, the individual certificates are signed documents, so you cannot change those, but the "fullchain.pem" file for example is just several certificates concatenated together, so if you remove a certificate from the file now it's not in your chain.
However, it will not be possible to check what happens exactly in September 2021, without travelling to September 2021. If you tell a computer today that it's already September 2021, it will say your certificates aren't trusted because they're out-of-date - you got them back in May (ie now). We can do some better experiments on the details in July and I believe there are plans for just that.
3 Likes
You can already test this today, if you have control over both client and server that you're testing -
You can issue a cert from Let's Encrypts staging enviroment. Staging is specifically configured to serve a chain up to a root that is already expired. However, the staging certificates are not trusted, so you need to temporarily add the Let's Encrypt staging certificates to your trust store do test. (Remove them after you're done).
The test certificates used in staging are documented here: Staging Environment - Let's Encrypt
I host a test server that is configured to serve a staging chain, if you don't want to set that up yourself: https://expired-root-ca-test.germancoding.com. Remember to add the staging certificates to your trust store when testing, and don't forget to remove them after that.
PS: As I just saw that the LE staging documentation is a bit confusing, here's the staging root certificates you need to add for testing.
https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem (This is a simulated "ISRG Root X1" certificate)
https://letsencrypt.org/certs/staging/letsencrypt-stg-root-dst.pem (This is a simulated "DST Root CA X3" that is already expired. Adding this to your trust store enables testing on how your trust store handles expired root certificates)
9 Likes
There is a test server that sends only the information to chain up to the ISRG Root X1 root: https://valid-isrgrootx1.letsencrypt.org/. It sends the leaf and the R3-signed-by-ISRG-Root-X1 intermediate, and so should be trusted if your client trusts ISRG Root X1. (But note that browsers may build other chains, so if you look in your web browser's certificate info you might also see the certificate of ISRG-Root-X1-signed-by-DST-Root-CA-X3 because your browser had seen that certificate elsewhere, even though the server itself didn't send it.)
There's also a test site that's configured with the "normal" chain rooted in DST Root CA X3, https://helloworld.letsencrypt.org/, but it hasn't renewed since the May 4 chain change so it's still using R3-signed-by-DST-Root-CA-X3 rather than the combo of R3-signed-by-ISRG-Root-X1 plus ISRG-Root-X1-signed-by-DST-Root-CA-X3.
Yes, if you request the "alternate" chain in your ACME client, it will send the chain without the ISRG-Root-X1-signed-by-DST-Root-CA-X3 certificate, so that your "root" can be the self-signed ISRG Root X1 in your trust store. This actually hasn't changed, and isn't expected to change for quite some time. The "valid-isrgrootx1" site mentioned above is set up the same way as a site that uses this approach. (See the great Production Chain Changes announcement that summarizes the plan.)
For the production chain, that is pretty true. But there is a staging environment, which is set up with a "mirror" of the real certs, except that the certificate corresponding to the DST Root CA X3 (called "(STAGING) Doctored Durian Root CA X3") is already expired.
So, what one could do, is have whatever system you're testing have both "(STAGING) Doctored Durian Root CA X3" and "(STAGING) Pretend Pear X1" in its trust store (be sure to remove after testing, as certificates from staging aren't audited or anything), or whichever combination of roots corresponds to the scenario you want to test. Then you can try using it to connect to systems that have been issued certificates from the staging environment (either the "normal" or "alternate" chain) to see what happens.
(Looks like @Nummer378 already said this while I was typing this up, but I already typed it so I'm leaving this here. 
)
@jple: Just bugging you again. Why isn't "(STAGING) Doctored Durian Root CA X3" mentioned or downloadable from the Staging Environment documentation? I'd think that needing to deal with it being or not being in one's trust store is the kind of thing that one would often be using the staging environment to test.
9 Likes
Is https://valid-isrgrootx1.letsencrypt.org/ an example of the new cert chain, including the Android workaround? I'd be interested to develop a Javascript routine that could reliabily test clients so that we might be able to show warnings to human users browsing our websites before Sep 30. (We did something similar for TLS 1.2 support in advance of disabling TLS <= 1.1 on our httpds.)
Ideally there'd be an https site up now using the new chain (with the Android workaround) hosting either a CORS-compatible URL returning a simple JSON object like {"success":true} that we could test with XmlHttpRequest or a 1x1 pixel image that could be tested with event handlers on an IMG element.
Thanks so much for all your work!
4 Likes
No, valid-isrgrootx1 is an example of the "alternate" chain, with ISRG Root X1 as the root.
You're unlikely to be able to reliably test web browsers, including mobile web browsers, because they go through extraordinary efforts to "guess" and "find" intermediates that aren't actually being sent by the web server, in order to work around that a lot of servers have incorrect configurations. (See for instance, what Firefox does to work around poorly-configured servers, and other web browsers do similar and different things.) So you can have two identically-configured devices, but one happened to go to a web site in the past to help it "know" about an intermediate, but the other didn't, and so when going to a server that should send that intermediate but doesn't, one will work but the other will fail. (At least, until the intermediate falls out of its "cache" on the first device. Maybe.) Testing from within a browser kind of requires it to be a "fresh" factory-reset device to have any sense of repeatability, and then of course you're not going to see much about how "real-world" usage will work.
The only "official" site that I know of is https://helloworld.letsencrypt.org, or maybe if you're looking for JSON you could try getting to https://acme-v02.api.letsencrypt.org/directory (I'm not sure what chain the API will be using long-term, though). It should be easy enough to spin up your own, though, since after all getting Let's Encrypt certificates are free. The default chain is, and will continue to be, rooted in DST Root CA X3. The tricky part is just that it's hard to test what will happen once that certificate expires later this year, without using the mirrored certificates from the staging environment as described above and messing with your device's trust store.
9 Likes
Not a solution for servers, but I just went to Chain of Trust - Let's Encrypt and installed ISRG Root X1 and ISRG Root X2 on my iOS device running 9.3.6 (the last 32-bit release from Apple).
Should this be enough for September?
4 Likes
All of our staging certs can be found here: website/static/certs/staging at master · letsencrypt/website · GitHub
I created a PR to add that to the staging environment (which, always feel free to do and tag me @jaykaypea on GitHub so I can review and get it merged!): Update staging-environment.md by jaykaypea · Pull Request #1236 · letsencrypt/website · GitHub
6 Likes
Yep! That should be enough.
5 Likes
I was reading the bulletin and AFAIK it will be a nonevent for most hosts. I use a roll your own host and I handle all the functions. certbot is fine until 7/21 before it rots away, but I am setting up a more powerful machine for the web server to handle more sites and traffic
1 Like
FYI, the new, longer default chain (R3 <= ISRG X1 <= DST X3) seems to be processed incorrectly by the popular SSL Labs test; instead it shows two possible chains: R3 <= ISRG X1, and R3 <= DST X3 with a missing intermediate.
Reported in an SSL Labs community post here: Qualys Discussions
3 Likes
@ghen TLS clients can build the chain however they want. It's perfectly fine to ignore intermediates not required for building a trusted chain.
That said, SSLLabs shouldn't penalize the webserver, as the chain is valid and it would make sense for SSLLabs to show the path the server has send the client too.
6 Likes
I understand the current chain is redundant and there are multiple ways to validate it. However the way SSL Labs displays an incomplete chain (with "extra download") will make users believe something is missing, whereas the corresponding intermediate is actually present.
2 Likes
The intermediate certificate marked "extra download" is not the present. The two intermediates (one present in the chain and the other via "extra download") have the same subject and key, however the issuer is different. Even in the image in your article shows that, there are two distinct R3 certificates.
You are right that Qualys SSL labs should present not two but three different trust path for the shake of completeness.
6 Likes
Hello all, and thanks to Let's Encrypt for the excellent work in sharing this information on the right time and absolutely clearly.
What about Wget and Curl in Raspbian? Is there any way that I'd make sure in advance, which versions of them will trust the new certificate? If I will not know it, I have to update every Raspberry Pi that I have sent to my customers before the end of September.
Thanks for your help! 
2 Likes
Raspbian is just slightly modified Debian, so the requirements are the same as for Debian:
Assuming the system is otherwise up to date (apt-get update && apt-get upgrade), Raspbian Stretch (9) or newer will have no problems.
Edit: This assumes that wget and curl both use OpenSSL and not GnuTLS. I believe this to be the case on Raspbian Looks like wget uses GnuTLS on Raspbian Buster. For GnuTLS the situation is more complicated, as only versions from June 2020 onwards are definetly fixed - those aren't shipped even in the most recent version of Debian. apt-get and some other internal tools use GnuTLS by default. It's possible that patches are or were backported though.
9 Likes
Let me echo that and agree with you: There's a lot of esoteric details here that most people don't ever think about since they just want their sites to work, and they're trying their best to give everybody what they need (which is a problem because there doesn't look to be a good way to have both old-Android support and old-OpenSSL support).
Well, it should just work as @Nummer378 says, but as discussed above it's difficult to test expirations ahead of time. I guess my questions are: (1) Are you in control of the servers being connected to by the Raspbian clients, and (2) Do that server also need to handle old-Android clients? If the answer to (1) is Yes and the answer to (2) is No, then I would suggest testing that the clients can connect to https://valid-isrgrootx1.letsencrypt.org/ (like curl https://valid-isrgrootx1.letsencrypt.org/ doesn't give an error) and if that works then configure the server to use the "alternate" chain that uses ISRG Root X1 directly as the root (how to do this exactly depends on your server's ACME Client, like for certbot you would add --preferred-chain "ISRG Root X1" I think). The main reason for doing that is that you can test it now and be sure it will continue working in the future, since then you're not dependent on the DST Root Expiration at all. But if you don't do anything you're probably still all set for the most likely scenarios.
9 Likes
As I happen to have some Raspberry's running Raspbian here, I just did the quick test:
# cat /etc/os-release
PRETTY_NAME="Raspbian GNU/Linux 10 (buster)"
# wget https://expired-root-ca-test.germancoding.com --ca-certificate=certs-combined.pem -d
-> Works ok, no error
# curl https://expired-root-ca-test.germancoding.com -v --cacert certs-combined.pem
-> * SSL certificate verify ok.
So everything ok, at least on Buster. I haven't checked Raspbian Stretch.
Another interesting fact is that curl definetly seems to use OpenSSL:
# curl -V
curl 7.64.0 (arm-unknown-linux-gnueabihf) libcurl/7.64.0 OpenSSL/1.1.1d zlib/1.2.11 libidn2/2.0.5 libpsl/0.20.2 (+libidn2/2.0.5) libssh2/1.8.0 nghttp2/1.36.0 librtmp/2.3
But wget seems to be compiled with GnuTLS support inbuild:
# wget -V
GNU Wget 1.20.1 built on linux-gnueabihf.
-cares +digest -gpgme +https +ipv6 +iri +large-file -metalink +nls
+ntlm +opie +psl +ssl/gnutls[...]
Link:
gcc -I/usr/include/p11-kit-1 -DHAVE_LIBGNUTLS -DNDEBUG -g -O2
[...]
gnutls.o http-ntlm.o ../lib/libgnu.a[...]
Yet both tools seem to work, that's interesting, because I thought GnuTLS wasn't fixed on Debian. Maybe they backported a patch.
10 Likes