Help thread for DST Root CA X3 expiration (September 2021)

Hello,

The test website for ISRG Root X1, https://valid-isrgrootx1.letsencrypt.org/ is also serving a second certificate with DST Root. This makes it difficult to test compatibility of ISRG Root X1. Can we have the website deliver only ISRG Root X1 issued certificates?

image

3 Likes

The test site does only send the R3-signed-by-ISRG-Root-X1 certificate over the TLS connection. But that doesn't mean that's what your browser will show when you look at it. You need to use a tool like openssl or an online tool designed for looking at the sent certificate chains in order to see what's actually getting transmitted. Browsers and operating systems will just show you a way they found a valid chain, even if it's not related to the intermediates that were sent over the connection.

12 Likes

This is one of the best tools I've found for viewing the actual chain being sent:

https://www.digicert.com/help/

7 Likes

Nice addition to the toolbox.
Thanks Griffin

7 Likes

I think the trick is to delete the old R3 (the one issued by DST Root X3) from your Windows machine under "manage user certificates" > Intermediate certification authorities (possibly under "manage computer certificates" as well), this forces windows to look at the newer R3. If you don't have that one in your "intermediate" store you can get it from https://letsencrypt.org/certs/lets-encrypt-r3.der but try it without.

I'm testing this stuff today as well (by setting a bunch of machines to be in the future!).

6 Likes

We were thinking that we'd do this around the time that DST Root CA X3 expires, but there's no reason not to do it earlier, really. It's now done, and http://r3.i.lencr.org is serving the version of R3 issued by ISRG Root X1, rather than the version signed by DST Root CA X3.

13 Likes

So I'm having an issue with my certs still showing R3 as an intermediate cert which expires 9/21.
When I pull up my full chain - the intermediate is a cert that expires in 2025, but when I click on the certs in Firefox or Chrome - I see the old R3.
I'm probably stupid and missing something, but can someone explain and reassure me all my stuff isn't gonna break next month?

3 Likes

Welcome to the Let's Encrypt Community :slightly_smiling_face:

You likely have the retired R3 (signed by DST Root CA X3) stored in your cache. Try clearing it.

You can use the following tool to see exactly what certificate chain your webserver is serving:

https://certlogik.com/ssl-checker/

4 Likes

thank you for your reply!
When I run checkers such as those found at ssllabs or digicert, they all throw errors about having an invalid chain and having to download the old R3 cert.
I want to believe it's a my browser issue, but I'm really nervous as I've got this in production and not sure if I should bail and go get a different cert for the next year to bridge this gap...

4 Likes

Modern browsers build their own chains. There's lots of logic involved including caching, pre-validation of CA signatures, preloading of CA certificates and many more.

All of this has the effect that the chain displayed by the browser has almost zero relationship with the chain send by the server - the browser effectively does what it wants. In fact, many browser configurations do not even care if you send the intermediates at all*. To diagnose certificate chains, browsers are the wrong tool nowadays.

Older, or more simple TLS clients still rely on the server sending a correct and valid chain - which you're probably doing.

*Some browsers not doing CA preloading need to "learn" the intermediate first, by receiving them once and then remember them (caching). In theory this doesn't need a certificate chain send by server, but can also be done via AIA loading, though I don't have knowledge how this is currently implemented in diffferent browsers.

8 Likes

If you used the tool I gave you and it says that your webserver is not serving the full chain then the problem is that your webserver is not serving the full chain. Your webserver should be serving all of the following (in this exact order):

your leaf cert
https://crt.sh/?id=3334561879
https://crt.sh/?id=3958242236

4 Likes

DST Root CA X3 expires at 10 pm on September 30, 2021. I want to confirm whether it will affect my website? Will the validity period of the root certificate be automatically extended?

1 Like

I have conducted additional testing for Windows server specifically and observed the following:

Windows may continue to serve Leaf > R3 > DST Root CA X3 after expiry if the expiring R3 is present in the Intermediate Certificate Authorities store at the level of Local System. In my testing the server did not automatically switch over to the non-expiring R3 given 30 mins after expiry. A reboot can be performed to force the correct R3 to be served.

Removing the expiring R3 cert manually can be done to expedite the changeover:

Powershell method:

  • Launch powershell as local system: psexec.exe -i -s powershell.exe
  • Delete the expiring R3:

Get-ChildItem cert:CurrentUser\CA\48504E974C0DAC5B5CD476C8202274B24C8C7172 | Remove-Item

[Edit]
Registry method (simplest):

Run cmd as Administrator and delete the registry entry for the R3 intermediate in the Local System CA store:
reg delete HKEY_USERS\S-1-5-18\Software\Microsoft\SystemCertificates\CA\Certificates\48504E974C0DAC5B5CD476C8202274B24C8C7172 /f
[/Edit]

If the correct chain is still not being served, edit an https binding in IIS (delete an https binding then add it again), alternatively you may need to reboot the server to force the new chain to be served.

Additionally it was observed that Firefox builds it's own chain (as already noted elsewhere) Leaf > R3 > ISRG Root X1 when the old R3 in Leaf > R3 > DST Root CA X3 expires, regardless of the chain served by the server.

5 Likes

Hello,

I have kind of a specific question about this.

We have a simple SSL certificate that we get from Let's Encrypt using certbot.
So the full chain PEM we get contains: server certificate -> R3 -> ISRG Root X1 (issued by DST Root CA X3).

Alternatively, clients (browsers, JRE, etc) can use the alternative chain: R3 -> ISRG Root X1 (self-signed).

There is no real issue about this, it works like this is described in Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt.

Now here is my question: is there any way to make openssl s_client use the alternate chain ?

I'm asking this because we are monitoring our servers using openssl s_client -showcerts -connect ${SERVER}:${PORT} -servername ${FQDN} to retrieve all certificates sent by the server.
Then we check that the end date is not near for each individual certificate retrieved + the root CA.

So basically our probe will always be red as Let's Encrypt decided to deliver the ISRG Root X1 certificate issued by DST Root CA X3 (instead of the self-signed one).

Any help will be appreciated :slight_smile:

1 Like

Usually, no - unless you have special needs. Please see this blog post and the compatibility table.

No. DST Root CA X3 will expire and remain expired. For the majority of clients, ISRG Root X1 will be the new root.

5 Likes

Your server sends the certificate chain and this is what tools like OpenSSL display. If you want to change that you need to change what your server sends - this is server-side, not client-side. Modern browsers tend to build their own chains and ignore the ones send by the server - OpenSSL doesn't do this.

You could also consider to just fix your way of checking expiry dates: Any correct implementation will stop verifying at the first trusted certificate (trust anchor). For example, if you look at your openssl output, you will see something like this [output of -showcerts omitted].

CONNECTED(00000184)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = community.letsencrypt.org
verify return:1
---
Certificate chain
 0 s:CN = community.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Note the verify output displayed above the chain; see how it stops verifying at ISRG Root X1 and doesn't check DST Root CA X3? That's because ISRG Root X1 is already a root and there is no need to go further up. If your checking tool can do the same, you won't have this problem.

8 Likes

It has never been clear to me how a Web Browser retrieves intermediate certificates if the server does not send them (i.e. Sent by server). How does a Web Browser know where to retrieve these intermediate certificates? I am curious so I can do the same retrieval to those intermediate certificates so I can manually add them to my server's fully chain if necessary. And this DST Root CA X3 expiration and other cross signing is a scenario where I may run into such a situation.

1 Like

The intermediate certificates are supposed to be sent by the webserver. Your ACME client should give you all the necessary intermediate certificates with your leaf certificate when you request a certificate.

3 Likes

Correct; but in the past I have had a missing intermediate certificate and just want to know how to retrieve the intermediate certificate properly (and get the right one :sunglasses:) So I can correct it if I run :running_man: into again.

1 Like

Here are the current intermediate certificates for the default chain being served by Let's Encrypt:


3 Likes