Help thread for DST Root CA X3 expiration (September 2021)

What @rg305 said is correct -- those certificate bundles are "all roots that a certificate used by a Microsoft 365 site might chain up to". Those bundles contain many roots other than DST Root CA X3, so it's safe to assume that Microsoft 365 has handled the upcoming expiration in their own way. Either way, since Microsoft 365 doesn't appear to use Let's Encrypt to issue certs for any of their services, it's not within the purview of this forum.

7 Likes

I have no special requirements, currently my certificate Leaf -> R3-> DST Root CA X3. After DST Root CA X3 expires, will it automatically become Leaf ->R3-> ISRG Root X1? I am just worried that it will affect the use of the website after September 30.

2 Likes

@fangze217 It now goes directly from R3 to DST Root CA X3?
If so, which ACME client are you using and when/how was that chain built?

5 Likes

Hello,

I have two questions:

  1. We use the Authority Information Access extension in the certificates to build the full chain. The R3 intermediate points to http://x1.c.lencr.org/ regardless of if it is signed by the self-signed X1 or the cross-signed X1. Is it possible to update the endpoint to point to the cross-signed X1? It currently returns the self-signed X1.
  2. If a certificate uses the R3 intermediate with the self-signed X1, can we simply forward the cross-signed X1 or do we need to re-generate the certificate? I suspect that it was generated with the cross-signed X1 since we did not specify to use the alternate chain, but I just want to make sure that the two are interchangeable (except by the older devices).

Thanks!

4 Likes

The keypairs of the self-signed X1 and cross-signed X1 certificates are the same. Both can be used to verify the R3 intermediate. (Which makes sense, as currently there also is just a single R3 intermediate cert, not two for each X1 cert.) You can think of the common name of the intermediate/root certificates as the name for the keypair and not as much as the name for a certificate.

8 Likes

Thank you, Osiris.

I guess we can try to create a custom solution to resolve this, but I am hoping that the endpoint can be updated, if only because it is the currently active signer.

3 Likes

win-acme.v2.0.10.444,

2 Likes

Not sure if it will make any difference, but they are up to: win-acme.v2.1.18.1119

3 Likes

After the expiration of the DST Root CA X3 certificate at 10 pm on 9:30, 2021, will it affect the normal operation of the website? Will my website become an insecure address? I am using win-acme.v2.0.10.444 client, do I need to update? I am worried, my friend.

2 Likes

Try this to disable use of the expiring R3 in your chain [on Windows servers]:

  • Open certlm.msc, expand the tree for [1] Intermediate Certification Authorities > Certificates and for [2] Untrusted Certificates > Certificates.
  • Drag R3 issued by DST Root CA X3 from [1] to [2], this will disallow the expiring R3. Don't do this for R3 issued by ISRG Root X1
  • Ensure you have R3 issued by ISRG Root X1 installed in Intermediate Certification Authorities > Certificates, if not you can get it from https://letsencrypt.org/certs/lets-encrypt-r3.der and install it to that store.

Check your served chain again, you may need a reboot.

6 Likes

Operate in your way, remove DST Root CA X3 and add ISRG Root X1. After restarting the server, there is no browser certificate and it is not updated. Do I need to wait until September 30 to change it?

2 Likes

Hi all.

We us the terraform ACME provider
provider "acme" {

server_url = "https://acme-v02.api.letsencrypt.org/directory"

}

and we are expecting that we may have to update the provider version or options in order to get the new root cert but from the provider docs we cannot see any options or version changes to do this?

We have tested today with a new cert and we are still getting the DST Root CA. Does anyone one now how we can use the acme terraform provider and get the new Root CA?

2 Likes

I have installed R3 issued by ISRG Root X1 in Intermediate Certification Authorities> Certificates and restarted the server. Why is the certificate still DST Root CA X3 in the browser?

2 Likes

Browser might cache the chain.

check it with one of that tools

https://decoder.link/sslchecker/

https://www.sslshopper.com/ssl-checker.html

4 Likes

@mark-summers, that URL will not need to be changed.
It should already be providing the longer life chain.
Please provide the domain, or the chain file your ACME client received, to confirm.

4 Likes

To be clear I was asking you to disable the R3 (intermediate) certificate issued by DST Root CA X3, not the root certificates (that would break stuff).

There are versions of your chain: the chain served by your web server, and the chain your client (browser) builds. They are not necessarily the same. On your client/desktop machine you can do the same process with the R3, however you can check your web server chain just using SSL Server Test (Powered by Qualys SSL Labs)

5 Likes

Here's a fun video with @aarongable explaining this expiration! DST Root CAX3 Expiration Sept 2021 - YouTube

5 Likes

Hello,
I have a problem/question with LDAPS and Active Directory for which I use let's encrypt.

  1. I import the newest key/certificate to AD in personal certificates (created today with certbot). The chain of trust is using DST Root X3 (OK)
  2. I disable DST Root X3 on the AD Server: the chain of trust is using ISRG Root X1 (perfect)
  3. any other let's encrypt certificate/key I use for https just works after I e.g. disable the DST in the certificate store in Firefox (all use ISRG Root X1)

however:
3) on the client side (ubuntu server 20.04 with SOGo and LDAPs authentication):
a) with the latest update of ca-certificates, the DST Root X3 was removed from the trusted store.
openssl s_client -showcerts -connect drty1.myserver.org:636 results in

....
Verify return code: 20 (unable to get local issuer certificate)
....

I had to manually import the old DST root certificate again. So far so good.

But, what will happen after the 30th of September - I cannot make the certificate used in LDAP to pick the correct chain of trust (to test that after 30th of Sept everything is still fine)
Scenario A:

  1. on the LDAPS AD Server I disable:
  • all DST Root CA X3 certificates (serial 44 af b0 ... f8 40 6b)
  • the "old" R3 intermediate certificate (serial 40 01 75 ... 16 cd df)
  1. on the LDAPS server, when clicking on the chain of trust of my Let's encrypt certificate, I get a valid chain of trust (using the new R3 intermediate certificate (serial 00 91 2b ... a7 5f 5a), and the correct ISRG root certificate

  2. on the LDAPs client (with SOGo) I make sure the ISRG Root X1 is trusted (I get a valid chain of trust using the testing domain/website for the ISRG root certificate). However, I still cannot get a valid chain. I get
    openssl s_client -showcerts -connect drty1.myserver.org:636:

CONNECTED(00000005)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 335 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

(so no certificate has been found?)

additionally, when I look at the fullchain.pem file, it seems that ISRG Root X1 is validated by DST Root CA X3 (which will expire end of Sept).

Is this the problem?

While I disabled all DST, and the old R3 on the AD server, on a testing VM I set the date after 30 september, made sure IRSG is imported, and get error 10 (certificate expired). I don't/can't change the date on the server, but it seems to me the certificate will not validate through the correct chain.

2 Questions:

  1. how can I test that I will have a correct chain after 30th of September
  2. how can I create a certificate for LDAPS which also works for linux clients (I read somewhere that LDAPS does not inform the client which CA to use? So how could I assure that?
2 Likes

We had the same problem Today, as our "preliminary" updated linux machines were not able to establish SSL connection to windows servers (IIS and windows services) . We manually restored the old intermidate certificate on all of our Linux servers to restore connectivity in the system. Nothing to say we are very upset with this situation. Hope this will help somebody.

2 Likes

I've got a fully patched (as of today) Ubuntu 20.04.3 box with openssl 1.1.1f that I've done nothing certificate related with. I also have a Windows Server 2019 domain controller with an LE cert that is currently serving the short alternate chain. Verifying the cert using openssl seems to work for me.

$ openssl s_client -connect le-dc.ad.poshacme.win:636
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = le-dc.ad.poshacme.win
verify return:1
---
Certificate chain
 0 s:CN = le-dc.ad.poshacme.win
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

If I tweak the cert stores on the DC so it serves the longer android-compatible default chain, it still works.

$ openssl s_client -connect le-dc.ad.poshacme.win:636
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = le-dc.ad.poshacme.win
verify return:1
---
Certificate chain
 0 s:CN = le-dc.ad.poshacme.win
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Can you verify which chain your DC is currently serving by posting more of the openssl output? You may need to tweak the cert stores if it's still serving the old expiring DST signed R3. At a minimum that usually involves moving that DST signed R3 into the Local Computer "Untrusted Certificates" store and then rebooting. But you may also need to clean up copies that might exist in the SYSTEM user's Current User store as well.

I have no idea how SOGo works or what TLS library it's using under the hood. But I'd ignore the issues with that until you get the basic openssl test working.

If it helps, I've created these reg files that can be used to switch back and forth between the longer android compatible default chain and the shorter ISRG self-signed chain on your Windows boxes. They're commented if you want to understand exactly what they're doing. The main difference between them is whether or not a copy of the DST signed ISRG Root lives in the Intermediate cert store. Don't forget to reboot after applying one or the other.
ISRG LongDefault HKLM.reg.txt (31.4 KB)
ISRG Short HKLM.reg.txt (21.2 KB)

6 Likes