Help thread for DST Root CA X3 expiration (September 2021)

Here's a fun video with @aarongable explaining this expiration! DST Root CAX3 Expiration Sept 2021 - YouTube

5 Likes

Hello,
I have a problem/question with LDAPS and Active Directory for which I use let's encrypt.

  1. I import the newest key/certificate to AD in personal certificates (created today with certbot). The chain of trust is using DST Root X3 (OK)
  2. I disable DST Root X3 on the AD Server: the chain of trust is using ISRG Root X1 (perfect)
  3. any other let's encrypt certificate/key I use for https just works after I e.g. disable the DST in the certificate store in Firefox (all use ISRG Root X1)

however:
3) on the client side (ubuntu server 20.04 with SOGo and LDAPs authentication):
a) with the latest update of ca-certificates, the DST Root X3 was removed from the trusted store.
openssl s_client -showcerts -connect drty1.myserver.org:636 results in

....
Verify return code: 20 (unable to get local issuer certificate)
....

I had to manually import the old DST root certificate again. So far so good.

But, what will happen after the 30th of September - I cannot make the certificate used in LDAP to pick the correct chain of trust (to test that after 30th of Sept everything is still fine)
Scenario A:

  1. on the LDAPS AD Server I disable:
  • all DST Root CA X3 certificates (serial 44 af b0 ... f8 40 6b)
  • the "old" R3 intermediate certificate (serial 40 01 75 ... 16 cd df)
  1. on the LDAPS server, when clicking on the chain of trust of my Let's encrypt certificate, I get a valid chain of trust (using the new R3 intermediate certificate (serial 00 91 2b ... a7 5f 5a), and the correct ISRG root certificate

  2. on the LDAPs client (with SOGo) I make sure the ISRG Root X1 is trusted (I get a valid chain of trust using the testing domain/website for the ISRG root certificate). However, I still cannot get a valid chain. I get
    openssl s_client -showcerts -connect drty1.myserver.org:636:

CONNECTED(00000005)
write:errno=104
---
no peer certificate available
---
No client certificate CA names sent
---
SSL handshake has read 0 bytes and written 335 bytes
Verification: OK
---
New, (NONE), Cipher is (NONE)
Secure Renegotiation IS NOT supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
Early data was not sent
Verify return code: 0 (ok)
---

(so no certificate has been found?)

additionally, when I look at the fullchain.pem file, it seems that ISRG Root X1 is validated by DST Root CA X3 (which will expire end of Sept).

Is this the problem?

While I disabled all DST, and the old R3 on the AD server, on a testing VM I set the date after 30 september, made sure IRSG is imported, and get error 10 (certificate expired). I don't/can't change the date on the server, but it seems to me the certificate will not validate through the correct chain.

2 Questions:

  1. how can I test that I will have a correct chain after 30th of September
  2. how can I create a certificate for LDAPS which also works for linux clients (I read somewhere that LDAPS does not inform the client which CA to use? So how could I assure that?
2 Likes

We had the same problem Today, as our "preliminary" updated linux machines were not able to establish SSL connection to windows servers (IIS and windows services) . We manually restored the old intermidate certificate on all of our Linux servers to restore connectivity in the system. Nothing to say we are very upset with this situation. Hope this will help somebody.

2 Likes

I've got a fully patched (as of today) Ubuntu 20.04.3 box with openssl 1.1.1f that I've done nothing certificate related with. I also have a Windows Server 2019 domain controller with an LE cert that is currently serving the short alternate chain. Verifying the cert using openssl seems to work for me.

$ openssl s_client -connect le-dc.ad.poshacme.win:636
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = le-dc.ad.poshacme.win
verify return:1
---
Certificate chain
 0 s:CN = le-dc.ad.poshacme.win
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---

If I tweak the cert stores on the DC so it serves the longer android-compatible default chain, it still works.

$ openssl s_client -connect le-dc.ad.poshacme.win:636
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = le-dc.ad.poshacme.win
verify return:1
---
Certificate chain
 0 s:CN = le-dc.ad.poshacme.win
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Can you verify which chain your DC is currently serving by posting more of the openssl output? You may need to tweak the cert stores if it's still serving the old expiring DST signed R3. At a minimum that usually involves moving that DST signed R3 into the Local Computer "Untrusted Certificates" store and then rebooting. But you may also need to clean up copies that might exist in the SYSTEM user's Current User store as well.

I have no idea how SOGo works or what TLS library it's using under the hood. But I'd ignore the issues with that until you get the basic openssl test working.

If it helps, I've created these reg files that can be used to switch back and forth between the longer android compatible default chain and the shorter ISRG self-signed chain on your Windows boxes. They're commented if you want to understand exactly what they're doing. The main difference between them is whether or not a copy of the DST signed ISRG Root lives in the Intermediate cert store. Don't forget to reboot after applying one or the other.
ISRG LongDefault HKLM.reg.txt (31.4 KB)
ISRG Short HKLM.reg.txt (21.2 KB)

6 Likes

I posted this in another thread, hoping to get some clarification:

We have a client who is using the DST Root CA cert chain currently as well as our current application cert expiring on 12/19. The DST Root CA and intermediate cert are expiring on 9/30 and I understand that it will be an automated switch to ISRG Root X1.

The problem is, they need time to update certs on their end. They have told us that they downloaded and deployed the ISRG Root X1 and got a chain peer identification issue during the handshake since it didn't match the cert chain.

Is there a way to manually switch it over before 9/30? Or run some kind of command? Apologies if I misunderstand the certs entirely, I'm not super knowledgeable about these things.

1 Like

A1. The chain supplied today is the same chain that will be supplied after Sept. 30th.

A2. Please have a look at the video posted here: Join us and learn about certificate chains! - #35 by jple

5 Likes

When testing things from command line it is useful to install faketime package and then one can do

faketime 2021-10-01 openssl s_client -connect

To test and check if things from your environment still work after expiry. Ditto any other commands like curl, wget, etc.

3 Likes

Hello,
thanks for your reply. How can you "tweak" DC to server the short ISRG-self signed chain? Simply by deleting the soon expiring R3 intermediate?

What I have found out so far:

  1. ON DC, I disable all DST X3, all old R3 (expiring 2021-09-29) on the DC. I then check my server certificate, and the certificate chain is correctly Server < R3-new < ISRG-self signed.
    However, on the client side (ubuntu 18.04 latest) it still tries to verify with the DST. I then delete the DST on the server, and yet the chain is still expiring (trying to use DST).

However, what works is this:

  1. I create a crt file with the new R3 < ISRG-self signed
  2. I refer it in a openssl call with : faketime '2021-10-04 20:30:00' openssl s_client -CAfile R3_ISRGX1-self-signed.crt -showcerts -connect drty1.DC.org:636

-> maybe not good practice to put an intermediate into the root store, but I need to somehow make this work in the coming days.

however, this doesn't work:

  1. adding only the new R3 via CAfile (although ISRG self-signed is trusted and linked in /etc/ssl/cert)
  2. adding only\ the ISRG-self-signed via CAfile (as the DC does not serve the new R3intermediate?)

so I thought, I just combine new R3 intermediate < ISRG-self signed in a crt file, and put it in my trust store on my ubuntu machine, and update-ca-certifictes. Its added, However, this does not work:
faketime '2021-10-04 20:30:00' openssl s_client -CAfile R3_ISRGX1-self-signed.crt -showcerts -connect drty1.DC.org:636
--> gives me a 10 certificate expired

I also removed the dst again from /etc/ssl/cert, still I get 10 expired if I try with faketime after 2021-10-01.

I think SOGo just needs a valid chain of trust based on what is available/working in /etc/ssl/cert/.

I really don't know what else to do anymore. I don't understand why ubuntu still tries to use the DST, although its not in the trust store anymore ...

1 Like

@alexsd64

The manual choice you can make is the same now as it will be after the expiry happen, you can choose to assemble a chain that goes Your certificate <- R3 <- ISRG-signed-by-DST or one that just goes Your certificate <- R3. The ACME client software you use to obtain certificates may offer a way to choose which, you will need to read its documentation if you need this.

The "automated switch" isn't so much a switch as the consequence of time passing. The DST Root CA X3 certificate expires at a fixed date in the near future, and much software (including all popular web browsers) will just go "I trust ISRG Root X1 and that's in this chain, so we're good" regardless of which of the above manually chosen chains it sees, since they do both involve ISRG Root X1.

Unfortunately some software, particularly older versions of the OpenSSL library, can't conceive of having other reasons to trust something, having discovered that DST Root CA X3 is expired, they conclude than any certificates matching that are untrustworthy, even if there's other reason to think they're OK.

If you're confident that all clients trying to connect to your service trust ISRG Root X1, you can send the shorter chain Your certificate <- R3 which doesn't mention DST Root CA X3 and thus shouldn't cause problems (so long as ISRG Root X1 is trusted). You should look at instructions for your ACME certificate client to have this happen automatically on renewal, but in terms of what you can do immediately to test this, if the certificate "chain file" for your service is just plain text you can open it in your preferred text editor, you should see it has several distinct blobs inside it that begin with BEGIN CERTIFICATE and end with END CERTIFICATE, if there are three blobs, make a copy first just in case but you can literally just remove the last blob, corresponding to a certificate for ISRG Root X1 by DST Root CA X3, and save the file and now it doesn't mention the expiring DST Root CA X3.

5 Likes

It's really important to emphasis that on Windows server, running typical windows based software (e.g. IIS etc), the OS builds the chain it's going to serve - it pretty much ignores whatever chain your ACME software may have prepared once it recognizes the intermediate then goes off on its own. So as Ryan says, you have to add/remove certificates in the machine certificate store (and often also in the Local System user certificate store) to get windows to serve a particular chain.

Here is my current advice for Windows users with chain issues (including suggested fixes): Let's Encrypt DST Root CA X3 expiry Sept 30th 2021 | Certify The Web Docs

On linux though, I can't help, each linux app has it's own idea of trust (as defined by the CA and intermediate stores the app or the library it uses, understands) and it varies depending on whether the app uses OpenSSL or not and what's in the trust store.

You probably do still get the same problem that up until Sept 29th, the expiring R3 will be preferred, because it's notBefore date is newer that the R3 you actually want it to use. I found that to test in the future you really had to have both the client and server dates set forward.

Personally, I'd say if things are getting really complex trying to get a trusted combination, then try an alternative CA.

6 Likes

Take a look at the reg files I attached to my post. You don't have to use them, but they have comments that explain exactly what they're doing. Notably, they don't do anything in the Trusted Roots store. All of the work is in the Intermediate and Untrusted stores. Most importantly, the expiring R3 goes into Untrusted rather than just being deleted.

On the Ubuntu side, I literally didn't have to do anything. openssl s_client just works with no CAfile parameter against my DC regardless of whether I'm serving the short or long chain, even using faketime from after the DST expiration. That's with version 1.1.1f-1ubuntu2.8 of the openssl package and 20210119~20.04.2 of the ca-certificates package. But my understanding is that any openssl version 1.1+ should work like this.

If you've messed with the trusted certs or /etc/ssl/openssl.cnf on the Linux side, you're in uncharted territory from my perspective. I'm not sure what advice to give beyond this.

8 Likes

tl;dr Problem caused by a manually installed root certificate in /etc/ssl/certs/ and was solved by removing this.

We experienced the same issue on our servers (Ubuntu 20.04.3 fully patched and running openssl 1.1.1f) on Friday, when the ca-certificates package was automatically updated.

Connecting to a domain with a LE certificate returned the error
verify error:num=2:unable to get issuer certificate

On a freshly installed and fully updated Ubuntu 20.04.3 running the same openssl version there was no such error.

What I could see from the output was that on our servers the certificate chain contained the DST Root CA X3 certificate while on the freshly installed server it contained the R3 certificate from LE.

I discovered that the reason for this was that we had manually installed a copy of the DST Root CA X3 into /etc/ssl/certs/ on our servers and for some reason this was being used by openssl instead of the R3 certificate.

Removing the manually installed certificate solved our problems.

Output with /etc/ssl/certs/lets-encrypt-r3-cross-signed.pem

$ openssl s_client -connect valid-isrgrootx1.letsencrypt.org:443 > /dev/null
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=2:unable to get issuer certificate
issuer= O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
issuer= C = US, O = Let's Encrypt, CN = R3
verify return:1

Output without /etc/ssl/certs/lets-encrypt-r3-cross-signed.pem

$ openssl s_client -connect valid-isrgrootx1.letsencrypt.org:443 > /dev/null
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
verify return:1

I hope that this information can be useful for someone having the same issue.

5 Likes

Correct, the issue is known with openssl less than 1.1
(Like with: 1.0.2*)

4 Likes

Please note that in Ubuntu, we have patched 1.0.2* based version in Xenial (16.04 LTS) and up.
And also in Ubuntu we hare removed DST Root CA X3 root certificate, meaning that unpatched copies of OpenSSL 1.0.2 also will continue to work after Friday, with either short or long chains.

3 Likes

I can confirm that:

Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-210-generic x86_64)

Now has available updates:

The following packages will be upgraded:
  libgnutls-openssl27 libgnutls30 ubuntu-advantage-tools
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

@xnox, big thanks to all that made that possible :slight_smile:

2 Likes

Hi, I imported the R3 intermediate certificate issued by ISRG Root X1 onto my laptop running Windows 10 to prepare for the DST Root CA X3 expiration since this intermediate certificate wasn't already on my laptop, however, I accidentally imported it twice since it appeared that it didn't work at first. Currently, in my intermediate certificates, there are two of these R3 certificates, identical, I was just wondering if this could potentially cause any issues.

2 Likes

Hi @bellaf, welcome to the LE community forum :slight_smile:

Unlikely.

Please double-click each one and post a screenshot of them.
[hold ALT+PrintScreen and then paste them here]

2 Likes

Thanks for responding @rg305 . Here's the first certificate:

2 Likes

And here's the second one:

2 Likes

Intermediates aren't supposed to be directly used by clients from some local store, only root trust anchors should. The client should build a trusted chain up to a trusted (root) anchor with aid of the intermediate send by the server, not by trusting the intermediate directly.

4 Likes