Help thread for DST Root CA X3 expiration (September 2021)

[alexsd64]

I posted this in another thread, hoping to get some clarification:

We have a client who is using the DST Root CA cert chain currently as well as our current application cert expiring on 12/19. The DST Root CA and intermediate cert are expiring on 9/30 and I understand that it will be an automated switch to ISRG Root X1.

The problem is, they need time to update certs on their end. They have told us that they downloaded and deployed the ISRG Root X1 and got a chain peer identification issue during the handshake since it didn't match the cert chain.

Is there a way to manually switch it over before 9/30? Or run some kind of command? Apologies if I misunderstand the certs entirely, I'm not super knowledgeable about these things.

[rg305]

@it-1313

Q1. how can I test that I will have a correct chain after 30th of September

A1. The chain supplied today is the same chain that will be supplied after Sept. 30th.

@it-1313

Q2. how can I create a certificate for LDAPS which also works for linux clients (I read somewhere that LDAPS does not inform the client which CA to use? So how could I assure that?

A2. Please have a look at the video posted here: Join us and learn about certificate chains! - #35 by jple

[xnox]

When testing things from command line it is useful to install faketime package and then one can do

faketime 2021-10-01 openssl s_client -connect

To test and check if things from your environment still work after expiry. Ditto any other commands like curl, wget, etc.

[it-1313]

Hello,
thanks for your reply. How can you "tweak" DC to server the short ISRG-self signed chain? Simply by deleting the soon expiring R3 intermediate?

What I have found out so far:

1. ON DC, I disable all DST X3, all old R3 (expiring 2021-09-29) on the DC. I then check my server certificate, and the certificate chain is correctly Server < R3-new < ISRG-self signed.
   However, on the client side (ubuntu 18.04 latest) it still tries to verify with the DST. I then delete the DST on the server, and yet the chain is still expiring (trying to use DST).

However, what works is this:

1. I create a crt file with the new R3 < ISRG-self signed
2. I refer it in a openssl call with : faketime '2021-10-04 20:30:00' openssl s_client -CAfile R3_ISRGX1-self-signed.crt -showcerts -connect drty1.DC.org:636

-> maybe not good practice to put an intermediate into the root store, but I need to somehow make this work in the coming days.

however, this doesn't work:

1. adding only the new R3 via CAfile (although ISRG self-signed is trusted and linked in /etc/ssl/cert)
2. adding only\ the ISRG-self-signed via CAfile (as the DC does not serve the new R3intermediate?)

so I thought, I just combine new R3 intermediate < ISRG-self signed in a crt file, and put it in my trust store on my ubuntu machine, and update-ca-certifictes. Its added, However, this does not work:
faketime '2021-10-04 20:30:00' openssl s_client -CAfile R3_ISRGX1-self-signed.crt -showcerts -connect drty1.DC.org:636
--> gives me a 10 certificate expired

I also removed the dst again from /etc/ssl/cert, still I get 10 expired if I try with faketime after 2021-10-01.

I think SOGo just needs a valid chain of trust based on what is available/working in /etc/ssl/cert/.

I really don't know what else to do anymore. I don't understand why ubuntu still tries to use the DST, although its not in the trust store anymore ...

[tialaramex]

@alexsd64

The manual choice you can make is the same now as it will be after the expiry happen, you can choose to assemble a chain that goes Your certificate <- R3 <- ISRG-signed-by-DST or one that just goes Your certificate <- R3. The ACME client software you use to obtain certificates may offer a way to choose which, you will need to read its documentation if you need this.

The "automated switch" isn't so much a switch as the consequence of time passing. The DST Root CA X3 certificate expires at a fixed date in the near future, and much software (including all popular web browsers) will just go "I trust ISRG Root X1 and that's in this chain, so we're good" regardless of which of the above manually chosen chains it sees, since they do both involve ISRG Root X1.

Unfortunately some software, particularly older versions of the OpenSSL library, can't conceive of having other reasons to trust something, having discovered that DST Root CA X3 is expired, they conclude than any certificates matching that are untrustworthy, even if there's other reason to think they're OK.

If you're confident that all clients trying to connect to your service trust ISRG Root X1, you can send the shorter chain Your certificate <- R3 which doesn't mention DST Root CA X3 and thus shouldn't cause problems (so long as ISRG Root X1 is trusted). You should look at instructions for your ACME certificate client to have this happen automatically on renewal, but in terms of what you can do immediately to test this, if the certificate "chain file" for your service is just plain text you can open it in your preferred text editor, you should see it has several distinct blobs inside it that begin with BEGIN CERTIFICATE and end with END CERTIFICATE, if there are three blobs, make a copy first just in case but you can literally just remove the last blob, corresponding to a certificate for ISRG Root X1 by DST Root CA X3, and save the file and now it doesn't mention the expiring DST Root CA X3.

[webprofusion]

It's really important to emphasis that on Windows server, running typical windows based software (e.g. IIS etc), the OS builds the chain it's going to serve - it pretty much ignores whatever chain your ACME software may have prepared once it recognizes the intermediate then goes off on its own. So as Ryan says, you have to add/remove certificates in the machine certificate store (and often also in the Local System user certificate store) to get windows to serve a particular chain.

Here is my current advice for Windows users with chain issues (including suggested fixes): Let's Encrypt DST Root CA X3 expiry Sept 30th 2021 | Certify The Web Docs

On linux though, I can't help, each linux app has it's own idea of trust (as defined by the CA and intermediate stores the app or the library it uses, understands) and it varies depending on whether the app uses OpenSSL or not and what's in the trust store.

You probably do still get the same problem that up until Sept 29th, the expiring R3 will be preferred, because it's notBefore date is newer that the R3 you actually want it to use. I found that to test in the future you really had to have both the client and server dates set forward.

Personally, I'd say if things are getting really complex trying to get a trusted combination, then try an alternative CA.

[rmbolger]

@it-1313

How can you "tweak" DC to server the short ISRG-self signed chain?

Take a look at the reg files I attached to my post. You don't have to use them, but they have comments that explain exactly what they're doing. Notably, they don't do anything in the Trusted Roots store. All of the work is in the Intermediate and Untrusted stores. Most importantly, the expiring R3 goes into Untrusted rather than just being deleted.

On the Ubuntu side, I literally didn't have to do anything. openssl s_client just works with no CAfile parameter against my DC regardless of whether I'm serving the short or long chain, even using faketime from after the DST expiration. That's with version 1.1.1f-1ubuntu2.8 of the openssl package and 20210119~20.04.2 of the ca-certificates package. But my understanding is that any openssl version 1.1+ should work like this.

If you've messed with the trusted certs or /etc/ssl/openssl.cnf on the Linux side, you're in uncharted territory from my perspective. I'm not sure what advice to give beyond this.

[tbjornli]

tl;dr Problem caused by a manually installed root certificate in /etc/ssl/certs/ and was solved by removing this.

We experienced the same issue on our servers (Ubuntu 20.04.3 fully patched and running openssl 1.1.1f) on Friday, when the ca-certificates package was automatically updated.

Connecting to a domain with a LE certificate returned the error
verify error:num=2:unable to get issuer certificate

On a freshly installed and fully updated Ubuntu 20.04.3 running the same openssl version there was no such error.

What I could see from the output was that on our servers the certificate chain contained the DST Root CA X3 certificate while on the freshly installed server it contained the R3 certificate from LE.

I discovered that the reason for this was that we had manually installed a copy of the DST Root CA X3 into /etc/ssl/certs/ on our servers and for some reason this was being used by openssl instead of the R3 certificate.

Removing the manually installed certificate solved our problems.

Output with /etc/ssl/certs/lets-encrypt-r3-cross-signed.pem

$ openssl s_client -connect valid-isrgrootx1.letsencrypt.org:443 > /dev/null
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=2:unable to get issuer certificate
issuer= O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
issuer= C = US, O = Let's Encrypt, CN = R3
verify return:1

Output without /etc/ssl/certs/lets-encrypt-r3-cross-signed.pem

$ openssl s_client -connect valid-isrgrootx1.letsencrypt.org:443 > /dev/null
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = valid-isrgrootx1.letsencrypt.org
verify return:1

I hope that this information can be useful for someone having the same issue.

[rg305]

@tbjornli

running openssl 1.1.1f

Correct, the issue is known with openssl less than 1.1
(Like with: 1.0.2*)

[xnox]

@rg305

Correct, the issue is known with openssl less than 1.1
(Like with: 1.0.2*)

Please note that in Ubuntu, we have patched 1.0.2* based version in Xenial (16.04 LTS) and up.
And also in Ubuntu we hare removed DST Root CA X3 root certificate, meaning that unpatched copies of OpenSSL 1.0.2 also will continue to work after Friday, with either short or long chains.

[rg305]

I can confirm that:

Welcome to Ubuntu 16.04.7 LTS (GNU/Linux 4.4.0-210-generic x86_64)

Now has available updates:

The following packages will be upgraded:
  libgnutls-openssl27 libgnutls30 ubuntu-advantage-tools
3 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

@xnox, big thanks to all that made that possible

[rg305]

Hi @bellaf, welcome to the LE community forum

@bellaf

I was just wondering if this could potentially cause any issues.

Unlikely.

@bellaf

Currently, in my intermediate certificates, there are two of these R3 certificates, identical

Please double-click each one and post a screenshot of them.
[hold ALT+PrintScreen and then paste them here]

[Osiris]

@bellaf

since this intermediate certificate wasn't already on my laptop

Intermediates aren't supposed to be directly used by clients from some local store, only root trust anchors should. The client should build a trusted chain up to a trusted (root) anchor with aid of the intermediate send by the server, not by trusting the intermediate directly.

[Osiris]

@bellaf

it appeared that my browser was automatically using the DST Root CA X3 certificate chain and while the R3 intermediate certificate issued by DST Root CA X3 was in the user certificates on my laptop, this certificate wasn't and I assumed this meant it would cause issues when the R3 intermediate certificate that was already listed expired.

It should not make a difference whether you install the intermediate or not: your wwebbrowser or other clients should behave normally without it.

[rmbolger]

Windows will (sometimes) automatically download and cache intermediate certs in the Intermediate store as part of its chain building. Usually these end up in your user account's copy of the store as opposed to the system-wide copy of the store. But they also occasionally end up in the system-wide copy. The reason you're seeing two is that you likely have a copy in both places and the view of your user stores are basically a combined view of your user store and the system store.

Having a duplicate copy shouldn't hurt anything. But @Osiris is correct that you also shouldn't need to manually muck with the intermediates as a normal web browsing client user. Windows is building to the DST chain because it's still valid for another few days. When it expires, it will automatically build to the ISRG chain.

[webprofusion]

This may have already been mentioned on this thread but I'm starting to see reports of Ubuntu etc removing trust (via ca-certificates updates) for the expiring R3. This is resulting in earlier than expected trust failures:

Postmortem: Partial RavenDB Cloud outage - RavenDB NoSQL Database

[Osiris]

@webprofusion

for the expiring R3

I think you meant to type X3 here?

Also, if you read the article closely, this is due to an incorrectly send certificate chain:

Digging a bit deeper, we can use OpenSSL to see the real details:

ayende@oren-pc:~$ openssl s_client -connect a.free.ayende.ravendb.cloud:443 CONNECTED(00000003) depth=1 C = US, O = Let's Encrypt, CN = R3 verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = *.free.ayende.ravendb.cloud verify return:1 --- Certificate chain 0 s:CN = *.free.ayende.ravendb.cloud i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 ---

Here you can clearly see they used the cross-signed R3 intermediate signed by the IdenTrust DST Root CA X3, which is NOT the default chain since May this year! All certificates issued with this old chain have already expired, so there really is no good reason to be using this older chain any longer.

I.e.: by using an incorrect, outdated chain, they had this problem.

[webprofusion]

I haven't read the whole article but no, I think I mean the expiring intermediate R3, which expires shortly before the DST Root CA X3. Some tools require pre-stored knowledge of the intermediate (as well as the root, regardless of the present server chain) and if you take that away then the chain loses it's trust.

As I've mentioned elsewhere, it's possible for Windows server (in particular) to be serving using the wrong R3, regardless of what their ACME client gave them, because the expiring R3 is preferred by Windows itself, due to it's newer notBefore date (until it actually expires). [In their case though, they were caching the old R3 on linux servers]

[Osiris]

@webprofusion

I think I mean the expiring intermediate R3, which expires shortly before the DST Root CA X3. Some tools require pre-stored knowledge of the intermediate (as well as the root, regardless of the present server chain) and if you take that away then the chain loses it's trust.

The currently in use R3 isn't expiring:

        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3

All currently active chains use the R3 signed by ISRG Root X1. The default chain uses a cross-signed ISRG Root X1 by DST Root CA X3, which has been renewed for the purpose of Android compatibility. The alternate chain uses a self-signed ISRG Root X1. R3 cross-signed by DST Root CA X3 has been retired and should not have been in use. Using that old chain is bad.

@webprofusion

As I've mentioned elsewhere, it's possible for Windows server (in particular) to be serving using the wrong R3, regardless of what their ACME client gave them, because the expiring R3 is preferred by Windows itself, due to it's newer notBefore date (until it actually expires).

That sounds like something a system operator should know and check. If any issue arises, this is probably due to a lack of understanding, care or competence IMO.

@webprofusion

In their case though, they were caching the old R3 on linux servers.

Which is also something they should have known, checked and fixed.

[nertzy]

Overnight we started seeing OpenSSL errors on a Heroku Rails application using the http gem.

OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)

The domain we are connecting to has a certificate that has paths to DST Root CA X3 and ISRG Root X1. I opened an issue with Heroku, but I wouldn't be surprised if many applications are in the same boat. My theory is that the base image trust store has DST Root CA X3, but not ISRG Root X1.