Expiring trust anchor, openssl 1.0

heffergm · January 21, 2021, 4:17pm

Regarding https://letsencrypt.org/2020/12/21/extending-android-compatibility.html

From what I understand, the cross sign from DST Root CA X3 (expiring soon) being introduced into the new chain for Android compatibility will break clients using openssl 1.0.

Can anyone confirm this?

griffin · January 21, 2021, 9:09pm

Welcome to the Let's Encrypt Community

Let me get you someone.

@aarongable

I think this one's for you.

_az · January 21, 2021, 9:14pm

This was my concern too because of this and how it went wrong, but I never got around to testing it.

Plus it's kind of difficult to test because you need an EE certificate which expires later than the Identrust root, which won't be possible until July-ish; libfaketime doesn't quite do the job .

aarongable · January 21, 2021, 9:45pm

Hi, and welcome! See my earlier response on this same topic here:

petercooperjr · January 23, 2021, 1:17am

Well, assuming that openssl's logic is the same whether it's using a built-in trust store or a trust store that you provide, you should be able to set up your own test CA root and intermediates and leafs all with expiration dates to emulate the after-July expiration scenarios (that is, have the test certificate be already expired instead), and then just run openssl with that test root as the -CAfile or whatever. Shouldn't be that hard for someone familiar with openssl or something like Boulder's ceremony tool, though it's not like I've tried to do it myself yet or anything.

Nummer378 · January 23, 2021, 2:58pm

I'm currently testing this. I don't have results yet and I'm still tuning various knobs here and there. However, if someone wants to test their own client(s) I have setup a (public) test site where anyone can do some experimenting.

The test server is https://expired-root-ca-test.germancoding.com/

The server uses custom root certificates that are under my own control. The private keys for these roots are kept offline, but I cannot guarantee long lived security for these keys (I do not own a HSM). I highly recommend that you remove the root certificates from your local trust store after you're done testing.

This is the expired root certificate (simulating an expired IdenTrust/DST Root CA X3):

Expired_Root_CA_-_TEST_ROOT_CERTIFICATE.pem (1.2 KB)

And the cross signed root certificate (non-expired) is this (simulating ISRG Root X1):

Valid_Cross-Signed_Root_CA_-_TEST_ROOT_CERTIFICATE.pem (2.0 KB)

If you add the first one to your trust store, you're simulating a client which only knows DST Root X3. If you add the second one, you're simulating the case where a client knows ISRG Root X1, but the server still sends a chain chaining up to the expired IdenTrust.

Again, please do not keep these certificates in your trust store, except for testing purposes.

The test server requires TLS 1.2 or 1.3 and SNI.

petercooperjr · January 23, 2021, 9:38pm

Thank you for doing that! Since this thread was asking about Openssl specifically, I thought I'd share results from my testing using that. This tests are from AWS's CloudShell, which is basically a VM running Amazon Linux 2 and is very similar to RHEL/CentOS in how it manages packages where security patches (only) are applied but the letter at the end of the OpenSSL version doesn't get incremented.

Openssl 1.0.2k

The setup:

[cloudshell-user@ip-10-0-128-94 ~]$ sudo yum install openssl -y
[…omitting output here…]
[cloudshell-user@ip-10-0-128-94 ~]$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017
[cloudshell-user@ip-10-0-128-94 ~]$ curl -s -L -o expired-root-test.pem https://community.letsencrypt.org/uploads/short-url/rfyc7llcf0zuwSLivstXSjbj39K.pem
[cloudshell-user@ip-10-0-128-94 ~]$ curl -s -L -o cross-signed-test.pem https://community.letsencrypt.org/uploads/short-url/2GICqyDLvxNwBeMlexlgQA4nt25.pem

Having just the root in the trust store

[cloudshell-user@ip-10-0-128-94 ~]$ openssl s_client -CAfile expired-root-test.pem -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com
CONNECTED(00000003)
depth=3 O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
verify error:num=10:certificate has expired
notAfter=Jan  1 00:00:00 2021 GMT
---
Certificate chain
 0 s:/O=GermanCodings Test CA/CN=expired-root-ca-test.germancoding.com
   i:/O=GermanCodings Test CA/CN=Intermediate CA - TEST CERTIFICATE
 1 s:/O=GermanCodings Test CA/CN=Intermediate CA - TEST CERTIFICATE
   i:/O=GermanCodings Test CA/CN=Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
 2 s:/O=GermanCodings Test CA/CN=Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
   i:/O=GermanCodings Test CA/CN=Expired Root CA - TEST ROOT CERTIFICATE
[…snip…]
Verify return code: 10 (certificate has expired)

Having both root and cross-signed root in trust store

[cloudshell-user@ip-10-0-128-94 ~]$ openssl s_client -CAfile <(cat expired-root-test.pem cross-signed-test.pem) -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com
CONNECTED(00000003)
depth=3 O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
verify error:num=10:certificate has expired
notAfter=Jan  1 00:00:00 2021 GMT
---
Certificate chain
 0 s:/O=GermanCodings Test CA/CN=expired-root-ca-test.germancoding.com
   i:/O=GermanCodings Test CA/CN=Intermediate CA - TEST CERTIFICATE
 1 s:/O=GermanCodings Test CA/CN=Intermediate CA - TEST CERTIFICATE
   i:/O=GermanCodings Test CA/CN=Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
 2 s:/O=GermanCodings Test CA/CN=Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
   i:/O=GermanCodings Test CA/CN=Expired Root CA - TEST ROOT CERTIFICATE
[…snip…]
Verify return code: 10 (certificate has expired)

So yes, I think this does confirm the hypothesis that Openssl 1.0 won't trust ISRG Root X1 if a version of it cross-signed to an expired root is presented.

Openssl 1.1.1

Additional setup

[cloudshell-user@ip-10-0-128-94 ~]$ sudo yum install openssl11 -y
[…omitting output…]
[cloudshell-user@ip-10-0-128-94 ~]$ openssl11 version
OpenSSL 1.1.1c FIPS  28 May 2019

Having just the root in the trust store

[cloudshell-user@ip-10-0-128-94 ~]$ openssl11 s_client -CAfile expired-root-test.pem -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com
CONNECTED(00000003)
depth=3 O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
verify error:num=10:certificate has expired
notAfter=Jan  1 00:00:00 2021 GMT
verify return:1
depth=3 O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
notAfter=Jan  1 00:00:00 2021 GMT
verify return:1
depth=2 O = GermanCodings Test CA, CN = Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
notAfter=Jan  1 00:00:00 2025 GMT
verify return:1
depth=1 O = GermanCodings Test CA, CN = Intermediate CA - TEST CERTIFICATE
notAfter=Jan  2 00:00:00 2022 GMT
verify return:1
depth=0 O = GermanCodings Test CA, CN = expired-root-ca-test.germancoding.com
notAfter=Jan  1 00:00:00 2022 GMT
verify return:1
---
Certificate chain
 0 s:O = GermanCodings Test CA, CN = expired-root-ca-test.germancoding.com
   i:O = GermanCodings Test CA, CN = Intermediate CA - TEST CERTIFICATE
 1 s:O = GermanCodings Test CA, CN = Intermediate CA - TEST CERTIFICATE
   i:O = GermanCodings Test CA, CN = Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
 2 s:O = GermanCodings Test CA, CN = Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
   i:O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
[…snip…]
Verification error: certificate has expired

Having both root and cross-signed root in trust store

[cloudshell-user@ip-10-0-128-94 ~]$ openssl11 s_client -CAfile <(cat expired-root-test.pem cross-signed-test.pem) -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com
CONNECTED(00000003)
depth=3 O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
verify error:num=10:certificate has expired
notAfter=Jan  1 00:00:00 2021 GMT
verify return:1
depth=3 O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
notAfter=Jan  1 00:00:00 2021 GMT
verify return:1
depth=2 O = GermanCodings Test CA, CN = Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
notAfter=Jan  1 00:00:00 2025 GMT
verify return:1
depth=1 O = GermanCodings Test CA, CN = Intermediate CA - TEST CERTIFICATE
notAfter=Jan  2 00:00:00 2022 GMT
verify return:1
depth=0 O = GermanCodings Test CA, CN = expired-root-ca-test.germancoding.com
notAfter=Jan  1 00:00:00 2022 GMT
verify return:1
---
Certificate chain
 0 s:O = GermanCodings Test CA, CN = expired-root-ca-test.germancoding.com
   i:O = GermanCodings Test CA, CN = Intermediate CA - TEST CERTIFICATE
 1 s:O = GermanCodings Test CA, CN = Intermediate CA - TEST CERTIFICATE
   i:O = GermanCodings Test CA, CN = Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
 2 s:O = GermanCodings Test CA, CN = Valid Cross-Signed Root CA - TEST ROOT CERTIFICATE
   i:O = GermanCodings Test CA, CN = Expired Root CA - TEST ROOT CERTIFICATE
[…snip…]
Verify return code: 10 (certificate has expired)

Which I think means it behaves the same way as 1.0. But I may be misunderstanding the openssl s_client output and perhaps that specific testing mode doesn't reflect how real-world applications make use of the openssl libraries to do validation?

heffergm · January 25, 2021, 5:27pm

Does LetsEncrypt by chance provide a chain that simulates the end result of what the new default chain will look like? i.e. an intermediate cross signed by an expired root? I'm struggling to try and generate the same sort of thing myself, and I'd like to just plug in a chain that I know accurately simulates what the end result will be. We also have the issue of client devices that have trust stores we cannot modify, so becoming our own CA to generate a test root/intermediate isn't an option.

aarongable · January 27, 2021, 7:26pm

See our new post on this topic:

aarongable · January 27, 2021, 7:31pm

What happens if, instead of putting both the expired-root-test and cross-signed-test in the CAfile, you instead put expired-root-test and a self-signed version of the unexpired root/intermediate? That more accurately reflects the state that real trust stores will be in, and that is the circumstance under which we observed OpenSSL 1.1.x working properly during the AddTrust External CA Root expiration.

Also, make sure that your site isn't serving the expired root directly, it should only be serving the chain up through the unexpired cross-sign.

Nummer378 · January 27, 2021, 7:47pm

Thanks for the feedback!

Yes I'm aware and already did that (the highest cert offered is the cross signed root, not the expired root itself)

Yeah I didn't think that would make a difference, but yes you're correct this is how it is supposed to be. I will update my post above accordingly.

Nummer378 · January 27, 2021, 8:03pm

I've corrected the link for the second certificate. The linked certificate was previously the cross-signed version (identical to the one send by the server), but this turned out to be actually incorrect:

In the trust store, a client would store the self-signed version (e.g the actual "root version" instead of the root-as-intermediate version), but I didn't link this certificate here.

This caused issues with OpenSSL, as @petercooperjr has already noticed (OpenSSL wasn't seeing the root as root but tried to build a chain up to the expired one, even with OpenSSL 1.1+). The newly linked certificate " Valid_Cross-Signed_Root_CA_-_TEST_ROOT_CERTIFICATE.pem" does work with newer OpenSSL and is thus better suited for testing.

The new blog post from Let's Encrypt already has some data about client compatiblity, however I will keep the test server running for a while, in case anyone wants to test something else that hasn't been covered yet. I'm also running my own client tests, but I haven't finished them yet.

system · February 26, 2021, 8:03pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.