Understanding DST Root expiry and the older default cert chain

psykid · November 3, 2021, 8:17pm

Hi, I am new to the certificates/ssl/letsencrypt. So please bear with me

I recently came to know about the letsencrypt's solution to "fix" the CA expiration issue with android. To test it out and see how the behavior would be I loaded the stackoverflow page (which uses letsencrypt certs) and checked the cert chain. Its using the "Alternative chain" mentioned in here Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt

Then I tried to write a c++ openssl client which simply connects to the same stackoverflow page and say that its hitting the "Old default chain":

SSL connection using ECDHE-RSA-AES128-GCM-SHA256
non null server cert
number of certs in chain is 2
0: Subject: CN = *.stackexchange.com
   Issuer:  C = US, O = Let's Encrypt, CN = R3
   Not after:    Mar  3 14:00:52 2021 GMT
1: Subject: C = US, O = Let's Encrypt, CN = R3
   Issuer:  O = Digital Signature Trust Co., CN = DST Root CA X3
   Not after:    Sep 29 19:21:40 2021 GMT
cert chain is non null
ERROR verifing connection 10   <---- error saying that the cert got expired

I ran both the tests on a fireos tablet which is based on Android 9.

My question is why the openssl client not seeing the "Alternative path" and still hitting the older default chain? Is there a way to modify the openssl code to hit the new "Alternative path"

The C++ code:
Processing: SslClient cpp.cpp...
SslClient.txt (5.4 KB)

rg305 · November 3, 2021, 9:02pm

psykid:

0: Subject: CN = *.stackexchange.com
   Issuer:  C = US, O = Let's Encrypt, CN = R3
   Not after:    Mar  3 14:00:52 2021 GMT
1: Subject: C = US, O = Let's Encrypt, CN = R3
   Issuer:  O = Digital Signature Trust Co., CN = DST Root CA X3
   Not after:    Sep 29 19:21:40 2021 GMT

They are serving the wrong chain.
That chain has expired and hasn't been provided since May 2021.
There is nothing you can do on the client side to fix such a mistake.

petercooperjr · November 3, 2021, 9:10pm

The stackoverflow pages I've just tested are actually sending the right chain, though. What's the exact URL you're using where you see the R3-signed-by-DST-Root-CA-X3 chain? I wonder if there's an SNI issue, or it's different at different CDN endpoints, or something?

psykid · November 3, 2021, 9:17pm

Just simply "stackoverflow.com". I am taking the first ip available in the gethostbyname output:

struct hostent *ent = gethostbyname(host.c_str());

    if (ent == NULL) {
        ALOGE("Unknown host %s", host.c_str());
        std::cout << "unknown host " << host.c_str() << std::endl;
        return -1;
    }
    int mSocket = socket(AF_INET, SOCK_STREAM, 0);

    // MakeSocketBlocking(mSocket, false);
    std::cout << "socket made blocking " << ent->h_name << std::endl;

    struct sockaddr_in remote;
    // memset(remote.sin_zero, 0, sizeof(remote.sin_zero));
    remote.sin_family = AF_INET;
    remote.sin_addr.s_addr = *(in_addr_t *)ent->h_addr_list[0];
    remote.sin_port = htons(port);

psykid · November 3, 2021, 9:20pm

Also as I mentioned in the post, when I load the same page over browser, I am seeing the right certs (i.e Alternative chain) (the image is for https://letsencrypt.org/ though)

petercooperjr · November 3, 2021, 9:21pm

Yeah, it's an SNI issue.

Sending SNI hostname:

[cloudshell-user@ip-10-1-146-204 ~]$ openssl11 s_client -connect stackoverflow.com:443 -servername stackoverflow.com
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = *.stackexchange.com
verify return:1
---
Certificate chain
 0 s:CN = *.stackexchange.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

Not sending SNI hostname:

[cloudshell-user@ip-10-1-146-204 ~]$ openssl11 s_client -connect stackoverflow.com:443 -noservername
CONNECTED(00000003)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = *.stackexchange.com
verify error:num=10:certificate has expired
notAfter=Mar  3 14:00:52 2021 GMT
verify return:1
depth=0 CN = *.stackexchange.com
notAfter=Mar  3 14:00:52 2021 GMT
verify return:1
---
Certificate chain
 0 s:CN = *.stackexchange.com
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

So, there are two problems:

StackOverflow sends a wrong certificate chain when a client isn't using SNI.
Your code should probably send the hostname you're trying to connect to using SNI.

SNI is pretty much supported by everything nowadays. I don't know offhand how to use openssl's C API to do so (I can barely work my way around the openssl command line), but I'm sure there's a way.

psykid · November 3, 2021, 9:28pm

Ohhh. I see. let me modify the c++ code once and see how it goes. Thanks for the prompt reply

psykid · November 3, 2021, 9:35pm

Yup, I see the difference now. The certs are using ISRG CA:

number of certs in chain is 3
0: Subject: CN = *.stackexchange.com
   Issuer:  C = US, O = Let's Encrypt, CN = R3
   Not after:    Jan  2 16:19:08 2022 GMT
1: Subject: C = US, O = Let's Encrypt, CN = R3
   Issuer:  C = US, O = Internet Security Research Group, CN = ISRG Root X1
   Not after:    Sep 15 16:00:00 2025 GMT
2: Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
   Issuer:  O = Digital Signature Trust Co., CN = DST Root CA X3
   Not after:    Sep 30 18:14:03 2024 GMT
cert chain is non null
ERROR verifing connection 10

But still seeing the verification error

petercooperjr · November 3, 2021, 9:39pm

So, when using SNI, Stack Overflow is sending the current correct "default" chain, which includes a signature from the expired DST Root CA X3 for old-Android compatibility. You mentioned "alternative" chain at some points in this thread, but that's not being used here. I don't know what trust store your code is using, but you'd need to ensure that it includes ISRG Root X1, and if you're using an older OpenSSL then you may need to use X509_V_FLAG_TRUSTED_FIRST to tell it that it can ignore the extra expired signature.

rg305 · November 3, 2021, 10:47pm

OR when the requested name doesn't match any served name.

See:
echo | openssl s_client -connect 151.101.1.69:443 -servername wrong.com | head

schoen · November 4, 2021, 4:02am

@psykid, this advice from @petercooperjr is very important because there is disagreement between different verifiers about whether the Let's Encrypt long chain should be considered valid. While Let's Encrypt believes it is justifiable, and many clients happily accept it, there are some that don't, and they can also present an argument for their behavior. The option that @petercooperjr mentions changes this behavior in the TLS library that your code is calling locally. (If you're using a different TLS client library than OpenSSL, there might be a different option whose behavior corresponds to this one.)

webprofusion · November 4, 2021, 4:17am

The discrepancy will be in their varnish cache server config but as the initial check doesn't use SNI it possibly quite rightly serves the legacy chain (which is Let's Encrypt default chain), assuming that if you don't speak SNI then you're at the bottom of the TLS food chain for everything else as well.

You mentioned how your computer (Windows) sees the chain, and that's a very different topic. Each client (including OpenSSL) decides how they will arrive at a verified chain and the results you're seeing are not the pure TLS conversation as it appears, just how it was interpreted.

Are you specifically interested in Windows server chains or just linux servers?

webprofusion · November 4, 2021, 4:24am

As an aside, for reference here is the source for the (very basic) chain reader used by https://chainchecker.certifytheweb.com/ (this version is an Azure function written in Go and just relies on the built-in Go tls stuff)

handler.go (7.4 KB)

psykid · November 4, 2021, 4:53am

If you're using an older OpenSSL then you may need to use X509_V_FLAG_TRUSTED_FIRST to tell it that it can ignore the extra expired signature.

Thanks for the pointers. I have tried to use the X509_V_FLAG_TRUSTED_FIRST on my system but still seeing the same "expired cert" error . FWIW, I have both the ISRG and Dst in my trust store on the android device. Since I am on android 9, I think the openssl version is 1.1(not sure though).

Are you are specifically interested in Windows server chains or just linux servers?

Well im on the client side mostly, so both windows and linux servers need to be working

sahsanu · November 4, 2021, 5:33am

I need a t-shirt with this

rg305 · November 4, 2021, 7:08am

^ Quote(s) of the day! ^

petercooperjr · November 4, 2021, 1:21pm

No, they're serving the R3-signed-by-DST-Root-CA-X3 intermediate when not using SNI, which isn't the default chain and isn't really valid anymore. The default chain (used by StackOverflow if you do send SNI) sends the R3-signed-by-ISRG-Root-X1 plus ISRG-Root-X1-signed-by-DST-Root-CA-X3.

rmbolger · November 4, 2021, 3:47pm

I actually reached out to one of the folks on their SRE team about this to get it fixed. I'm guessing it's a simple oversight in their monitoring toolchain that they'll be happy to fix and fix monitoring for. Both the cert and the copy of R3 they're sending are expired.

webprofusion · November 4, 2021, 11:57pm

Thanks Peter, yes I totally mis-read that.

psykid · November 5, 2021, 7:58am

BTW I am on android system and the default ssl impl is boring SSL. The version values:
SSLeay_version(SSLEAY_VERSION): "BoringSSL"
OPENSSL_VERSION_NUMBER: 269484159
OPENSSL_VERSION_TEXT: OpenSSL 1.1.0 (compatible; BoringSSL)

After adding the SNI in before SSL_connect, the ubuntu box is able to verify the certs correctly but the android is still showing the issues. Not sure why android is having the problem even if the certs are all having valid NotAfter values.