# Understanding DST Root expiry and the older default cert chain **URL:** \ **Category:** Help\ **Created:** [November 3, 2021, 8:17pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681 "2021-11-03T20:17:33Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 3, 2021, 8:17pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/1 "2021-11-03T20:17:33Z")
Hi, I am new to the certificates/ssl/letsencrypt. So please bear with me 🙂 I recently came to know about the letsencrypt's solution to "fix" the CA expiration issue with android. To test it out and see how the behavior would be I loaded the stackoverflow page (which uses letsencrypt certs) and checked the cert chain. Its using the "Alternative chain" mentioned in here [Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt](https://letsencrypt.org/2020/12/21/extending-android-compatibility.html) Then I tried to write a c++ openssl client which simply connects to the same stackoverflow page and say that its hitting the "Old default chain": ```nohighlight SSL connection using ECDHE-RSA-AES128-GCM-SHA256 non null server cert number of certs in chain is 2 0: Subject: CN = *.stackexchange.com Issuer: C = US, O = Let's Encrypt, CN = R3 Not after: Mar 3 14:00:52 2021 GMT 1: Subject: C = US, O = Let's Encrypt, CN = R3 Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3 Not after: Sep 29 19:21:40 2021 GMT cert chain is non null ERROR verifing connection 10 <---- error saying that the cert got expired ``` I ran both the tests on a fireos tablet which is based on Android 9. My question is why the openssl client not seeing the "Alternative path" and still hitting the older default chain? Is there a way to modify the openssl code to hit the new "Alternative path" The C++ code: Processing: SslClient cpp.cpp... [SslClient.txt](https://community.letsencrypt.org/uploads/short-url/hHa4tKKPBzMSefMn657h0wFQ5bO.txt) (5.4 KB) ---
**Author:** ![rg305](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/rg305/32/91314_2.png) [@rg305](https://community.letsencrypt.org/u/rg305)\ **Post date:** [November 3, 2021, 9:02pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/3 "2021-11-03T21:02:59Z")
> [@psykid](#): > > ```nohighlight > 0: Subject: CN = *.stackexchange.com > Issuer: C = US, O = Let's Encrypt, CN = R3 > Not after: Mar 3 14:00:52 2021 GMT > 1: Subject: C = US, O = Let's Encrypt, CN = R3 > Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3 > Not after: Sep 29 19:21:40 2021 GMT > > ``` They are serving the wrong chain. That chain has expired and hasn't been provided since May 2021. There is nothing you can do on the client side to fix such a mistake. ---
**Author:** ![petercooperjr](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/petercooperjr/32/84698_2.png) [@petercooperjr](https://community.letsencrypt.org/u/petercooperjr)\ **Post date:** [November 3, 2021, 9:10pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/4 "2021-11-03T21:10:40Z")
The stackoverflow pages I've just tested are actually sending the _right_ chain, though. What's the exact URL you're using where you see the R3-signed-by-DST-Root-CA-X3 chain? I wonder if there's an SNI issue, or it's different at different CDN endpoints, or something? ---
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 3, 2021, 9:17pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/5 "2021-11-03T21:17:04Z")
Just simply "[stackoverflow.com](http://stackoverflow.com)". I am taking the first ip available in the gethostbyname output: ```nohighlight struct hostent *ent = gethostbyname(host.c_str()); if (ent == NULL) { ALOGE("Unknown host %s", host.c_str()); std::cout << "unknown host " << host.c_str() << std::endl; return -1; } int mSocket = socket(AF_INET, SOCK_STREAM, 0); // MakeSocketBlocking(mSocket, false); std::cout << "socket made blocking " << ent->h_name << std::endl; struct sockaddr_in remote; // memset(remote.sin_zero, 0, sizeof(remote.sin_zero)); remote.sin_family = AF_INET; remote.sin_addr.s_addr = *(in_addr_t *)ent->h_addr_list[0]; remote.sin_port = htons(port); ``` ---
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 3, 2021, 9:20pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/6 "2021-11-03T21:20:45Z")
Also as I mentioned in the post, when I load the same page over browser, I am seeing the right certs (i.e Alternative chain) (the image is for [https://letsencrypt.org/](https://letsencrypt.org/) though) ![fos7 ISRG root x1](https://global.discourse-cdn.com/letsencrypt/original/3X/1/f/1f65f16e538946cbc743532fd5b7dc0de37f2ae8.png) ---
**Author:** ![petercooperjr](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/petercooperjr/32/84698_2.png) [@petercooperjr](https://community.letsencrypt.org/u/petercooperjr)\ **Post date:** [November 3, 2021, 9:21pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/7 "2021-11-03T21:21:06Z")
Yeah, it's an SNI issue. Sending SNI hostname: ```nohighlight [cloudshell-user@ip-10-1-146-204 ~]$ openssl11 s_client -connect stackoverflow.com:443 -servername stackoverflow.com CONNECTED(00000003) depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1 verify return:1 depth=1 C = US, O = Let's Encrypt, CN = R3 verify return:1 depth=0 CN = *.stackexchange.com verify return:1 --- Certificate chain 0 s:CN = *.stackexchange.com i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:C = US, O = Internet Security Research Group, CN = ISRG Root X1 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1 i:O = Digital Signature Trust Co., CN = DST Root CA X3 --- ``` Not sending SNI hostname: ```nohighlight [cloudshell-user@ip-10-1-146-204 ~]$ openssl11 s_client -connect stackoverflow.com:443 -noservername CONNECTED(00000003) depth=1 C = US, O = Let's Encrypt, CN = R3 verify error:num=20:unable to get local issuer certificate verify return:1 depth=0 CN = *.stackexchange.com verify error:num=10:certificate has expired notAfter=Mar 3 14:00:52 2021 GMT verify return:1 depth=0 CN = *.stackexchange.com notAfter=Mar 3 14:00:52 2021 GMT verify return:1 --- Certificate chain 0 s:CN = *.stackexchange.com i:C = US, O = Let's Encrypt, CN = R3 1 s:C = US, O = Let's Encrypt, CN = R3 i:O = Digital Signature Trust Co., CN = DST Root CA X3 --- ``` So, there are two problems: 1. StackOverflow sends a wrong certificate chain when a client isn't using SNI. 2. Your code should probably send the hostname you're trying to connect to using SNI. SNI is pretty much supported by everything nowadays. I don't know offhand how to use openssl's C API to do so (I can barely work my way around the openssl command line), but I'm sure there's a way. ---
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 3, 2021, 9:28pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/8 "2021-11-03T21:28:11Z")
Ohhh. I see. let me modify the c++ code once and see how it goes. Thanks for the prompt reply 🙂 ---
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 3, 2021, 9:35pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/9 "2021-11-03T21:35:34Z")
Yup, I see the difference now. The certs are using ISRG CA: ```nohighlight number of certs in chain is 3 0: Subject: CN = *.stackexchange.com Issuer: C = US, O = Let's Encrypt, CN = R3 Not after: Jan 2 16:19:08 2022 GMT 1: Subject: C = US, O = Let's Encrypt, CN = R3 Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1 Not after: Sep 15 16:00:00 2025 GMT 2: Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1 Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3 Not after: Sep 30 18:14:03 2024 GMT cert chain is non null ERROR verifing connection 10 ``` But still seeing the verification error 😑 ---
**Author:** ![petercooperjr](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/petercooperjr/32/84698_2.png) [@petercooperjr](https://community.letsencrypt.org/u/petercooperjr)\ **Post date:** [November 3, 2021, 9:39pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/10 "2021-11-03T21:39:27Z")
So, when using SNI, Stack Overflow is sending the current correct "default" chain, which includes a signature from the expired DST Root CA X3 for old-Android compatibility. You mentioned "alternative" chain at some points in this thread, but that's not being used here. I don't know what trust store your code is using, but you'd need to ensure that it includes ISRG Root X1, and if you're using an older OpenSSL then [you may need to use `X509_V_FLAG_TRUSTED_FIRST`](https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/) to tell it that it can ignore the extra expired signature. ---
**Author:** ![rg305](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/rg305/32/91314_2.png) [@rg305](https://community.letsencrypt.org/u/rg305)\ **Post date:** [November 3, 2021, 10:47pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/11 "2021-11-03T22:47:20Z")
> [@petercooperjr](#): > > StackOverflow sends a wrong certificate chain when a client isn't using SNI. OR when the requested name doesn't match any served name. See: `echo | openssl s_client -connect 151.101.1.69:443 -servername wrong.com | head` ---
**Author:** ![schoen](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/schoen/32/79_2.png) [@schoen](https://community.letsencrypt.org/u/schoen)\ **Post date:** [November 4, 2021, 4:02am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/12 "2021-11-04T04:02:14Z")
> [@petercooperjr](#): > > You mentioned "alternative" chain at some points in this thread, but that's not being used here. I don't know what trust store your code is using, but you'd need to ensure that it includes ISRG Root X1, and if you're using an older OpenSSL then [you may need to use `X509_V_FLAG_TRUSTED_FIRST`](https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/) to tell it that it can ignore the extra expired signature. @psykid, this advice from @petercooperjr is very important because there is disagreement between different verifiers about whether the Let's Encrypt long chain should be considered valid. While Let's Encrypt believes it is justifiable, and many clients happily accept it, there are some that don't, and they can also present an argument for their behavior. The option that @petercooperjr mentions changes this behavior in the TLS library that your code is calling locally. (If you're using a different TLS client library than OpenSSL, there might be a different option whose behavior corresponds to this one.) ---
**Author:** ![webprofusion](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/webprofusion/32/85310_2.png) [@webprofusion](https://community.letsencrypt.org/u/webprofusion)\ **Post date:** [November 4, 2021, 4:17am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/13 "2021-11-04T04:17:06Z")
The discrepancy will be in their varnish cache server config but as the initial check doesn't use SNI it possibly quite rightly serves the legacy chain (which is Let's Encrypt default chain), assuming that if you don't speak SNI then you're at the bottom of the TLS food chain for everything else as well. You mentioned how your computer (Windows) sees the chain, and that's a very different topic. Each client (including OpenSSL) decides how they will arrive at a verified chain and the results you're seeing are not the pure TLS conversation as it appears, just how it was interpreted. Are you specifically interested in Windows server chains or just linux servers? ---
**Author:** ![webprofusion](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/webprofusion/32/85310_2.png) [@webprofusion](https://community.letsencrypt.org/u/webprofusion)\ **Post date:** [November 4, 2021, 4:24am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/14 "2021-11-04T04:24:53Z")
As an aside, for reference here is the source for the (very basic) chain reader used by [https://chainchecker.certifytheweb.com/](https://chainchecker.certifytheweb.com/) (this version is an Azure function written in Go and just relies on the built-in Go tls stuff) [handler.go](https://community.letsencrypt.org/uploads/short-url/pvfXT7SoE4BBZrtwqFyeP8F1pUA.go) (7.4 KB) ---
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 4, 2021, 4:53am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/15 "2021-11-04T04:53:57Z")
> If you're using an older OpenSSL then you may need to use X509\_V\_FLAG\_TRUSTED\_FIRST to tell it that it can ignore the extra expired signature. Thanks for the pointers. I have tried to use the X509\_V\_FLAG\_TRUSTED\_FIRST on my system but still seeing the same "expired cert" error . FWIW, I have both the ISRG and Dst in my trust store on the android device. Since I am on android 9, I think the openssl version is 1.1(not sure though). > Are you are specifically interested in Windows server chains or just linux servers? Well im on the client side mostly, so both windows and linux servers need to be working ---
**Author:** ![sahsanu](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/sahsanu/32/89984_2.png) [@sahsanu](https://community.letsencrypt.org/u/sahsanu)\ **Post date:** [November 4, 2021, 5:33am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/16 "2021-11-04T05:33:38Z")
> [@webprofusion](#): > > you're at the bottom of the TLS food chain I need a t-shirt with this 😃 ---
**Author:** ![rg305](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/rg305/32/91314_2.png) [@rg305](https://community.letsencrypt.org/u/rg305)\ **Post date:** [November 4, 2021, 7:08am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/17 "2021-11-04T07:08:48Z")
# ^ Quote(s) of the day! ^ ---
**Author:** ![petercooperjr](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/petercooperjr/32/84698_2.png) [@petercooperjr](https://community.letsencrypt.org/u/petercooperjr)\ **Post date:** [November 4, 2021, 1:21pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/18 "2021-11-04T13:21:48Z")
> [@webprofusion](#): > > as the initial check doesn't use SNI it possibly quite rightly serves the legacy chain (which is Let's Encrypt default chain) No, they're serving the R3-signed-by-DST-Root-CA-X3 intermediate when not using SNI, which isn't the default chain and isn't really valid anymore. The default chain (used by StackOverflow if you do send SNI) sends the R3-signed-by-ISRG-Root-X1 plus ISRG-Root-X1-signed-by-DST-Root-CA-X3. ---
**Author:** ![rmbolger](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/rmbolger/32/27474_2.png) [@rmbolger](https://community.letsencrypt.org/u/rmbolger)\ **Post date:** [November 4, 2021, 3:47pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/19 "2021-11-04T15:47:36Z")
> [@petercooperjr](#): > > No, they're serving the R3-signed-by-DST-Root-CA-X3 intermediate when not using SNI, which isn't the default chain and isn't really valid anymore. I actually reached out to one of the folks on their SRE team about this to get it fixed. I'm guessing it's a simple oversight in their monitoring toolchain that they'll be happy to fix and fix monitoring for. Both the cert and the copy of R3 they're sending are expired. ---
**Author:** ![webprofusion](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/webprofusion/32/85310_2.png) [@webprofusion](https://community.letsencrypt.org/u/webprofusion)\ **Post date:** [November 4, 2021, 11:57pm UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/20 "2021-11-04T23:57:24Z")
Thanks Peter, yes I totally mis-read that. ---
**Author:** ![psykid](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/psykid/32/55308_2.png) [@psykid](https://community.letsencrypt.org/u/psykid)\ **Post date:** [November 5, 2021, 7:58am UTC](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681/21 "2021-11-05T07:58:57Z")
BTW I am on android system and the default ssl impl is boring SSL. The version values: SSLeay\_version(SSLEAY\_VERSION): "BoringSSL" OPENSSL\_VERSION\_NUMBER: 269484159 OPENSSL\_VERSION\_TEXT: OpenSSL 1.1.0 (compatible; BoringSSL) After adding the SNI in before SSL\_connect, the ubuntu box is able to verify the certs correctly but the android is still showing the issues. Not sure why android is having the problem even if the certs are all having valid NotAfter values. [Next page](https://community.letsencrypt.org/t/understanding-dst-root-expiry-and-the-older-default-cert-chain/164681.md?page=2)