I actually reached out to one of the folks on their SRE team about this to get it fixed. I'm guessing it's a simple oversight in their monitoring toolchain that they'll be happy to fix and fix monitoring for. Both the cert and the copy of R3 they're sending are expired.
8 Likes
Thanks Peter, yes I totally mis-read that.
3 Likes
BTW I am on android system and the default ssl impl is boring SSL. The version values:
SSLeay_version(SSLEAY_VERSION): "BoringSSL"
OPENSSL_VERSION_NUMBER: 269484159
OPENSSL_VERSION_TEXT: OpenSSL 1.1.0 (compatible; BoringSSL)
After adding the SNI in before SSL_connect, the ubuntu box is able to verify the certs correctly but the android is still showing the issues. Not sure why android is having the problem even if the certs are all having valid NotAfter values.
3 Likes
I am outside my zone of expertise with your s/w combination but are you sure this is correct for Android? From your code:
#define CERT_DIRECTORY "/etc/ssl/certs/"
SSL_CTX_load_verify_locations(ctx, NULL, CERT_DIRECTORY);
Can you verify any domain with this client on Android? What about one like:
https://www.digicert.com/
3 Likes
Tested google.com and it worked fine, will test this site as well in a bit.
Edit: digicert.com works as expected:
number of certs in chain is 2
0: Subject: businessCategory = Private Organization, 1.3.6.1.4.1.311.60.2.1.3 = US, 1.3.6.1.4.1.311.60.2.1.2 = Utah, serialNumber = 5299537-0142, C = US, ST = Utah, L = Lehi, O = "DigiCert, Inc.", CN = www.digicert.com
Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
Not after: May 4 23:59:59 2022 GMT
1: Subject: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert SHA2 Extended Validation Server CA
Issuer: C = US, O = DigiCert Inc, OU = www.digicert.com, CN = DigiCert High Assurance EV Root CA
Not after: Oct 22 12:00:00 2028 GMT
cert chain is non null
hostname verification success 1
verify result passed
completed non blocking connect
end main
4 Likes
@psykid Interesting. Oh well. You might also try
https://acme-v02.api.letsencrypt.org/
It sends the "short chain" ending in ISRG Root X1 whereas the LE public websites (like this one) use the "long chain". Might be useful debug info - maybe not to me - but someone 
3 Likes
The standard test sites I'd recommend are https://helloworld.letsencrypt.org/ for the "default" chain and https://valid-isrgrootx1.letsencrypt.org/ for the "alternate" chain.
4 Likes
Sure. With a friendly caveat that the Hello World page is, um, dated.
Says LE is a "new" CA and links to announce page from 2015 but it works
4 Likes
Oh sure. It's just officially "from" Let's Encrypt, has been using the default chain all along, and is a short simple page. The valid-isrgrootx1 is an Official Test Page that's required to be up-to-date, and while the helloworld site may not be Official, it seems to work well enough for me.
3 Likes
Output for valid-isgrrotx1:
adb shell rtsptest valid-isrgrootx1.letsencrypt.org
start main 2
url and host set
socket made blocking origin.letsencrypt.org
remote done 0
convert socket to secure 0xe876d008
Using OpenSSL version "BoringSSL" vs 269484159 vs OpenSSL 1.1.0 (compatible; BoringSSL)
socket converted to secure
starting non blocking connect
mssl is null? false
ssl connect result is 1
ssl connect error is 0
after while loop
non null server cert
number of certs in chain is 2
0: Subject: CN = valid-isrgrootx1.letsencrypt.org
Issuer: C = US, O = Let's Encrypt, CN = R3
Not after: Jan 4 15:00:21 2022 GMT
1: Subject: C = US, O = Let's Encrypt, CN = R3
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Not after: cert chain is non null
Sep 15 16:00:00 2025 GMT
hostname verification success 1
verify result passed
completed non blocking connect
end main
Output for helloworld:
adb shell rtsptest helloworld.letsencrypt.org
start main 2
url and host set
socket made blocking origin.letsencrypt.org
remote done 0
convert socket to secure 0xeb5ed008
Using OpenSSL version "BoringSSL" vs 269484159 vs OpenSSL 1.1.0 (compatible; BoringSSL)
socket converted to secure
starting non blocking connect
mssl is null? false
ssl connect result is 1
ssl connect error is 0
after while loop
non null server cert
number of certs in chain is 3
0: Subject: CN = helloworld.letsencrypt.org
Issuer: C = US, O = Let's Encrypt, CN = R3
Not after: Jan 25 15:00:21 2022 GMT
1: Subject: C = US, O = Let's Encrypt, CN = R3
Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Not after: Sep 15 16:00:00 2025 GMT
2: Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1
Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
Not after: Sep 30 18:14:03 2024 GMT
cert chain is non null
hostname verification success 1
ERROR verifing connection 10
error while nonblocking connect 10
Is it possible that my cross signed root CA certs are invalid on my android device?
1 Like
Also I am observing that even though my ubuntu box only has self signed isgrootx1 certificate mentioned in this website: Chain of Trust - Let's Encrypt, Its able to verify the cross signed certs without issues while android couldn't.
Is it expected to have the https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem cert as well in the system to be able to validate the "new default chain" ? If so how can ubuntu work while android can't?
1 Like
can I ask what version of android you used? entire purpose of that long chain is to make older andorid happy, which they don't have ISRG root in their trust store but they don't care about expiration of root certificates. it opened a new can of worm of clients that climb the given chain onto top and start verifying that chain.
3 Likes
I am using FireOs 7 device (should be based on android pie).
1 Like
Yeah I was reading that android doesnt care about the expiration dates in many external websites as well as LetsEncrypt's docs. Do we have any links to the code/commits to support that?
I assume this behavior is implemented by BoringSSL rather than android itself?
1 Like
I suspect amazon brought different ssl stack into thier for what you saying?. if so I'm not sure what we can do about that: (unless import some trickey that ceriticate for Dst root x3's key with some other self-signed cert into your trust store. but I'm don't know how to do such thing.
2 Likes
Dont have a rooted android device with me, else would've tested in that itself. FWIW, I don't think Amazon made any such changes to trust store (at least I can see the same pem files content for DST root and ISRG X11 certs).
Also as expected the DST root cert has these dates on android:
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
Signature Algorithm: sha1WithRSAEncryption
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
Not Before: Sep 30 21:12:19 2000 GMT
Not After : Sep 30 14:01:15 2021 GMT
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
1 Like
Update: removing the DST root cert (2e5ac55d.0) file from /etc/security/cacerts-sha-1/ folder did the trick. However I am not sure how this works or even if this is a valid fix.
After all this I still have bunch of doubts:
Info I already know
- Lets encrypt is providing the "New default chain" so that they can be backward compatible with the older devices.
- There is also a "Alternative chain" for devices which are meant to be for the client which have ISRG root CA installed already.
- Having "New Default Chain" works because Android intentionally does not enforce the expiration dates of certificates used as trust anchors. In our case its DST Root cert.
- IdenTrust has agreed to issue a 3-year cross-sign for our ISRG Root X1 from their DST Root CA X3.
Queries:
- How do servers know which chain to vend (I am assuming they can vend whatever chain they want)? Or is it on the client side to ask for specific chain?
- If its on the client side to ask for the type of chain (which I don't think it is), is there a sample c/c++ client code?
- If the servers only vend "New default chain", how would non-android devices work (since #3 above applies only for android).
- In the "New default chain", since the root CA is still DST Root (which already expired), will calling SSL_get_verify_result on such cert cause errors? (specifically cert expired errors).
- How is removing the DST root cert from the device is working? How can the system validate the cert without the root with which it got signed? (fyi, I used SSL_CTX_load_verify_locations and gave the /etc/security/cacerts-sha1 as the ca certs folder, so even if there is another file in the system I dont think it should matter).
1 Like
Also see:
Previously posted by petercooperjr but related to Q #4
https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/
4 Likes
This is great thanks, it explains all my queries.
The https://www.openssl.org link was mentioned multiple times here but some how I always skipped the first solution 
Thanks for the responses and apologies for dragging this more time than required.
3 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.