Choice of default long chain vs short chain

I’ve noticed that the cert chain that LE’s ACME interface returns yields a different chain from what is gotten by following the CA Issuers chain.

For example, I recently issued for felipegasper.com. The CA chain I got back had 2 intermediates:

        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
-----
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

But, if you follow the CA Issuers from the leaf certificate:

        Subject: CN = felipegasper.com
                CA Issuers - URI:http://r3.i.lencr.org/
-----
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
                CA Issuers - URI:http://x1.i.lencr.org/
-----
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

So, the API gave me back an “ISRG Root X1” that isn’t self-signed, whereas the CA Issuers chain gives me an “ISRG Root X1” that is self-signed.

Of note: CentOS 7’s OpenSSL validates the chain from the CA Issuers but rejects the chain from LE’s API.

@FGasper
I get a completely different cert:

---
Certificate chain
 0 s:/CN=felipegasper.com
   i:/C=US/ST=TX/L=Houston/O=cPanel, Inc./CN=cPanel, Inc. Certification Authority
 1 s:/C=US/ST=TX/L=Houston/O=cPanel, Inc./CN=cPanel, Inc. Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
---

As for:

There is what the server provides and there is what the client choses to use.
In that particular case, there are three possible trust paths and they didn't all make the same choice.

@rg305 I didn’t install the LE cert that I provisioned; I just fetched it to test.

1 Like

@FGasper
Can you show the fullchain.pem file?

@rg305 I don’t have such a file. I used Perl’s Net::ACME2 to provision the cert.

1 Like

Dang it! LOL
Well it should have a chain file in there somewhere.
Which should show you what the "server" would serve.
It is however only a hint to the client - which can always do whatever it was programmed to do.
If left to it's own devices, the client could pick one of three possible paths.

@rg305 OpenSSL verification, though, will use the certs that it’s given directly. The cert chain that LE’s API provides fails that verification on CentOS 7, whereas the chain gotten from following CA Issuers URLs passes it.

Is that disparity intended? Which is intended as the “primary” chain? Right now I can’t see how I’d intuit whether the publicly-inferrable chain or the one LE gave me directly would suit any specific use case better than the other. Is one meant to be more compatible or some such?

The API is returning the chain that we believe to be the most compatible with aging implementations which have not been able to update their root stores. <leaf> issued by R3, R3 issued by X1, and X1 issued by expired DST Root CA X3. If compatibility with older root stores is not necessary, it is fine to chain instead to X1 issued by X1, as published at the AIA URI.

What sort of chain validation on CentOS 7 is failing? Is it due to outdated OpenSSL as discussed in this thread and others?

2 Likes

Is it due to outdated OpenSSL as discussed in this thread and others?

I believe so, yes. The weird part, though, is that CentOS 7 is fine with the self-signed X1. So the “most compatible with aging implementations” one is actually incompatible with a widely-deployed, supported OS.

The weird part is that on Sep 14 there was an update to CentOS 7’s root certs. I would have thought that would fix this.

Right now I’m looking at configuring OpenSSL as discussed in OpenSSL’s workaround #2.

1 Like

I think that update deleted the decommissioned DST Root CA X3, which helps the issue, but doesn't solve the OpenSSL problem.

It is possible with the API to request the shorter chain. Certbot and many other ACME clients have implemented this, and it sounds like this would be a great option on CentOS
7 where X1 is directly trusted.

2 Likes

Looking at this again (vesperal domesticities now fulfilled).

LE’s API sends 3 certs, the last of which is an intermediate signed by DST Root CA X3. Isn’t that the one that’s now expired?

Which devices are compatible with that cert chain versus ISRG Root X1?

Hi @FGasper ! I think this post has all the answers you're looking for: Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt.

4 Likes

OK. If I’m understanding this properly, the API-default chain is as it is in order to capitalize on a quirk of Android’s trust logic whereby it doesn’t actually care when its roots are expired.

So anything that uses OpenSSL, or whatever other library that cares whether its roots are expired, has to update its configuration to use the shorter chain—which appears basically to mean tossing away the “faux” ISRG Root X1.

I should note that OpenSSL 1.1.1j appears to fail this verification as well, so it’s not just limited to older OpenSSLs.

So basically, the default cert chain that LE gives with cert orders is going to fail every verification logic other than Android?

1 Like

That's the gist of it, though rather than "quirk" I would say this is an intentional design decision in Android, to allow smoother root transitions (like this one). It's something that rarely comes up, and many other validation libraries don't make that same choice, so it feels rare - but it's a legit design decision. One way to think of it is that OSes store "trust anchors", which are not really the same as root certificates but are often expressed that way for convenience.

Close, though I would say tossing away the cross-signed ISRG Root X1. It's not "faux" at all - just unnecessary on platforms that already have ISRG Root X1 in their trust store. And clarification on OpenSSL versions below.

If you're experiencing this symptom, it most likely means that ISRG Root X1 is not in your trust store, and you should update it.

3 Likes

If you're experiencing this system, it most likely means that ISRG Root X1 is not in your trust store, and you should update it.

No, it’s definitely there; if I throw away the non-self-signed ISRG Root X1 then the verification succeeds.

You can see this yourself:

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIEeDCCA2CgAwIBAgISBOPHpMeoUsomNXiyfY5TuGcVMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MzAxNzQzMjlaFw0yMTEyMjkxNzQzMjhaMBsxGTAXBgNVBAMT
EGZlbGlwZWdhc3Blci5jb20wdjAQBgcqhkjOPQIBBgUrgQQAIgNiAARejOiIEMMQ
A+3GzSo+KdLnRDmK0kF0hrDphuR9bY3HiE851c9TiLjsX4T8bkm2oiXFbeepnbAq
SIp1aFX5E+e/JRwGFtTMPES/EwCjBShQZThoVAMo0zjskfcdsC56aFCjggJLMIIC
RzAOBgNVHQ8BAf8EBAMCB4AwHQYDVR0lBBYwFAYIKwYBBQUHAwEGCCsGAQUFBwMC
MAwGA1UdEwEB/wQCMAAwHQYDVR0OBBYEFKZAWStyIV8LrdjLLmub5qbMWwjpMB8G
A1UdIwQYMBaAFBQusxe3WFbLrlAJQOYfr52LFMLGMFUGCCsGAQUFBwEBBEkwRzAh
BggrBgEFBQcwAYYVaHR0cDovL3IzLm8ubGVuY3Iub3JnMCIGCCsGAQUFBzAChhZo
dHRwOi8vcjMuaS5sZW5jci5vcmcvMBsGA1UdEQQUMBKCEGZlbGlwZWdhc3Blci5j
b20wTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAmBggrBgEF
BQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEEAdZ5AgQC
BIH1BIHyAPAAdgBElGUusO7Or8RAB9io/ijA2uaCvtjLMbU/0zOWtbaBqAAAAXw4
BMfdAAAEAwBHMEUCIQC8142jw2SF5LaABl0vvwj3WRUh1Zy8wv3/AeRE3NOtywIg
NSsHToTQzboCodUNKRw6oDG9QRzz7sNxa/PURYrC8IEAdgB9PvL4j/+IVWgkwsDK
nlKJeSvFDngJfy5ql2iZfiLw1wAAAXw4BMf/AAAEAwBHMEUCIQD/1o45eq5Ifoe/
ZBSRckr7oBABbJ4h4V0pyCtGlMvMFQIgXtkcj86wfxz4Biw5rXVoNUuNL3HMq7wu
3VicpHQTcCgwDQYJKoZIhvcNAQELBQADggEBAI0Lkb05wwlL3mk5vdyKX4fL2S0K
cih854oD0FjfWTLA4kZyUepjocF2Lan0yOvhUB/Mj4rZD2lh4qrARQB9WQ28ry9r
tDLKCzSqfOOxGbmV1clwwGt98s96lkq3aGjS39m5+6XwNpJnPzV+qOlMlGmcGezU
g6Ks+dyinLPhWu3iP5BLFzuRZ7sYTBGtb7ILPYLkhbAK69kxPruXS+fJoCxdSV07
I+Buk14mBGqnTKGqM2mpWdzZGJqbg1z+JU68R46eiAFSfOJ0xqdYBq8DWwUB9jwZ
MykiedsTBR90es+P8mDh9olXrWFmn8w0si4X7bOYGWlJc3ztnqivjSsexuc=
-----END CERTIFICATE-----

First try it w/ all three certs, then delete the first one and retry.

Can you share the output of openssl version and the command you are running to validate?

OpenSSL 1.1.1j  16 Feb 2021

The command is openssl verify felipegasper.com.chain.

I'm not sure of the ins and outs of how openssl verify is supposed to work. What we know and have tested is that OpenSSL 1.1.0+ works when you use it to validate a TLS connection. For instance, if you have curl compiled against OpenSSL, can you curl https://letsencrypt.org? If so, you have a working system with OpenSSL.

1 Like

"delete the first one"
Shouldn't that be:
delete the last one
OR
are you counting upwards?

I literally had this same (type) discussion with my wife on the phone earlier today...
Open the cabinet door
OK
It's on the first shelf
Is "first" the top self? or the bottom shelf?
...argument ensues...
LOL

1 Like