Choice of default long chain vs short chain

FGasper · September 30, 2021, 6:55pm

I’ve noticed that the cert chain that LE’s ACME interface returns yields a different chain from what is gotten by following the CA Issuers chain.

For example, I recently issued for felipegasper.com. The CA chain I got back had 2 intermediates:

        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
-----
        Issuer: O = Digital Signature Trust Co., CN = DST Root CA X3
        Validity
            Not Before: Jan 20 19:14:03 2021 GMT
            Not After : Sep 30 18:14:03 2024 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

But, if you follow the CA Issuers from the leaf certificate:

        Subject: CN = felipegasper.com
                CA Issuers - URI:http://r3.i.lencr.org/
-----
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
                CA Issuers - URI:http://x1.i.lencr.org/
-----
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Jun  4 11:04:38 2015 GMT
            Not After : Jun  4 11:04:38 2035 GMT
        Subject: C = US, O = Internet Security Research Group, CN = ISRG Root X1

So, the API gave me back an “ISRG Root X1” that isn’t self-signed, whereas the CA Issuers chain gives me an “ISRG Root X1” that is self-signed.

Of note: CentOS 7’s OpenSSL validates the chain from the CA Issuers but rejects the chain from LE’s API.

rg305 · September 30, 2021, 7:00pm

@FGasper
I get a completely different cert:

---
Certificate chain
 0 s:/CN=felipegasper.com
   i:/C=US/ST=TX/L=Houston/O=cPanel, Inc./CN=cPanel, Inc. Certification Authority
 1 s:/C=US/ST=TX/L=Houston/O=cPanel, Inc./CN=cPanel, Inc. Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
 2 s:/C=GB/ST=Greater Manchester/L=Salford/O=COMODO CA Limited/CN=COMODO RSA Certification Authority
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=AAA Certificate Services
---

As for:

There is what the server provides and there is what the client choses to use.
In that particular case, there are three possible trust paths and they didn't all make the same choice.

FGasper · September 30, 2021, 7:02pm

@rg305 I didn’t install the LE cert that I provisioned; I just fetched it to test.

rg305 · September 30, 2021, 7:06pm

@FGasper
Can you show the fullchain.pem file?

FGasper · September 30, 2021, 7:08pm

@rg305 I don’t have such a file. I used Perl’s Net::ACME2 to provision the cert.

rg305 · September 30, 2021, 7:11pm

Dang it! LOL
Well it should have a chain file in there somewhere.
Which should show you what the "server" would serve.
It is however only a hint to the client - which can always do whatever it was programmed to do.
If left to it's own devices, the client could pick one of three possible paths.

FGasper · September 30, 2021, 7:13pm

@rg305 OpenSSL verification, though, will use the certs that it’s given directly. The cert chain that LE’s API provides fails that verification on CentOS 7, whereas the chain gotten from following CA Issuers URLs passes it.

Is that disparity intended? Which is intended as the “primary” chain? Right now I can’t see how I’d intuit whether the publicly-inferrable chain or the one LE gave me directly would suit any specific use case better than the other. Is one meant to be more compatible or some such?

ezekiel · September 30, 2021, 8:13pm

The API is returning the chain that we believe to be the most compatible with aging implementations which have not been able to update their root stores. <leaf> issued by R3, R3 issued by X1, and X1 issued by expired DST Root CA X3. If compatibility with older root stores is not necessary, it is fine to chain instead to X1 issued by X1, as published at the AIA URI.

What sort of chain validation on CentOS 7 is failing? Is it due to outdated OpenSSL as discussed in this thread and others?

FGasper · September 30, 2021, 8:28pm

Is it due to outdated OpenSSL as discussed in this thread and others?

I believe so, yes. The weird part, though, is that CentOS 7 is fine with the self-signed X1. So the “most compatible with aging implementations” one is actually incompatible with a widely-deployed, supported OS.

The weird part is that on Sep 14 there was an update to CentOS 7’s root certs. I would have thought that would fix this.

Right now I’m looking at configuring OpenSSL as discussed in OpenSSL’s workaround #2.

ezekiel · September 30, 2021, 8:34pm

I think that update deleted the decommissioned DST Root CA X3, which helps the issue, but doesn't solve the OpenSSL problem.

It is possible with the API to request the shorter chain. Certbot and many other ACME clients have implemented this, and it sounds like this would be a great option on CentOS
7 where X1 is directly trusted.

FGasper · October 1, 2021, 12:35am

Looking at this again (vesperal domesticities now fulfilled).

LE’s API sends 3 certs, the last of which is an intermediate signed by DST Root CA X3. Isn’t that the one that’s now expired?

Which devices are compatible with that cert chain versus ISRG Root X1?

jsha · October 1, 2021, 12:42am

Hi @FGasper ! I think this post has all the answers you're looking for: Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt.

FGasper · October 1, 2021, 1:06am

OK. If I’m understanding this properly, the API-default chain is as it is in order to capitalize on a quirk of Android’s trust logic whereby it doesn’t actually care when its roots are expired.

So anything that uses OpenSSL, or whatever other library that cares whether its roots are expired, has to update its configuration to use the shorter chain—which appears basically to mean tossing away the “faux” ISRG Root X1.

I should note that OpenSSL 1.1.1j appears to fail this verification as well, so it’s not just limited to older OpenSSLs.

So basically, the default cert chain that LE gives with cert orders is going to fail every verification logic other than Android?

jsha · October 1, 2021, 1:10am

That's the gist of it, though rather than "quirk" I would say this is an intentional design decision in Android, to allow smoother root transitions (like this one). It's something that rarely comes up, and many other validation libraries don't make that same choice, so it feels rare - but it's a legit design decision. One way to think of it is that OSes store "trust anchors", which are not really the same as root certificates but are often expressed that way for convenience.

Close, though I would say tossing away the cross-signed ISRG Root X1. It's not "faux" at all - just unnecessary on platforms that already have ISRG Root X1 in their trust store. And clarification on OpenSSL versions below.

If you're experiencing this symptom, it most likely means that ISRG Root X1 is not in your trust store, and you should update it.

FGasper · October 1, 2021, 1:11am

If you're experiencing this system, it most likely means that ISRG Root X1 is not in your trust store, and you should update it.

No, it’s definitely there; if I throw away the non-self-signed ISRG Root X1 then the verification succeeds.

FGasper · October 1, 2021, 1:13am

You can see this yourself:

First try it w/ all three certs, then delete the first one and retry.

jsha · October 1, 2021, 1:13am

Can you share the output of openssl version and the command you are running to validate?

FGasper · October 1, 2021, 1:14am

OpenSSL 1.1.1j  16 Feb 2021

The command is openssl verify felipegasper.com.chain.

jsha · October 1, 2021, 1:22am

I'm not sure of the ins and outs of how openssl verify is supposed to work. What we know and have tested is that OpenSSL 1.1.0+ works when you use it to validate a TLS connection. For instance, if you have curl compiled against OpenSSL, can you curl https://letsencrypt.org? If so, you have a working system with OpenSSL.

rg305 · October 1, 2021, 1:27am

"delete the first one"
Shouldn't that be:
delete the last one
OR
are you counting upwards?

I literally had this same (type) discussion with my wife on the phone earlier today...
Open the cabinet door
OK
It's on the first shelf
Is "first" the top self? or the bottom shelf?
...argument ensues...
LOL