Questions re: OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates

These clients used to verify certificate chains in a bad way, here's a short explanation:

Older versions of OpenSSL, GnuTLS (+ many others, e.g LibreSSL) basically validate a certificate chain by walking up the chain as send by the server, then searching the trust store for the "highest" certificate. That means those clients always validate up the highest certificate in the chain.

This is an issue, as Let's Encrypt is going to send a chain that ends with DST Root CA X3, which expires later this year. This means that those clients will fail, as they no longer consider the highest root a trusted certificate.

The correct way to verify the certificate chain is by looking at the intermediates: In the chain send by Let's Encrypt certificates, there's an intermediate "ISRG Root X1 signed by DST Root CA X3" that is in fact a root in itself. If a chain verifier realizes this, it can stop verifying at ISRG Root X1 and does not need to verify the expired DST Root CA X3 cert.

Older versions of many non-browser verifiers sadly do not have this feature, and will sadly fail handshakes.

The OpenSSL fix to this issue was the introduction of a new flag X509_V_FLAG_TRUSTED_FIRST. If a client application sets this flag, it causes OpenSSL to search the trust store before processing the intermediate. This causes OpenSSL to stop looking at the chain when ISRG Root X1 is encountered, preventing issues with an expired DST Root CA X3.

This flag is available since OpenSSL 1.0.2 (must be manually requested by the application using OpenSSL) and is set by default since OpenSSL 1.1+, which is why that version is considered fixed.

Yes, altough the initial description is highly confusing.

Let's Encrypt has already started to use the new chain (since May 4), but it isn't causing issues yet, as DST Root CA X3 is not yet expired. You can't test using production, as Let's Encrypt certificates are only valid for 90 days, but the expiry is more than 90 days away.

You can however test using the staging enviroment, quoting myself:

6 Likes

Debian 9 does not ship OpenSSL 1.1 by default (but it does ship 1.0.1, which is a similar number), are you running custom versions of OpenSSL? Are you certain you had 1.1 when you had issues? Yeah, confused jessie and stretch.

I did testing with OpenSSL 1.1.1 earlier this year and it handled the new chain (more specifially: A test replica of the new chain) just fine. It did not matter if the expired CA was in the trust store, as it wasn't used for validation anyway.

I remember that some Python version had an issue that caused it to fail when an expired Root CA was present in the trust store, but I don't remember enough details to make a conclusion.

2 Likes

@Nummer378 :
Debian 9 does not ship OpenSSL 1.1 by default (but it does ship 1.0.1, which is a similar number), are you running custom versions of OpenSSL? Are you certain you had 1.1 when you had issues?

Debian 9 currently have OpenSSL 1.1.0l (check [stretch] part, not jessie ).
https://packages.debian.org/search?keywords=openssl

I'm also not sure which was the fixed version of OpenSSL. But I've experienced the broken certificate chain because of expiration last year in Debain Stretch. And from https://popcon.debian.org/ , Debian Stretch still have a lots of online machine. It will be helpful if you can confirm OpenSSL 1.1.0* in Debian Stretch will be fine this time. Or if you can give me some guide, I can also help confirm that.

I can onfirm Debian 10(which shipped 1.1.1 by default) will be fine when expired CA in the trust store.

Thank you.

1 Like

From my point of view (also see my post above), the fix was the introduction of X509_V_FLAG_TRUSTED_FIRST in OpenSSL. Per the official documentation, this flag is set by default since OpenSSL 1.1.0. The docs don't say if they mean 1.1.0 in general, or a specific patch release of 1.1.0.

https://www.openssl.org/docs/manmaster/man3/X509_VERIFY_PARAM_set_flags.html

When X509_V_FLAG_TRUSTED_FIRST is set, which is always the case since OpenSSL 1.1.0, construction of the certificate chain in X509_verify_cert(3) searches the trust store for issuer certificates before searching the provided untrusted certificates. Local issuer certificates are often more likely to satisfy local security requirements and lead to a locally trusted root. This is especially important when some certificates in the trust store have explicit trust settings (see "TRUST SETTINGS" in openssl-x509(1)).

Edit: I just checked the source code. X509_V_FLAG_TRUSTED_FIRST is set by default since Check chain extensions also for trusted certificates · openssl/openssl@0daccd4 · GitHub, which is part of the very first OpenSSL 1.1.0 release.

There's some confusion to this, as you sometimes also need to set X509_V_FLAG_PARTIAL_CHAIN, which is not set by default in 1.1.0, but I believe that this isn't neccessary in this case though.

I also want to highlight that your trust store needs to include ISRG Root X1, otherwise all of this is moot.

2 Likes

I just ran a few tests with a Debian Stretch test VM, default settings.

# cat /etc/os-release
PRETTY_NAME = "Debian GNU/Linux 9 (stretch)"

# wget https://letsencrypt.org/certs/staging/letsencrypt-stg-root-x1.pem
# wget https://letsencrypt.org/certs/staging/letsencrypt-stg-root-dst.pem
# cat letsencrypt-stg-root-dst.pem letsencrypt-stg-root-x1.pem > certs-combined.pem

# openssl version
OpenSSL 1.1.0l 10 Sep 2019

Simulating that only ISRG Root X1 is in the trust store:

# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile letsencrypt-stg-root-x1.pem
-> Verification: OK

Simulating that only DST Root CA X3 is in the trust store:

# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile letsencrypt-stg-root-dst.pem
-> Verification error: certificate has expired

Simulating that both DST Root CA X3 and ISRG Root X1 are in the trust store:

# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile certs-combined.pem
-> Verification: OK
8 Likes

I think it's probably better to think about the situation as a Directed Graph (in the mathematical sense) in which the nodes are named public keys and the edges are certificates issued by one node. The goal of a verifier is to decide if there are a series of edges that lead from named public key they trust (e.g. ISRG Root X1) to the leaf certificate that a TLS server is showing them to prove its identity. In this understanding it is these named keys which are trusted at the root, not certificates for them.

A verifier that trusts ISRG Root X1 never needs to contemplate the "ISRG Root X1 signed by DST Root CA X3" certificate, because this represents an edge between ISRG Root X1 (trusted) and DST Root CA X3 (also trusted but soon to expire) so who cares?

It can stop once it sees just the "R3 signed by ISRG Root X1" certificate, it knows what ISRG Root X1 is, and it can verify using the public key it knows for that name that this certificate is valid, therefore R3 is trustworthy, therefore the leaf certificate is acceptable.

Unfortunately graph traversal problems are hard and so there is software that gets this wrong.

4 Likes

@tialaramex explains it well, though I would a caveat:

I would say "if there is a series of valid edges the lead from the the end entity certificate to any trust anchor (aka root certificate)." In other words, the path builder component of a verifier shouldn't be looking for a specific trust anchor, but for any one for which there's a path.

A couple of good but dense blog posts on the subject:

7 Likes

There are some Ubuntu releases that will be affected by this, whilst they are still supported or under extended security maintenance.

I've open bug reports to start tracking this issue in the affected series for openssl & gnutls28.

https://bugs.launchpad.net/bugs/+bugs?field.tag=letsencryptexpiry

5 Likes

I'm using Certbot to get my certs and it provide as of today:

  • fullchain.pem: leaf <- R3 <- ISRG Root X1
  • chain.pem: R3 <- ISRG Root X1

How will they look in the future?

1 Like

leaf <-R3 <- ISRG Root X1 <- DST root X3 IIRC

3 Likes

When will the change be rolled out - I think I have seen notes about June, but apparently it has not yet taken place?

1 Like

The new "long" chain is in use since May 4.

Your fullchain/chain includes R3, signed by ISRG Root X1 and ISRG Root X1, signed by DST Root CA X3. This is the current and future chain (for the foreseeable future) - no further changes.

4 Likes

I wonder, if between now and 2024 one could change to use a different, non-letsencrypt CA, from someone else that is both trusted by old Linux distros and old Android.

I fear of making a choice if I care more about "openssl s_client 1.0x" versus "android web browser 4.x..7.x". Cause I really don't know which has the largest impact (api usage, versus humans tap tap).

Is there a way to lookup what trust store CAs 4.x android shipped with?

2 Likes

If old client compatibility is your highest concern, another CA is probably your best option.

As Android 4.0+ is ~2012+, they should work fine with Sectigo's roots.

2 Likes

@beautifulentropy can someone else but DST sign ISRG Root X1 ? I.e. In android 7 i see GlobalSign with expiry in 2038, VeriSign until 2038, etc...

2 Likes

The fundamental problem is that having trust store updates is really a fundamental part of having security updates, and having any concept of a "secure connection" with a system that doesn't get security updates is an oxymoron. One can work around some issues for some amount of time with tricks like the expired-root-signature that Let's Encrypt is employing, but really the problem isn't with the CA or with the server, it's with the client that doesn't really know anymore which roots are trustworthy. (That is, if an old long-lived root somehow got its key compromised, there's no way for systems that don't get trust store updates to know about it.)

ISRG Root X1 itself only has 14 years of validity left. A system that needs to be handle secured communications over a many-year timeframe without updates really needs something different than the WebPKI can give it.

In theory, sure. But most CAs aren't thrilled with signing roots for what are basically competitors, and there's a lot of cost involved (and a lot of risk, since signing someone else's root means you're taking responsibility for everything that other root does). At this point in time, Let's Encrypt really is ready to stand on its own feet, and I'm guessing the financial/auditing/etc. requirements are too onorous for starting anything beyond their existing relationship with IdenTrust, though if somebody wanted to pay for it maybe they'd figure it out? I'm just guessing, though.

4 Likes

Over at Bug #1928989 “expiring trust anchor compatibility issue " : Bugs : openssl package : Ubuntu

You will find patch for OpenSSL 1.0.2g series as shipped in Ubuntu 16.04 LTS (xenial) and also a PPA built with this update for Xenial.

Testing things with it seems to make everything work. Instead of backporting all the features it really simply only sets the trusted-first flag by default. Please review these changes and let me know if anyone has any concerns about it. To me this makes openssl 1.0.2g be able to limp along with the new letsencrypt default chain, when the host otherwise trusts the ISRG Root X1 CA.

4 Likes

Although basic support for Ubuntu 16.04 LTS (xenial) has now ended, and only Extended Security Maintenance is offered (see Ubuntu Extended Security Maintenance | Security | Ubuntu), this issue is critical enough that openssl 1.0.2g update has now been published to xenial-security to address this compatibility issue.

https://launchpad.net/ubuntu/+source/openssl/1.0.2g-1ubuntu4.20

Xenial OpenSSL usage should thus not be affected by the upcoming change.

gnutls is currently still affected at the moment.

8 Likes

For gnutls I have prepared patches for Ubuntu 16.04 LTS (xenial) and Ubuntu 18.04 LTS (bionic) at

Ubuntu 18.04 LTS Bionic packages available from:

Ubuntu 16.04 LTS Xenial packages available from:

Hoping to get them reviewed and released into updates before the expiry, but not yet sure if it will happen in time.

4 Likes

This topic was automatically closed after 120 days. New replies are no longer allowed.