Questions re: OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates

Yes, that summary is correct, modulo the language "rooted at".

The ACME API will offer two chains for clients to select between when downloading their newly-issued certificate:

  • leaf <- R3 <- ISRG Root X1 <- DST Root CA X3 (self-signed)
  • leaf <- R3 <- ISRG Root X1 (self-signed)

The first of those will be the default, which clients will get unless they have been configured to request an alternate chain.

The second of those will be the one used by the ACME API to conduct TLS handshakes when clients first connect to it, since we do not have clients which run on Android and so do not need the extra compatibility provided by the default chain.

6 Likes

That's surprising to me.

2 Likes

I'm not really into mobile development, but I'd guess TLS servers on mobile OSes aren't terribly common unless except maybe for people running Android on IoT devices.

4 Likes

I used to run one for mobile development, but Google really hosed it when they tweaked some "security concerns" regarding what certain apps were doing.

5 Likes

Even if the server isn't on Android, a mobile ACME client using DNS-01 to get a certificate and upload it to a server sure isn't the craziest idea I've ever heard.

And yeah, an IOT-type device might want to get a cert, and running Android on one of those wouldn't be the craziest idea either.

But hopefully in either case it's be easy enough to ensure that ISRG Root X1 was in the trust store.

3 Likes

If you really want to use an ACME client on Android, or anything that handles private key material, you should make sure you have a version of Android that still receives security fixes and updates (and make sure to install them!) - and these should not require the compatibilty chain anyway.

5 Likes
3 Likes

Hello,

While Android will continue to honor DST Root CA X3 even after expiry, does anyone have details about iOS compatibility for either of root certs - ISRG Root Certificate or DST Root CA X3?

Thanks

3 Likes

All modern versions (at least iOS 10 and newer) of iOS trust ISRG Root X1. You should anticipate that the vast majority of iPhone/ iPad users run a new enough version of the OS to trust ISRG Root X1 because Apple's tight control over iOS ensures most of its customers can upgrade for several years after purchasing a product, whereas some Android products are shipped running an already obsolete version and are never updated.

3 Likes

FYI, I somewhat recently updated Certificate Compatibility - Let's Encrypt - Free SSL/TLS Certificates. It now lists the major root stores that trust ISRG Root X1, along with the first version that included it. @tialaramex is correct that iOS 10 trusts ISRG Root X1. I've confirmed iOS 9 does not, and included a link on that page. Any corrections are welcome!

4 Likes

I believe you can add current PS 4 firmware to the list which trusts ISRG Root X1 based on this document. If it's important to check the reality matches the claims, I can have a friend with a PS4 visit any test site you prefer as a favour, let me know.

3 Likes

This is a great news to me. Thank you!

Our system have some technical concerns for update openssl from 1.0.2 to 1.1.0.

On the other hand, We have found that openssl 1.0.2 has a flag (X509_V_FLAG_TRUSTED_FIRST) which can verify certificate from trusted ca store first.
The following is the certificate chain used in our experiment environment

a (expired) -> b (cross sign) -> c -> d (simulate Android Compatible Chain)
b (self sign) -> c -> d (simulate Openssl 1.0.2 Compatible Chain)

Server Provide certificate chain: a (expired) -> b (cross sign) -> c ->d

Client use openssl 1.0.2 can successfully verify the new chain when X509_V_FLAG_TRUSTED_FIRST is enabled and certificate b (self sign) is within system ca store.

We would like to know wheterh is it recommend to open this flag at openssl 1.0.2?

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Last year when ADDTrust expired we have openssl issue of 1.1.x.
Check details below at " Condition 2 - Broken Clients - Known Examples ( OpenSSL based client software )":
https://calnetweb.berkeley.edu/calnet-technologists/incommon-sectigo-certificate-service/addtrust-external-root-expiration-may-2020#affected

But I saw message that you are compatible with openssl 1.1.x without problem. Could you help confirm is that really true?
Because last year, we have manually remove expired CA from server and execute update-ca-certificates on Debian 9 to mitigate the issue.

2 Likes

I am still struggling to understand what was fixed in OpenSSL and in GnuTLS.

Is the OpenSSL change this one?

And the GnuTLS change is it this one?

And related merge-requests?

I am slightly concerned that the default chain will be changed to the one that "helps android" whilst at the same time "kill Linux which are already trusting ISRG Root X1"

Are you already operating demo sites with the longer / next default chain, in the similar fashion to badssl.com? Such that we can test our clients against them?

1 Like

These clients used to verify certificate chains in a bad way, here's a short explanation:

Older versions of OpenSSL, GnuTLS (+ many others, e.g LibreSSL) basically validate a certificate chain by walking up the chain as send by the server, then searching the trust store for the "highest" certificate. That means those clients always validate up the highest certificate in the chain.

This is an issue, as Let's Encrypt is going to send a chain that ends with DST Root CA X3, which expires later this year. This means that those clients will fail, as they no longer consider the highest root a trusted certificate.

The correct way to verify the certificate chain is by looking at the intermediates: In the chain send by Let's Encrypt certificates, there's an intermediate "ISRG Root X1 signed by DST Root CA X3" that is in fact a root in itself. If a chain verifier realizes this, it can stop verifying at ISRG Root X1 and does not need to verify the expired DST Root CA X3 cert.

Older versions of many non-browser verifiers sadly do not have this feature, and will sadly fail handshakes.

The OpenSSL fix to this issue was the introduction of a new flag X509_V_FLAG_TRUSTED_FIRST. If a client application sets this flag, it causes OpenSSL to search the trust store before processing the intermediate. This causes OpenSSL to stop looking at the chain when ISRG Root X1 is encountered, preventing issues with an expired DST Root CA X3.

This flag is available since OpenSSL 1.0.2 (must be manually requested by the application using OpenSSL) and is set by default since OpenSSL 1.1+, which is why that version is considered fixed.

Yes, altough the initial description is highly confusing.

Let's Encrypt has already started to use the new chain (since May 4), but it isn't causing issues yet, as DST Root CA X3 is not yet expired. You can't test using production, as Let's Encrypt certificates are only valid for 90 days, but the expiry is more than 90 days away.

You can however test using the staging enviroment, quoting myself:

6 Likes

Debian 9 does not ship OpenSSL 1.1 by default (but it does ship 1.0.1, which is a similar number), are you running custom versions of OpenSSL? Are you certain you had 1.1 when you had issues? Yeah, confused jessie and stretch.

I did testing with OpenSSL 1.1.1 earlier this year and it handled the new chain (more specifially: A test replica of the new chain) just fine. It did not matter if the expired CA was in the trust store, as it wasn't used for validation anyway.

I remember that some Python version had an issue that caused it to fail when an expired Root CA was present in the trust store, but I don't remember enough details to make a conclusion.

2 Likes

@Nummer378 :
Debian 9 does not ship OpenSSL 1.1 by default (but it does ship 1.0.1, which is a similar number), are you running custom versions of OpenSSL? Are you certain you had 1.1 when you had issues?

Debian 9 currently have OpenSSL 1.1.0l (check [stretch] part, not jessie ).
https://packages.debian.org/search?keywords=openssl

I'm also not sure which was the fixed version of OpenSSL. But I've experienced the broken certificate chain because of expiration last year in Debain Stretch. And from https://popcon.debian.org/ , Debian Stretch still have a lots of online machine. It will be helpful if you can confirm OpenSSL 1.1.0* in Debian Stretch will be fine this time. Or if you can give me some guide, I can also help confirm that.

I can onfirm Debian 10(which shipped 1.1.1 by default) will be fine when expired CA in the trust store.

Thank you.

1 Like

From my point of view (also see my post above), the fix was the introduction of X509_V_FLAG_TRUSTED_FIRST in OpenSSL. Per the official documentation, this flag is set by default since OpenSSL 1.1.0. The docs don't say if they mean 1.1.0 in general, or a specific patch release of 1.1.0.

https://www.openssl.org/docs/manmaster/man3/X509_VERIFY_PARAM_set_flags.html

When X509_V_FLAG_TRUSTED_FIRST is set, which is always the case since OpenSSL 1.1.0, construction of the certificate chain in X509_verify_cert(3) searches the trust store for issuer certificates before searching the provided untrusted certificates. Local issuer certificates are often more likely to satisfy local security requirements and lead to a locally trusted root. This is especially important when some certificates in the trust store have explicit trust settings (see "TRUST SETTINGS" in openssl-x509(1)).

Edit: I just checked the source code. X509_V_FLAG_TRUSTED_FIRST is set by default since Check chain extensions also for trusted certificates · openssl/openssl@0daccd4 · GitHub, which is part of the very first OpenSSL 1.1.0 release.

There's some confusion to this, as you sometimes also need to set X509_V_FLAG_PARTIAL_CHAIN, which is not set by default in 1.1.0, but I believe that this isn't neccessary in this case though.

I also want to highlight that your trust store needs to include ISRG Root X1, otherwise all of this is moot.

2 Likes