After using short certificate chain, trust is broken in certificate pinning mechanism

Hello. In our project we use certificate pinning mechanism between App and Backend Domain.
Public Certificate for Backend domain is issued by Let's encrypt.
Certificate short chain looks followingly: Public certificate -> R3 -> ISRG Root X1
Immediatelly after using this short chain without DST Root CA X3 included - trust between ALL Android Apps and Backend is broken. For iOS it works also with short chain (Shortening the Let's Encrypt Chain of Trust - Let's Encrypt)

In the app, we already removed DST Root CA X3 from app trust store couple of months ago and only ISRG Root X1 is included. So actually, I have no idea, why we get following error on Android immediatelly after using short certificate chain for public certificate:
<-- HTTP FAILED: javax.net.ssl.SSLHandshakeException: untrusted CA based on A_15887-03: CN=ISRG Root X1, O=Internet Security Research

Why there is SSLHandshakeException as app should trust all public certificates issued by ISRG Root X1 ? Do you have any hints ?

With long certificate chain it works also on Android. But log chain can't be used for long as it will not be supported by let's encrypt from 06.06.2024

Which version of Android?

3 Likes

@webprofusion Because of time pressure, we managed to test it only with Android 13 devices, but we assume it is reproducible for all Android versions

I'm not really familiar with Android development myself, but if you're using some sort of custom trust store, you might need to share some code or at least a description of how you're doing it. Building and using a custom trust store is an advanced technique that's not usually needed.

4 Likes

@petercooperjr I am actually not a developer, but operation guy responsible for our backend (including renewal of backend public certificate).

Let's take it from different approach. Would it be possible to request new public certificate slightly before 06.06.2024 with a long chain ? We should buy ourselves 2-3 months more time until it's renewed again. Even after renewal, we could manually add DST Root CA X3 to the certificate chain
Would it be feasible approch at least until 30.Sep 2024, when cross signed X3 will expire ?

Yes, if your ACME Client supports that. Which one are you using? Certbot v1.12 and later have a --preferred-chain option for example to select the alternate chain.

3 Likes

Yes, ACME Client support that as we already use as preferred chain long chain with DST Root CA X3 included. (by default it was stopped by let's encrypt in february). But even though we have this set as preferred-chain, it will not be added after 06.06. by let's encrypt based on my understanding.

Okay. But you asked if you could request a new cert before Jun6 for the long chain.

LE changed the default chain to the "short" one in Feb but the "long" chain is available until Jun6 (in your case with preferred-chain). You are correct that after Jun6 there will no longer be a "long" chain offered so the preferred-chain won't work for that.

3 Likes

Thanks. So If I request new public certificates from Let's encrypt couple of days before 06.06.2024, then I will buy us max. 3 months more time until certificate with long chain is expired (beginning of September). Is my understanding correct ?

1 Like

Yes, the certs are just files. Once you have what you want before Jun6 they will work the same until the leaf expires (which is 90 days from issuance).

Make sure you don't have an auto-renew running after Jun6 that might overlay the cert and chain you got prior to Jun6.

You should probably request a fresh cert a little before like Jun2. This allows for any problems you might have. Make a special backup of these to avoid accidental loss.

4 Likes

Thanks @MikeMcQ

Theoretically, if public certs are just files. Would it be feasible to do this as the workaround all the time ?
So after 3 months when certificates with "long" chain will expire, request new "short" chain public certificate and manually add DST Root CA X3 in certificate chain. I know this is not a solution, but would it be feasible ?

Solution must come from app developers and time is valuable, so I would like to know, if there is anything, I can do as a workaround (until it's fixed by app developers in mobile app which will trust also "short" chain backend certificate)

1 Like

Yes, that would be a workaround until the cross-sign from DST Root CA X3 expires.

5 Likes

Can you use Firefox on Android?
It has its own list of trusted anchor root certificate in Firefox itself and doesn't use Androids.

1 Like

Android devices ignore the expiration on the root certificate, but still require the leaf/end-entity and intermediate certificates to have valid dates.

This will only work between June 6th and September 30th – when the cross-signed intermediate version of X1 expires – so you could only yield a maximum of 24 extra days (covering September 6th - 30th) for this effort. You only get one-shot at this, but you should be migrated long before.

You absolutely need to have a solution/plan in place before September 6th. You should immediately prioritize a solution in the next sprint for your product and development teams. Do not keep pushing this back until this manual last-resort manual-chain building period, and expect them to suddenly find something that works in that time. Aside from downtime, you seriously risk losing your staff or vendors.

5 Likes

In your app I suspect you are using your own trust store instead of using the device defaults, which is not something we see regularly. This means you need to push an update to your app to include ISRG Root X1. The root cert can be downloaded as PEM format etc from Chains of Trust - Let's Encrypt

You're likely using some variation of this technique in the app:

As a short term fix you can acquire a cert for your API service using the longer chain using your acme clients "preferred issuer" option set to DST Root CA X3, which will buy you a few weeks. Alternatively I would normally suggest trying a different CA (zerossl, google trust etc) but we don't know which CA roots you currently trust because you have your own custom list.

5 Likes

@Bruce5051 I don't understand. Yes, I can use Firefox on Android, but how is this related to reported issue ?

@jvanasco Yes, this TOPIC has high priority and we are not pushing this back. I justed wanted to know, what is maximum manual last-resort manual-chain building period. Thanks for confirmation

@webprofusion Yes, we use own trust store and I can confirm only ISRG Root X1 (self-signed) is included as PEM format, so let's encrypt CA root is only one which is trusted in our custom list.

Thanks, I've confused though - why does your app fail to connect to your API if ISRG Root X1 (Self signed) is trusted? Can you post the PEM of the version of ISRG Root X1 you have in your app trust store?

1 Like

@webprofusion I'm confused also and I don't get it how is this feasible. It's this PEM (MIIFazC...):
crt.sh | 9314791, which is in our app trust store