Just wondering what the advice will be for Android mobile clients to workaround this for 3rd party APIs. Does this get fixed by adopting the Google Play Provider? Does that get updated CA certs?
Or is only workaround to ask clients to ship additional CA cert that they install in addition to the system certs?
Essentially if you were to write the stackoverflow answer for the mobile app devs who will be impacted by this, what would you write?
I've raised https://github.com/square/okhttp/issues/6403 to workout what we can advise when it get's raised. It seems like a similar case to Sectigo expiry earlier this year, adopting Conscrypt Android was a fix there because it was cross signed (IIUC) but older Android VMs were tripping anyway.
I'm only rarely an Android developer, but it seems to me that neither Conscrypt nor Google Play Services' Security Provider would affect the device's list of trusted roots.
Instead, the trusted roots are provided externally (by platform default or developer API), which the chosen Security provider then consults when authenticating a certificate chain.
It's probably best to get the okhttp maintainers to verify that, though.
You could, as an app developer, just bundle an up-to-date trust store (e.g. the Mozilla one https://curl.haxx.se/docs/caextract.html) to feed to okhttp. It would be an inelegant solution as it puts the burden of keeping the list up-to-date on the app developer.
This problem seems like it should be solved by the platform, though it's not clear to me whether that's achievable in the land of Android today. tl;dr; I don't know.
For context, I'm one of the OkHttp maintainers. Yes, generally we generally just use the system certificates. But we could provide a 20 line example for app developers to use if we know what that looks like.
val cert: X509Certificate = """
-----BEGIN CERTIFICATE-----
MIIBFzCBwgIJAIVAqagcVN7/MA0GCSqGSIb3DQEBBAUAMBMxETAPBgNVBAMMCGNh
c2guYXBwMB4XDTE5MDkwNzAyMjg0NFoXDTE5MDkwODAyMjg0NFowEzERMA8GA1UE
AwwIY2FzaC5hcHAwXDANBgkqhkiG9w0BAQEFAANLADBIAkEA8qAeoubm4mBTD9/J
ujLQkfk/fuJt/T5pVQ1vUEqxfcMw0zYgszQ5C2MiIl7M6JkTRKU01q9hVFCR83wX
zIdrLQIDAQABMA0GCSqGSIb3DQEBBAUAA0EAO1UpwhrkW3Ho1nZK/taoUQOoqz/n
HFVMtyEkm5gBDgz8nJXwb3zbegclQyH+kVou02S8zC5WWzEtd0R8S0LsTA==
-----END CERTIFICATE-----
""".trimIndent().parsePemCertificate().toX509Certificate()
val handshakeCertificates = HandshakeCertificates.Builder()
.addPlatformTrustedCertificates()
.addTrustedCertificate(cert)
.build()
val client = OkHttpClient.Builder()
.sslSocketFactory(handshakeCertificates.sslSocketFactory(), handshakeCertificates.trustManager)
.build()
With the Sectigo issue, there was an existing replacement CA certificate but bugs in older Android stopped it getting used. Switching to the Conscrypt TrustStore impl was an effective fix.
IMHO We wouldn't ship root certificates in OkHttp, it's not our job and makes us the weakest point.
A few concrete questions
Is there a test server already?
Can we write and self answer the stackoverflow question for Android OkHttp users now ahead of this hitting client?
Yeah, https://valid-isrgrootx1.letsencrypt.org serves a chain that should fail on Android <7.1.1's trust store. You could use it as a test case on a sufficiently old version of Android.
There's no test site yet for the upcoming ECDSA root "ISRG Root X2".
Certainly, I meant to suggest that app developers could choose to consult an external trust store in their individual apps. As e.g. Firefox Android already does.
If it were me, I would just use the Mozilla root list verbatim, rather than splicing certificates from two programs together. YMMV.
I have an understanding that the Sectigo and Let's Encrypt situations are completely different.
In the case of the Sectigo incident, certain TLS implementations (OpenSSL 1.0.x, GnuTLS) had validation routines which immediately errored upon seeing an expired root in one possible path, even though they could have successfully built a path to a valid root a different way.
I think that is why using the Conscrypt security provider fixed things for you. It didn't change anything about the device's trust store, it just changed an undesirable behavior in the validation routine.
On the other hand, these Android devices straight up do not have the ISRG root in their trust stores. The problem is much simpler and can only be fixed by replacing or supplementing the trust store.
I'm aware that the issue is different, but hoping to find a similar complexity answer to give app developers an option to deal with this. A way forward.
App developers can't easily encourage tens of thousands of users to install new certificates, but they could proactively include a new certificate if they know it's likely to happen. Or when it hits for a re-issued server, then search stackoverflow, find a relevant answer and make the fix.
That's what I will probably suggest, write the stackoverflow answer ahead of time.
The decisions Mozilla makes with regards to the inclusion or exclusion of CA certificates in its root store are directly tied to the capabilities and behaviours of the software Mozilla distributes. Sometimes, a security change is made wholly or partly in the software instead of the root store. Further, Mozilla does not promise to take into account the needs of other users of its root store when making such decisions.
Therefore, anyone considering bundling Mozilla's root store with other software needs to be aware of the issues surrounding providing a root store, and committed to making sure that they maintain security for their users by carefully observing Mozilla's actions and taking appropriate steps of their own. On a best-efforts basis, Mozilla maintains a list of the additional things users of our store might need to consider.
So, if I were developing a mobile app that needed to connect to my back-end API, and I secured that back-end API using Let's Encrypt, adding the ISRG roots to the trust store for my connections makes sense. If one needs to connect to arbitrary servers without knowing which CA they're using, then there's really no good solution if your platform's trust store isn't updated regularly. Needing Let's Encrypt's root is just one issue here; in general if your trust store isn't being updated then what will you do if some CA gets its root compromised or is otherwise distrusted by the community (or just the next "good" CA after Let's Encrypt needs to rotate a root certificate for whatever reason)?
I think both@_az's and @petercooperjr's suggestions are good. On balance, I think the best approach is to have apps add just the ISRG Root X1 on top of the OS root store. Managing a root store is tough, and even though the root store on older Android devices is not generally getting updated, this approach will be better when the app ships on up-to-date devices, since it will honor distrust by the OS (for roots other than ISRG Root X1). If you wanted to be robust to the possible distrust of ISRG Root X1 (unlikely, I hope ), you could add the root only when the OS is known to be pre-7.1.
It looks like that's also the approach you've taken on the okhttp issue, so great!
Maybe not. I'm not convinced that these distrusts were ever implemented in Android's security provider. (Outside of the stock AOSP browser and Chrome, that is).
For example, this site has an unexpired Symantec certificate which is supposed to be distrusted.
Firefox and Chrome both complain about it.
On the other hand, I can quite happily connect to that site using a simple HttpsURLConnection in a demo app on Android 9 .
Sorry, I get that this is moot by now and devolves into a philosophical discussion about how to securely cope with running an app on an old OS. I thought it'd be worthwhile to check and share what Android/AOSP actually does, especially since there appears to be no public authoritative information on the topic.
curl (built with either nss or openssl) doesn't seem to care either, if we want to draw comparisons to how popular tooling in the ecosystem operates.
I think what's happening here is that the relevant Symantec root has been distrusted by special logic in FF / Chrome that is implemented in code rather than by removing a root outright. This has been a moderately common tactic in recent distrusts, which are complicated multi-stage processes to minimize breakage. Which is what @petercooperjr is talking about and linking to - I think we're all on the same page here.
So a refinement of my point above would be: If the OS actually removes a root (as opposed to a browser applying harsh restrictions on that root), it's useful for the app to honor that rather than have an embedded and possibly out-of-date copy of the whole Mozilla store. Adding just one root rather than replacing the store outright achieves that goal.
I do agree that this is all pretty much on the margins, and for devices that aren't getting updates it doesn't make much difference one way or the other. However, there's definitely benefit in not encouraging more app makers to start distributing their own root store without realizing what they're getting into.
If an app is only targeting Android 6+ there are several advantages:
It just requires adding two files (CA file, android_network_security.xml) and a modification of the AndroidManifest.xml. Therefore this would be even possible on apps if someone has lost the app source code.
It affects not only one HTTP client library but every HTTP client in the app and it should also affect (Chrome based) WebViews used by the app (not tested).
This last option looks like the one that should be used, does support for this mechanism go back to Android 5.0 (edit: missed this when I posted initially)? That's what modern OkHttp targets, so as long as that is true, then I'm happy to suggest this as the optimal fix?
If we advertise including a stackoverflow question and self answer, then I would also include my previous suggestion to add a single additional CA via code. This is still in play because libraries using let's encrypt can't easily make the manifest change, but they can embed additional certificates in their library.
Unfortunately the Android Network Security configuration was added with Android 6, therefore it has no effect on versions prior to Android 6.
Considering Android 6+ devices this would raise the supported devices to 85% according to the September Android version graph.
However on Android 4 and 5 you could simply install the Let's Encrypt Root CA certificate as user certificate which will add it as trusted CA certificate, something that does not work on Android 6 devices and newer because of the existing app-specific Network Security Configuration.
I don't think any solutions requiring user installation steps are viable for app developers. We could present that as a third solution for intranet apps etc. But for something shipped via the play store, it is not even worth mentioning.
I agree, that having the user to install something on the device is not a desirable solution, it is more a last resort and still better than being not able to use the device anymore.
Whith some small programming effort it would be even possible to provide this solution via play store: It would be possible to provide an "Let's Encrypt CA installer app" via play store that contains the root CA certificate and once started allows the user to click a button, that extracts the CA certificate to /sdcard and the starts the certificate installation process for the extracted file. Then the certificate import dialog opens and the user has to click OK to continue.
The process was originally designed to import .p12 file, but it should also work for Root CA certificates (not sure if it has to be wrapped in a PKCS#12 file or can directly be imported as PEM or DER encoded certificate.
From OkHttp side I'm not going anywhere near that. Something for your team to consider. It would be a great solution, but still low effectiveness for public apps. It's high touch solution, and some number of users will have problems with your fix app and want support.
We tested yesterday and found that the Android Network Security configuration didn't work on Android 6 but did on 7 (including WebViews created by the app).
I'm putting some tests of client workarounds on this PR https://github.com/square/okhttp/pull/6412/files if anyone wants something to test out of Android Studio or the command line with a few clicks.
), you could add the root only when the OS is known to be pre-7.1.
.