Let's encrypt certificate and Android app mismatch

Help
#1

Hello!

I've been working on an Android app that connects to a server via SSL using Let's encrypt certificates.
My domain is enginev2.genomapp.com.
My server is an Ubuntu with Nginx and docker.

Let's encrypt installed these files in my server:
cert.pem
chain.pem
fullchain.pem
key.pem
and in my Android app I'm using this cert.pem file to make the match.

This has been working perfectly for a long time. My cert has been renewing without problems and my Android app has been working with that cert.pem file so far.

This year, on May, I received an email from Let's encrypt telling that letsencrypt-nginx-proxy-companion will no longer supports ACME v1 endpoints and I would have to updated it to ACME v2.
So I updated my Let's encrypt to v2, everything went well, my cert was renewing without problems, my Android app kept working, etc...

Last night Let's encrypt renewed my cert (as usual I though) but this time my Android app stopped working. The Android error was that there was a mismatch in my SSL cert and the handshake couldn't be made.
I've had to take the new cert.pem file into my Android app and upload a patch quickly to Play Store.

The logs related to this renewal were completely normal.

So my doubts:

  1. Has anything changed in the way Let's encrypt renew the certs so now they don't match with the cert file used on Android app?
  2. Is it OK to use the cert.pem file in my Android app or should I use one of the others?
  3. What happened last night, does it mean that from now on everytime Let's encrypt renew my cert I will have to update my Android app and upload a new version to Play Store? I don't think so.
  4. But, If this is the case (everytime Let's encrypt renew my cert my Android app will stop working), what should I do to avoid this?

Thank you very much and kind regards!

1 Like
#2

Welcome Back to the Let's Encrypt Community, Ruben :slightly_smiling_face:

The transition from the Let’s Encrypt Authority X3 intermediate certificate to the R3 intermediate certificate is probably the cause.

You should probably be serving fullchain.pem, which contains both cert.pem and chain.pem (R3).

If your server is serving the correct chain (your certificate and the R3 intermediate certificate), your app should be irrelevant.

Serve fullchain.pem as already mentioned.

1 Like
#3

Hello Griffin!
Thank you for your answer!
When you say "serves" ("If the server is serving the correct chain" and " Serve fullchain.pem") what do you mean?
What I'm doing currently is coping manually my cert.pem file (from now on I will use the fullchain.pem as you suggested) from my server to my Android app. When you say "serves" do you mean that my Andorid app has to retrieve this fullchain.pem file from the server in some automatic way?

1 Like
#4

This is probably my misunderstanding of your setup. Typically, when a client app (e.g. browser) connects to a server, the server "serves"/transfers the certificate chain to the client (e.g. browser). This enables handshaking to occur that results in a secure connection.

If I'm understanding correctly, you're "preinstalling" the certificate with the app rather than having the app retrieve the certificate at the beginning of the connection. Correct?

If possible, it would be far superior to have your app check for a newer certificate on your server then update the local cached certificate. This would save you a lot of headaches. If you want to keep your certificate (cert.pem) and the intermediate certificate (chain.pem) separate rather than updating both together (fullchain.pem), you can use a similar mechanism of updating each individually. Just be sure that your app is using the current intermediate certificate when checking your certificate. Obviously, the intermediate certificate can only be changed when your certificate changes, so you'll never update the intermediate certificate alone.

1 Like
#5

Hi griffin!

In my case, the client app is not a browser but and Android app that calls an API Rest in that server.
In my case, the Andorid app is using Retrofit as the library that manages API calls. With Retrofit, as with many other network libraries, you have to configure the SSL connection. One of the parts of this configuration is providing the library with all trusted certificates. These trusted certificates are stored in the resources/raw folder of the app project and are injected to Retrofit on configuration step. In my case I have stored the cert.pem file.

So, taking into account your answer I think with every API call, the server "serves" the certificate to this Retrofit library inside my app, which checks if this certificate and the one I configured matches.

So this is the reason why I'm not sure if I can take the server certificate "automatically", because with Android apps I have to store all trusted certificates in my resources/raw folder.

Sorry if I'm not explaining this well.

1 Like
#6

I'm suspecting it may be much simpler than that. The idea behind a publicly-trusted certificate is that a publicly-trusted CA vouches for the mapping of a public key to one or more entities (e.g. domain name or IP address). If the server were serving a certificate chain to your app then the app could verify the intermediate certificate sent with the chain using a stored root certificate (ISRG Root X1 or DST Root CA X3) then verify your leaf certificate using the verified intermediate certificate. Note that it's not good practice to store the intermediate certificate (for the very reason this topic was originally created).

I have a strong suspicion that the Retrofit library is simply assuming a known mapping between your leaf certificate and your server. That is, your app already trusts your leaf certificate in its store because you put it there.

2 Likes
#7

So, am I doing this right? I'll follow your advise and I'll use fullchain.pem instead of cert.pem in my app, but apart of this, I'm not sure if I'm doing this in the right way.

And, I'm still confuse, if Let's encrypt renew the server cert, will I have to update my cert/fullchain file o it will keep working with the same file?

1 Like
#8

So what you're saying you're pinning cert.pem in your app as the only trusted server certificate? Well, that's not a very smart thing to do, as Let's Encrypt certificates only have a lifetime of 90 days.

Is there something agains pinning a root certificate like @griffin stated?

2 Likes
#9

I'm not familiar with your server software or your app framework, so I can only speak at a high level, but here's what the best practice would be:

  1. When your server renews, your ACME client needs to download the complete certificate chain (often called fullchain.pem)
  2. Your server needs to serve that complete certificate chain to its clients.
  3. Your app needs to validate the server's chain against the root in its trust store. Your trust store in the app (if your server is using Let's Encrypt certificates) should have at least DST Root X3, ISRG Root X1, and ISRG Root X2 in it, as well as certificates from at least one other CA (possibly one you make yourself) in case Let's Encrypt becomes unavailable on short notice.

You shouldn't need to have intermediates on the app side, just have the server send them and the client can validate it against the roots, as the roots change much less often.

(Maybe none of that is possible with your particular framework, I don't know, but it's how I would expect a well-written one to work.)

1 Like
#10

I understand what you say Osiris, and it's logical to me, but, what it's driving me crazy is: why has my app been working for 2 years with the same cert.pem? During this period Let's encrypt has renewed my server cert multiple times and I haven't had to change this cert.pem file, my app kept working all this time.

Regarding to pinning a root certificate I don't see any problem to do that, but I will have to investigate how to do it. I guess that won't be as easy as change cert.pem file for another file.

Sorry if I'm saying something silly but I'm not an expert with servers and certs :-S

1 Like
#11

Perhaps your library uses the internal Android root store too?

1 Like
#12

Thank you petercooperjr! I will read Android documentation about how to implement those steps.

1 Like
#13

This would make sense, I'm trying to find it out.

1 Like
#14

So, what you're saying, Peter, is that you concur 100% with exactly what I wrote three posts before yours... :wink:

I'm just busting your chops, but c'mon man... :laughing:

To be fair, you did mention the X2 root.

1 Like
#15

Well, thank you all of you for your answers. Finally I think I've made it work following the advise of using the root certs:

  • I've deleted the content of my cert.pem file > obviously my app stopped working.
  • Then I've added to the cert.pem file only the certificates DST Root X3, ISRG Root X1, and ISRG Root X2 > the app started working.

So, taking into account that these root certificates change every couple of years I'm sure I won't have to update my app frequently.
I guess I will have to be aware of when these roots certificates are going to be renewed and then upload my Android app with the newer ones.

3 Likes
#16

I think that Peter has more details that he's busy writing, but you basically need to have the full chain available to verify from root to leaf (yours). Ideally, the intermediate and leaf should be either automatically updated or sent with the connection. The root won't change that often, except DST Root CA X3 that expires in September.

2 Likes
#17

Those roots change pretty rarely (barring a major screwup that requires early replacement). The intermediates (which may be what you were trusting before), change more often (though the one for Let's Encrypt hadn't changed for quite some time until recently).

You may want to subscribe to the API Announcements category on this forum, where you can get an email when Let's Encrypt staff post there about major changes.

You may also want to add a CA to your device's trust store besides the ones for Let's Encrypt. While Let's Encrypt is likely to be around for the long haul, it could experience an extended-length problem around a time you need to renew your server certificates, or suddenly run out of funding, or otherwise not be where your next certificate comes from. This could be a certificate authority that you set up yourself. Check out this somewhat-recent thread with some thoughts on things to consider when building your own root store for a device.

Well, I suppose. I guess I was trying to separate out more clearly what the server-side needed to do from what the client-side needed to do, since the message seemed muddled in my probably-too-quick read through the thread. I certainly wasn't trying to disagree. :slight_smile:

3 Likes
#18

Don't worry, brother, if I find we're basically saying the same thing it usually means I didn't miss something. You tend to be more than a bit more meticulous in certain areas than me. :blush:

3 Likes
#19

Once I tested this, once I re-read the posts and I've understood everything I think all of you said the same thing ;-). But since I wasn't an expert with this topic I didn't realize it before.
I'll subscribe to that link as peter suggested and I'll investigate deeply this topics, but after these posts I understand them much better than 2 days ago, so thanks again!

2 Likes
#20

If I had to guess, @rfdevelopments, I'd say that Peter's speculation is the same as mine: you've have the Let's Encrypt Authority X3 intermediate certificate in your trust store (pinned as Osiris mentioned). Now you need the R3 intermediate certificate. Problem is that this practice of pinning intermediate certificates will become a headache when the intermediate certificates begin cycling more often.

2 Likes