Public Key Changed?

Issuance Tech
#1

About 1.5 years ago I inherited a backend service that was originally built by a 3rd-party contractor and I don't fully understand how this aspect of the system works yet, so please bare with me. It's a Node.js + nginx service running on EC2 and Amazon Linux.

The service has been humming along for the past year and a half no problem and certbot has been automatically renewing certificates every month no problem... until now. Something happened during the last renewal that broke certificate pinning in one of our Android apps. After some digging it seems that for some reason the public key (chain.pem) changed, causing the signatures to be different and thus our app can no longer connect to our server.

For the time being, I temporarily updated the Letsencrypt symlinks on the server to point back to the previous certs (since they're still good for another 2 months) until I can figure out what to do. I've been reading up as much as possible about how all this works and have a general understanding, but it's still not clear to me why "all of a sudden" the public key would change during renewal and what the best course of action would be.

Should I try to "fix" the certs somehow so that they use the "old" public key once again? If so, how can I do this?

If this is not possible and we must update the Android app to recognize the new signatures, how would that even work? Wouldn't older versions of the app stop working forever with no way to self-update to the new version if it can't even connect to the server to get the new update? Seems like a chicken-egg/catch-22 scenario.

Just looking for a little guidance as I am way out of my wheelhouse here. Appreciate it, thanks!

#2

Hi Kid from Agusta, and welcome to the LE community forum :slight_smile:

You need to update the app ASAP.
It should never pin intermediate certs.
They can change at any given moment.
And you should always include a trusted local CA (as a precautionary safety net).

As far as the previous cert (Intermediate X3), there is NO going back - only going forward now.
Today R3 is the intermediate (but, again, that can change without any real notice).

So you will need to dig into the code and find all the nitty gritty on how/where/when it PINs certs.
And redesign it so that you don't get caught locked out of your own systems.

2 Likes
#3

Hi and thanks for the super quick response!

According to the documentation left behind by the contractor, the service uses 2 levels of certificate pinning:

leaf certificate: it’s the service certificate. Guarantees with close to 100% certainty that this is your certificate.
Intermediate certificate: it’s the backup certificate which helps when the primary pinned certificate has problem. By pinning against an intermediate certificate you are trusting that the intermediate certificate authority to not mis-issue a certificate for your server(s).

When I look in the nginx config, I see 3 separate pins specified: The first one is allegedly the primary key which is the fingerprint of the service’s certificate (leaf certificate). The other 2 certs are the intermediate Let’s Encrypt’s X3 and X4 certificates (backup certs).

So, my understanding is that they tried to pin it to the leaf certificate w/the expectation that this would never change. Is this expectation wrong?

#4

Hi @nub340,

According to Let's Encrypt's guidance on pinning, you should only pin root certificates, not intermediate certificates. You can see some other recent discussions about the X3→R3 intermediate change

https://community.letsencrypt.org/search?q=pin%20intermediates

where the conclusion from Let's Encrypt staff is not to update the intermediate pin from the X3 to R3 intermediate, but instead to pin root certificates (potentially including a root certificate that you control yourself!).

3 Likes
#5

The leaf should change every 60-90 days!
[in a normal world]

1 Like
#6

Interesting... there's an archive of previous certs on the server and I just used the openssl cli to check the expiry of each one and confirmed that the previous 17 chain.pem files are indeed identical and have the same expiry (Mar 2021) and only the most recent is different. How should I interpret this? Did they set this up, abnormally? :slight_smile:

#7

That is the recently retired/changed Intermediate cert chain pem file.
That's not a leaf cert.

The interpretation is:
The cert.pem contains the leaf cert.
The chain.pem contains the then current Intermediate that signs the leaf/cert and links it to the signing root/tree.
[The root/tree cert is never provided as that should already be in everyone's trusted store]

2 Likes
#8

I see. Ok thanks, this pinning stuff is all new to me. There is: cert.pem, chain.pem, fullchain.pem and privkey.pem. Of these chain.pem is the only one that seems to have remained static over the past 17 months, so I assumed that must be the "leaf" one. It seems I need to continue reading and get my ducks in a row.

1 Like
#9

Assumptions built on misconceptions... leads to nothing good.
There take was incorrect - don't build on that.
The leaf changes every time it is renewed.
The only constant is... change!
So if you must PIN, pin the root (and an added locally managed CA root)
Because even though it may seem like it's stable, even the root will eventually expire/need to be changed.
Never the leaves.
Never the Intermediates/chains/branches.

1 Like
#10

@nub340, you can read more about these changes at

Note that the most recent change happened in December. There are several forum threads from other people who had problems due to pinning the X3 intermediate (which was just discontinued), and an important conclusion of each thread was "intermediates may change in the future!".

cert.pem is your leaf certificate, chain.pem is a recommended version of the intermediate certificate that issued your leaf certificate, fullchain.pem is the combinations of cert.pem+chain.pem (which is used by most web server software in order to give clients a "full chain" from a trusted root to the leaf), and privkey.pem is the private key corresponding to the public key that is the subject of the certificate.

None of these files' contents is suggested for pinning. :slight_smile:

3 Likes
#11

Yep! This is why I came here, I knew I had assumptions that we're prob incorrect! :slight_smile:

Pin the ROOT certificate. Got it!

Thanks a bunch guys!!

2 Likes
#12

Thanks, @schoen - this might explain why the behavior changed in December.

2 Likes
#13

Your assumptions were logical - their base was incorrect and there is nothing you can build up from it that will stand the test of time.

2 Likes
#14

Ideally, more than just one! After all, a certificate authority could conceivably go out of business or ban particular users for some reason.

5 Likes
#15

Now that we've fixed the Thou Shalt Not PIN situation...
I wonder exactly what security issue(s) were being "overcome" by such PINnings?
[everyone can have preconceived notions about what benefits PINning may provide or may have actual previous experience with PINnings (that may, or may not, coincide with this particular case)]

If you back UP a few hundred feet... what role does PINning play in current implementation of the grand secure design?
Maybe we can also address that angle.
OR
We can just call it a night!

Cheers from Miami :beers:

#16

...but the private key (and therefore the public key) need not. For some clients, I think it's even the default behavior to keep the keys the same; for many clients, it's behavior that can be specified. Still doesn't mean that key should be pinned, of course.

#17

Taken out of context of:

Yes, one can force the key to never ever change.
But that is far from normal nor desirable.

If one is savvy enough to force one's will onto it, one is savvy enough to secure it properly.

If you really want to cover ALL target audiences, then we should also include those that are capable of but are debating whether or not to, run their own CA.
In the scenario presented, I would think that running one's own CA would far outweigh the trivial process to do so.
Starting with the most obvious benefits:

  • The cert life could be as long as... {until whatever the largest possible date field can hold}.
    [essentially never having to be renewed]
  • creating more of a turn-key setup than a must be maintained (in-the-field) setup
#18

Though I can't track down specifics, I seem to recall that this was the default behavior for a popular client in the not-too-distant past. In fact, it's looking to me like it's the current default behavior of acme.sh (I'm determining that from the fact that, on a system that's regularly obtaining certs using acme.sh, the private key file has a timestamp almost three years old, while the cert, csr, chain, and fullchain are dated about two weeks ago). It may not be the default behavior with certbot, but it's not like you have to go to esoteric clients or explicit requests to make this happen.

1 Like
#19

Sadly yes.
But there is a parameter to handle that:
--always-force-new-domain-key

#20

...which itself isn't particularly clear, what is a "domain key"? But yes, correct behavior can be specified--though it leaves the question of why this would be the default.