Certificate failure due to X3 to R3 intermediate certification transition

[tom_bbd]

Hi all,

Today we've renewed one of our Let's Encrypt certificate using certbot, and it seems it signed it with the new intermediate certificate due to the transition to ISRG's Root.

Unfortunately, we had our certificate distributed on a desktop app with SSL pinning relying on it. This means that our application basically broke.

Just in case, can you think in any way to recover from this situation? Thanks in advance!

Edit: Adding more context, we renewed the cert with the new R3 intermediate cert, while our apps are expecting the old X3 one.

[JuergenAuer]

Hi @tom_bbd

@tom_bbd

Unfortunately, we had our certificate distributed on a desktop app with SSL pinning relying on it. This means that our application basically broke.

unpin it and wait. That's all.

Pinning intermediate certificates without reading the announcements is a mistake.

The old X3 is gone. Every new certificate uses the R3 intermediate.

[tom_bbd]

Thanks @JuergenAuer,

The problem is that every communication fails between our servers and the apps fail due to the ssl pinning, so we don't have a way to unpin it for all our installed base.

Just to confirm, once signed up with the new R3, there is not any chance of getting it signed back with the R3, correctly?

At least we need to try, hope you understand. Thx in advance!

[JuergenAuer]

@tom_bbd

The problem is that every communication fails between our servers and the apps fail due to the ssl pinning, so we don't have a way to unpin it for all our installed base.

That's expected. Pinning works.

If you use dynamite the wrong way, you have a problem.

[tom_bbd]

Thx @JuergenAuer. We'll learn from this one!

[JuergenAuer]

@tom_bbd

Edit: Adding more context, we renewed the cert with the new R3 intermediate cert, while our apps are expecting the old X3 one.

PS: Never, never, never use hard coded intermediate certificates. They can always change, it's always possible that such a certificate must be revoked.

See

Chain of Trust - Let's Encrypt

[tom_bbd]

@JuergenAuer

PS: Never, never, never use hard coded intermediate certificates. They can always change, it's always possible that such a certificate must be revoked.

Thx for the tip @JuergenAuer,

What's would you consider the best alternative then? Sorry for my lack of knowledge here, obviously I have a lot of gaps regarding the proper use of certs .

[Osiris]

@tom_bbd

and it seems it signed it with the new intermediate certificate due to the transition to ISRG's Root.

This is incorrect. The change from X3 to R3 is not related to the ISRG root transition. Intermediates can change at any time and this should not cause any problems. If it does, than the implementation is incorrect.

@tom_bbd

What's would you consider the best alternative then? Sorry for my lack of knowledge here, obviously I have a lot of gaps regarding the proper use of certs .

The ACME server always sends the complete chain, i.e. end leaf certificate including the intermediate certificate used to sign it. Your ACME client should use this feature to either separately store the used intermediate, offer the chain as a complete set or both of these two options.

Two possibilities exist:

1. You're using an ACME client which is not properly designed, or
2. You're not using the intermediate certificate provided by your ACME client.

[JuergenAuer]

@tom_bbd

What's would you consider the best alternative then? Sorry for my lack of knowledge here, obviously I have a lot of gaps regarding the proper use of certs

I don't understand your question.

Such a setup

@tom_bbd

while our apps are expecting the old X3 one.

is wrong. Certificate deploy -> deploy the intermediate too, that's all.

PS: @Osiris has the longer explantation. Letsencrypt gives you always the intermediate, so use it.

[tom_bbd]

@Osiris

This is incorrect . The change from X3 to R3 is not related to the ISRG root transition. Intermediates can change at any time and this should not cause any problems. If it does, than the implementation is incorrect.

Thx Osiris, I've updated the title.

Reading your answer now.

[tom_bbd]

@JuergenAuer

Letsencrypt gives you always the intermediate, so use it.

Thx @JuergenAuer,

I think we've been using the one that Let's Encrypt provided, but it changed recently from that X3 to the new R3, may it be?

Just trying at this point to learn how to do it better in the future and avoid repeating the same mistakes. Thx in advance.

[Osiris]

@tom_bbd

but it changed recently from that X3 to the new R3, may it be?

Yes, but since that change onwards, Let's Encrypt also provided R3 when it issued any certificate.

[tom_bbd]

@Osiris

The ACME server always sends the complete chain, i.e. end leaf certificate including the intermediate certificate used to sign it. Your ACME client should use this feature to either separately store the used intermediate, offer the chain as a complete set or both of these two options.

Two possibilities exist:

1. You're using an ACME client which is not properly designed, or
2. You're not using the intermediate certificate provided by your ACME client.

Thanks @Osiris ,

In theory we used certbot to renew it. If that was the case is there any way we could have messed things up with it?

[Osiris]

@tom_bbd

In theory we used certbot to renew it.

With every renewal, certbot updates all the files, including the intermediate which was used to sign the certificate. You can find the intermediate certificate in chain.pem and the end leaf cert plus intermediate in fullchain.pem.

[tom_bbd]

@Osiris

With every renewal, certbot updates all the files, including the intermediate which was used to sign the certificate. You can find the intermediate certificate in chain.pem and the end leaf cert plus intermediate in fullchain.pem .

Thx @Osiris,

not sure if the way we run the command had an impact on the issue. This is how we have been renewing them:

letsencrypt -it certbot/dns-cloudflare certonly --reuse-key --dns-cloudflare -d *.[example.org](http://example.org/) --preferred-chain “DST Root CA X3” --dns-cloudflare-credentials /etc/letsencrypt/cloudflare.ini

Could you spot any problem? Thx in advance!

[Osiris]

The problem is not getting the certificate, it's what you do with the files afterwards.

If you use chain.pem only once for your pinning and you don't update it when it changes, well, that's bad.

[griffin]

@JuergenAuer, if I could like that post five times I would!

As great as your tech skills are, I think they might be rivaled by your deadpan comedy potential.

[schoen]

@Osiris

If you use chain.pem only once for your pinning and you don't update it when it changes, well, that's bad .

Yeah, Certbot will change the contents of chain.pem when you do a renewal if the intermediate suggested by the CA (which is based on the intermediate that signed the leaf certificate) has changed. If some part of your environment elsewhere (like TLS clients) assumes that it will never change, then you won't be able to take proper advantage of the way that Certbot is giving you an up-to-date chain.pem file!

[tom_bbd]

Thx all,

Really good answers. What would be in your opinion the proper design to do ssl pinning between a server and a desktop app using the intermediate certificate? I assume that we should have a way to update this cert remotely (for updating old versions). Another option is to stop using this intermediate cert as it could effectively change.

Thanks in advance for your help.

[JuergenAuer]

@tom_bbd

What would be in your opinion the proper design to do ssl pinning between a server and a desktop app using the intermediate certificate?

There

@tom_bbd

Another options is to stop using this intermediate cert as it could effectively change.

is your answer. You can't control the intermediate certificate.

Chain of Trust - Let's Encrypt

contains a backup intermediate certificate, if the intermediate must be revoked.