Cert-manager+LE giving unwanted DST X3 chain after Feb 8

Issuance Tech
1

We had an outage today caused by cert-manager and letsencrypt. Letsencrypt changed their default chain from X1,DST X3 to just X1. But we had already included a config param to use only X1, and that param stopped working and instead gave us the unwanted DST X3 instead. Beware.

More details in comments…

1 Like
2

This link was helpful for us to understand what was changing. But the change didn’t work like it was described here because cert-manager also did something unexpected

Also found a link to cert manager issue:

So the outcome was that while we had included a PreferredChain param in the config to request ISRG Root X1, instead we started getting ISRG X1 + DST X3 after Feb 8. First intermediate cert was fine but the extra DST was added. Chopping off the last cert in the chain gave us a working chain.

1 Like
3

It seems like the best option is to stop setting the PreferredChain X1 setting, as that will be the default going forward.

Thanks for sharing the cert-manager bug.

6 Likes
4

looks like it grabbed anything alt version of chain when it configured, not actually looking at chain's name?

4 Likes
5

I read their bug: it looks for an alternate chain containing X1, so it finds X1 in the long chain. That’s different from Certbot and other clients that look for a chain ending in X1, including the default chain.

10 Likes
6

If that's the case, how would that option have worked when the long chain was the default? :thinking: It would never have selected the X1 short chain with such logic.

Or would it only have searched in the alternative chains? Because then it could work indeed with just a single alternative chain offered anyway.

By the way, for anyone who wondered: Certbot does not have this bug due to the fact it includes the default chain when searching for the selected issuer common name (and it also only looks at the last certificate in the chain):

And the other part of this bug (also looking at intermediate certs) was fixed in Certbot 1.12.0 back in the beginning of 2021. Alt-chain selection was introduced in 1.6.0 which also looked at the default chain, but stopped looking at intermediates.

3 Likes
7

I recall in our case that DST was used some time back and we had to switch to a shorter chain (Last Sep I think). At that time adding preferred chain annotation as ISRG Root X1 was the correct choice.

8

This issue and workarounds are now documented on the cert-manager release notes

3 Likes
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.