Following a recent certificate renewal, we have started experiencing certificate validation failures.
It appears that the latest ca-certificates package bundle shipped with Ubuntu (20240203 : ca-certificates package : Debian) does not support sites issued with the new Let’s Encrypt R10 certificate. When I manually specify the certificate file for the client (Chains of Trust - Let's Encrypt), it works fine.
Output from openssl s_client:
Certificate chain
0 s:CN = <REDACTED>
i:C = US, O = Let's Encrypt, CN = R10
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: <REDACTED> GMT; NotAfter: Dec 22 <REDACTED> 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = R11
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
...
Verify return code: 21 (unable to verify the first certificate)
Is there something that can be done via this forum, or should I file a bug in Debian/Ubuntu?
Intermediates (like R10) generally should not be included in the ca-certificates package.
Your s_client output shows a misconfiguration: You have a certificate signed by R10, but then an intermediate R11. This is why it is unable to build a chain from the leaf to root, as it needs the R10-signed-by-X1 intermediate.
How to fix this depends on your ACME client and how your webserver is configured. We can provide advice if you can tell us what software you're using, and show us the configuration of TLS in your webserver.
Are you referring to the subject not being equal? because this is more/less the same case with: acme-v02.api.letsencrypt.org (official API of Let's Encrypt):
Certificate chain
0 s:CN = acme-v02.api.letsencrypt.org
i:C = US, O = Let's Encrypt, CN = R10
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Sep 4 14:46:39 2024 GMT; NotAfter: Dec 3 14:46:38 2024 GMT
1 s:C = US, O = Let's Encrypt, CN = R10
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
Usually your ACME client just handles all that automatically, and you just configure your web server to use the certificate and chain that it gives you. You'd need to say more details about what software you're using to get the certificate and how you have it set up.
Yeah, don't manually hardcode intermediate certificates. That's the official instruction. Let the ACME client do all the work, as it gets the correct intermediate from the ACME server at issuance.
Note that the intermediates are randomly chosen between R10 and R11 (or E5 and E6 for ECDSA certs with three intermediate per key type as spares) and there's no way to know which intermediate is going to be used. Thus better not hardcode it.
If you did not hardcode the chain yourself, you what ACME client and version are you using? Many years ago, when LetsEncrypt first launched, most clients implemented the anti-pattern of hardcoding chains. Pretty much all had fixed themselves the first time Intermediates or Roots had changed, but some projects were abandoned and some packaging systems were never updated.