Missing new Chains of Trusts certs in ca-certificates bundle

Help
1

Hello all,

Following a recent certificate renewal, we have started experiencing certificate validation failures.

It appears that the latest ca-certificates package bundle shipped with Ubuntu (20240203 : ca-certificates package : Debian) does not support sites issued with the new Let’s Encrypt R10 certificate. When I manually specify the certificate file for the client (Chains of Trust - Let's Encrypt), it works fine.

Output from openssl s_client:

Certificate chain
 0 s:CN = <REDACTED>
   i:C = US, O = Let's Encrypt, CN = R10
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: <REDACTED> GMT; NotAfter: Dec 22 <REDACTED> 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R11
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
...
Verify return code: 21 (unable to verify the first certificate)

Is there something that can be done via this forum, or should I file a bug in Debian/Ubuntu?

2

Intermediates (like R10) generally should not be included in the ca-certificates package.

Your s_client output shows a misconfiguration: You have a certificate signed by R10, but then an intermediate R11. This is why it is unable to build a chain from the leaf to root, as it needs the R10-signed-by-X1 intermediate.

How to fix this depends on your ACME client and how your webserver is configured. We can provide advice if you can tell us what software you're using, and show us the configuration of TLS in your webserver.

5 Likes
3

Issuer of the second cert is ISRG Root X1.

Are you referring to the subject not being equal? because this is more/less the same case with: acme-v02.api.letsencrypt.org (official API of Let's Encrypt):

Certificate chain
 0 s:CN = acme-v02.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R10
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep  4 14:46:39 2024 GMT; NotAfter: Dec  3 14:46:38 2024 GMT
 1 s:C = US, O = Let's Encrypt, CN = R10
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Mar 13 00:00:00 2024 GMT; NotAfter: Mar 12 23:59:59 2027 GMT
4

On the chain you posted, the issuer is:

i:C = US, O = Let's Encrypt, CN = R10

The intermediate has subject:

1 s:C = US, O = Let's Encrypt, CN = R11

That is the incorrect intermediate. In the example from our own API, they are both R10.

5 Likes
5

Ok, I've found the issue.

The intermediate cert configured in the server was set to R11, but it is wrong, it suppose to be the R10.

Are there any official instructions on how to properly configure Apache HTTPD with Let's Encrypt auto-renewal?

6

Usually your ACME client just handles all that automatically, and you just configure your web server to use the certificate and chain that it gives you. You'd need to say more details about what software you're using to get the certificate and how you have it set up.

5 Likes
7

Yeah, don't manually hardcode intermediate certificates. That's the official instruction. Let the ACME client do all the work, as it gets the correct intermediate from the ACME server at issuance.

Note that the intermediates are randomly chosen between R10 and R11 (or E5 and E6 for ECDSA certs with three intermediate per key type as spares) and there's no way to know which intermediate is going to be used. Thus better not hardcode it.

6 Likes
8

Adding to what everyone else said above:

That situation means one of two things:

  1. You hardcoded the Intermediate Chain
  2. Your ACME Client hardcoded the Intermediate Chain

If you did not hardcode the chain yourself, you what ACME client and version are you using? Many years ago, when LetsEncrypt first launched, most clients implemented the anti-pattern of hardcoding chains. Pretty much all had fixed themselves the first time Intermediates or Roots had changed, but some projects were abandoned and some packaging systems were never updated.

4 Likes
9

The configuration was changed recently and wrongly to use an hardcoded CA chain.

I'm working now to change the settings to not hardcode the Intermediate CA Chain.

Thanks everyone!

5 Likes
10

If you run into any problems or have questions, please ask for guidance!

4 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.