Mozilla's certificate bundle is out of date

Help
22
  • For the long chain, many clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, they will proceed to the ISRG Root X1 intermediate certificate and look for the self-signed DST Root CA X3 trust anchor in their trust stores. If said trust anchor is found on an older Android device, the expiration of the self-signed DST Root CA X3 trust anchor will be ignored and the chain validation will succeed. Otherwise, the chain validation will fail.
  • For the short chain, clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, the chain validation will fail.

What follows is the evidence that is available to me.

bundle.pem

This is Mozilla's bundle, generated by the patched mk-ca-bundle.pl, that is, without the 'DST Root CA X3 certificate.

This is the certificate chain:

> echo QUIT | openssl s_client \
-CAfile bundle.pem \
-connect [ip-address-of-example.com]:443 \
-servername [example.com] \
-showcerts \
>chain.txt

Inside chain.txt there are three certificates:

level0.pem:

subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R3

level1.pem:

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

level2.pem:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

Is the chain valid? Let see...

> grep 'ISRG Root X1' bundle.pem
ISRG Root X1

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

> cat level1.pem >>bundle.pem

>openssl verify -verbose -CAfile bundle.pem level0.pem
level0.pem: OK

OK

Let us add the live certificate to the bundle. The live certificate was returned by openssl's code above (chain.txt) is signed by "DST Root CA X3".

>cat level2.pem >>bundle.pem

>openssl verify -verbose -CAfile bundle.pem level0.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error level0.pem: verification failed

This fails because Mozilla's bundle no longer trusts "DST Root CA X3".

If "DST Root CA X3" is in the bundle, as currently the case with Firefox, the verification fails as described by the first post in this thread.

These last two sentences should help clarify the problem at hand.

In summary, the chain verification works only if you ignore the live certificate and use Mozilla's bundled one instead.

This whole problem will find solution when the live certificate aligns to the self-signed certificate in the bundle, that is, when openssl/libressl's command above will return the "ISRG Root X1" self-signed certificate.

On Firefox, I updated my version yesterday, and it works.

My Chromium-based browsers worked on the 4th of October.

I answered my own question.

23

As repeated: including the ISRG Root X1-signed-by-DST Root CA X3 in the chain is by design. Please read Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt. Server operators can choose between the default chain as stated or choose the alternative shorter chain without the ISRG Root X1-signed-by-DST Root CA X3. This might depends on the ACME client used.

Further more, as far as I can tell, DST Root CA X3 is not present in the Mozilla bundle:

osiris@erazer tmp $ wget https://raw.githubusercontent.com/curl/curl/master/lib/mk-ca-bundle.pl
--2021-10-08 12:25:59--  https://raw.githubusercontent.com/curl/curl/master/lib/mk-ca-bundle.pl
Resolving raw.githubusercontent.com... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21394 (21K) [text/plain]
Saving to: ‘mk-ca-bundle.pl’

mk-ca-bundle.pl               100%[=================================================>]  20.89K  --.-KB/s    in 0s      

2021-10-08 12:25:59 (42.1 MB/s) - ‘mk-ca-bundle.pl’ saved [21394/21394]

osiris@erazer tmp $ chmod +x mk-ca-bundle.pl 
osiris@erazer tmp $ ./mk-ca-bundle.pl -f -i -l -b ./ca-bundle.crt
==============================================================================
Script Version                   : 1.28
Perl Version                     : 5.034000
Operating System Name            : linux
Getopt::Std.pm Version           : 1.13
Encode::Encoding.pm Version      : 2.08
MIME::Base64.pm Version          : 3.16
LWP::UserAgent.pm Version        : 6.55
LWP.pm Version                   : 6.55
Digest::SHA.pm Version           : 6.02
==============================================================================
SHA256 of old file: 0
Downloading certdata.txt ...
Get certdata with curl!
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1173k  100 1173k    0     0  83591      0  0:00:14  0:00:14 --:--:-- 87601
Downloaded certdata.txt
SHA256 of new file: c8f6733d1ff4e6a4769c182971a1234f95ae079247a9c439a13423fe8ba5c24f
Processing  'certdata.txt' ...
Done (127 CA certs processed, 25 skipped).
osiris@erazer tmp $ grep 'DST Root CA X3' ca-bundle.crt
osiris@erazer tmp $

The commands for mk-ca-bundle.pl are a literal copy/paste from your post earlier.

Also, it looks like this thread is going to be a broken record. If it actually becomes a broken record without any substantial and notable input, I'm enclined to close it.

4 Likes
24

As repeated: including the ISRG Root X1-signed-by-DST Root CA X3 in the chain is by design.

Please read carefully my last post.

25

I did read it. The only thing I noticed out of the ordinary is the claim that Mozillas bundle contains the DST Root CA X3, which I cannot verify.

Further more, failing verification with openssl verify is to be expected, as DST Root CA X3 has indeed expired.

3 Likes
26

Technically it did prior to September 30, 2021 update fix. That has been fixed if you update to latest version.

3 Likes
27

OP was posted on Oct 4, 8:29 PM.

4 Likes
28

Self-signed root:

Cross-signed root:

If one doesn't work for you, then just use the other.
If both don't work (well enough) for you, then switch to another ACME friendly (free) CA.
[which will, no doubt, be using a completely different trust path]

2 Likes
29

If you go to Mozilla's observatory, section "TLS Observatory", it says "This site uses an untrusted or invalid certificate.":

https://observatory.mozilla.org/analyze/community.letsencrypt.org#tls

If you further select "certificate explainer", ISRG Root X1 is shown as intermediate:

https://tls-observatory.services.mozilla.com/static/certsplainer.html?id=188677175

This shows that the problem is still felt around.

Update: Now shows as root, but the previous page still shows the error message.

30

That's probably due to the presence of the ISRG Root X1-signed-by-DST Root CA X3 intermediate certificate in the chain.

Sites specifically designed to check a certificate chain might indeed choke on the expired DST Root CA X3. That doesn't mean there's an actual problem though.

Please elaborate more about what actual problem you have. Because 30 posts later we've already established that many sites using Let's Encrypt certificates use the default certificate chain which contains the cross-signed ISRG Root X1-signed-by-DST Root CA X3. And that this is by design for specific Android compatibility. We've also already established that the Mozilla root store contained, but currently does not contain since the latest update, the now-expired DST Root CA X3.

Can we now please focus about the actual problem you're having with your Firefox?

2 Likes
31

Reading the entirety of the thread straight through, the crux of these problems seems to be that you appear to lack a working knowledge of OpenSSL and Certificate technology, while believing that you possess it. This is apparent across many of the comments, but this one best illustrates it:

The public keys must be extracted from the certificates. You have had the proof all along, but you failed to look at it. Two of the easiest methods:

  • Display the public key in PEM format:

    openssl x509 -pubkey -in /path/to/chain -noout

  • Display the cert in TEXT format, look at the "Subject Public Key Info" section

    openssl x509 -text -in /path/to/chain -noout

I suggest you re-examine your understanding of SSL Certificates and OpenSSL. The other posters have exhibited great patience and virtue in trying to explain and re-explain the same points to you. With a better understanding of these concepts, you would likely have no problems.

5 Likes
32

By definition, both the self-signed ISRG Root X1 root certificate and the cross-signed ISRG Root X1 intermediate certificate (signed by DST Root CA X3) use the same private key for signing intermediate certificates (R3 and R4) and therefore contain the same public key. This is very easy to verify. Just decode both certificates:

https://redkestrel.co.uk/products/decoder/

https://letsencrypt.org/certs/isrgrootx1.pem

Issuer: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Validity
Not Before: Jun 4 11:04:38 2015 GMT
Not After: Jun 4 11:04:38 2035 GMT
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:
7e:3c:70:bf:ad:29:22:06:f3:e4:c0:e6:80:ae:e2:
4b:8f:b7:99:7e:94:03:9f:d3:47:97:7c:99:48:23:
53:e8:38:ae:4f:0a:6f:83:2e:d1:49:57:8c:80:74:
b6:da:2f:d0:38:8d:7b:03:70:21:1b:75:f2:30:3c:
fa:8f:ae:dd:da:63:ab:eb:16:4f:c2:8e:11:4b:7e:
cf:0b:e8:ff:b5:77:2e:f4:b2:7b:4a:e0:4c:12:25:
0c:70:8d:03:29:a0:e1:53:24:ec:13:d9:ee:19:bf:
10:b3:4a:8c:3f:89:a3:61:51:de:ac:87:07:94:f4:
63:71:ec:2e:e2:6f:5b:98:81:e1:89:5c:34:79:6c:
76:ef:3b:90:62:79:e6:db:a4:9a:2f:26:c5:d0:10:
e1:0e:de:d9:10:8e:16:fb:b7:f7:a8:f7:c7:e5:02:
07:98:8f:36:08:95:e7:e2:37:96:0d:36:75:9e:fb:
0e:72:b1:1d:9b:bc:03:f9:49:05:d8:81:dd:05:b4:
2a:d6:41:e9:ac:01:76:95:0a:0f:d8:df:d5:bd:12:
1f:35:2f:28:17:6c:d2:98:c1:a8:09:64:77:6e:47:
37:ba:ce:ac:59:5e:68:9d:7f:72:d6:89:c5:06:41:
29:3e:59:3e:dd:26:f5:24:c9:11:a7:5a:a3:4c:40:
1f:46:a1:99:b5:a7:3a:51:6e:86:3b:9e:7d:72:a7:
12:05:78:59:ed:3e:51:78:15:0b:03:8f:8d:d0:2f:
05:b2:3e:7b:4a:1c:4b:73:05:12:fc:c6:ea:e0:50:
13:7c:43:93:74:b3:ca:74:e7:8e:1f:01:08:d0:30:
d4:5b:71:36:b4:07:ba:c1:30:30:5c:48:b7:82:3b:
98:a6:7d:60:8a:a2:a3:29:82:cc:ba:bd:83:04:1b:
a2:83:03:41:a1:d6:05:f1:1b:c2:b6:f0:a8:7c:86:
3b:46:a8:48:2a:88:dc:76:9a:76:bf:1f:6a:a5:3d:
19:8f:eb:38:f3:64:de:c8:2b:0d:0a:28:ff:f7:db:
e2:15:42:d4:22:d0:27:5d:e1:79:fe:18:e7:70:88:
ad:4e:e6:d9:8b:3a:c6:dd:27:51:6e:ff:bc:64:f5:
33:43:4f
Exponent: 65537 (0x10001)

https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem

Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Validity
Not Before: Jan 20 19:14:03 2021 GMT
Not After: Sep 30 18:14:03 2024 GMT
Subject: C=US, O=Internet Security Research Group, CN=ISRG Root X1
Subject Public Key Info:
Public Key Algorithm: rsaEncryption
RSA Public-Key: (4096 bit)
Modulus:
00:ad:e8:24:73:f4:14:37:f3:9b:9e:2b:57:28:1c:
87:be:dc:b7:df:38:90:8c:6e:3c:e6:57:a0:78:f7:
75:c2:a2:fe:f5:6a:6e:f6:00:4f:28:db:de:68:86:
6c:44:93:b6:b1:63:fd:14:12:6b:bf:1f:d2:ea:31:
9b:21:7e:d1:33:3c:ba:48:f5:dd:79:df:b3:b8:ff:
12:f1:21:9a:4b:c1:8a:86:71:69:4a:66:66:6c:8f:
7e:3c:70:bf:ad:29:22:06:f3:e4:c0:e6:80:ae:e2:
4b:8f:b7:99:7e:94:03:9f:d3:47:97:7c:99:48:23:
53:e8:38:ae:4f:0a:6f:83:2e:d1:49:57:8c:80:74:
b6:da:2f:d0:38:8d:7b:03:70:21:1b:75:f2:30:3c:
fa:8f:ae:dd:da:63:ab:eb:16:4f:c2:8e:11:4b:7e:
cf:0b:e8:ff:b5:77:2e:f4:b2:7b:4a:e0:4c:12:25:
0c:70:8d:03:29:a0:e1:53:24:ec:13:d9:ee:19:bf:
10:b3:4a:8c:3f:89:a3:61:51:de:ac:87:07:94:f4:
63:71:ec:2e:e2:6f:5b:98:81:e1:89:5c:34:79:6c:
76:ef:3b:90:62:79:e6:db:a4:9a:2f:26:c5:d0:10:
e1:0e:de:d9:10:8e:16:fb:b7:f7:a8:f7:c7:e5:02:
07:98:8f:36:08:95:e7:e2:37:96:0d:36:75:9e:fb:
0e:72:b1:1d:9b:bc:03:f9:49:05:d8:81:dd:05:b4:
2a:d6:41:e9:ac:01:76:95:0a:0f:d8:df:d5:bd:12:
1f:35:2f:28:17:6c:d2:98:c1:a8:09:64:77:6e:47:
37:ba:ce:ac:59:5e:68:9d:7f:72:d6:89:c5:06:41:
29:3e:59:3e:dd:26:f5:24:c9:11:a7:5a:a3:4c:40:
1f:46:a1:99:b5:a7:3a:51:6e:86:3b:9e:7d:72:a7:
12:05:78:59:ed:3e:51:78:15:0b:03:8f:8d:d0:2f:
05:b2:3e:7b:4a:1c:4b:73:05:12:fc:c6:ea:e0:50:
13:7c:43:93:74:b3:ca:74:e7:8e:1f:01:08:d0:30:
d4:5b:71:36:b4:07:ba:c1:30:30:5c:48:b7:82:3b:
98:a6:7d:60:8a:a2:a3:29:82:cc:ba:bd:83:04:1b:
a2:83:03:41:a1:d6:05:f1:1b:c2:b6:f0:a8:7c:86:
3b:46:a8:48:2a:88:dc:76:9a:76:bf:1f:6a:a5:3d:
19:8f:eb:38:f3:64:de:c8:2b:0d:0a:28:ff:f7:db:
e2:15:42:d4:22:d0:27:5d:e1:79:fe:18:e7:70:88:
ad:4e:e6:d9:8b:3a:c6:dd:27:51:6e:ff:bc:64:f5:
33:43:4f
Exponent: 65537 (0x10001)
3 Likes
33

And you are still failing to see that they are different certificates, because one is self-signed while the other is signed by DST Root CA X3. Repeating over and over again something false does not make it true. The certificates are not the same.

34

:woozy_face:

I just proved @jvanasco's point in black and white.

Please compare the modulus and exponent of both keys.

3 Likes
35

The certificates are not, but the signing keypair called "ISRG Root X1" is.

4 Likes
36

Griffin, going all the way to check whether they were generated by the same private key is not really going to help here. I can sign different documents with the same private key, and this does not make them the same document. Also, their private key is private, we have no access to it.

What I see, and have shown here, is that

  • leaf validation succeeds against the new Mozilla's bundle (which contains the self-signed ISRG Root X1) + the intermediate R3. it fails against the same bundle + the live certificate;

  • leaf validation fails when using the live certificate for "ISRG Root X1". Openssl, with the standard code I have given above, returns the live certificate. So, until the live certificate is cross-signed, the problem will keep occurring. This holds also on Mozilla's own observatory and SSL Labs test site.

Everybody, this thread is now closed.

37

The certificates are not, but the signing keypair called "ISRG Root X1" is .

I am interested in the certificates, because they are used in the chain validation, which is the problem I discussed. And they are different, as they lead to different results in the validation. Stating that they have the same signing keypar does not change the fact that one validates while the other fails.

Enough here. I am happy with my findings and the solution I have in place.

So long, and thanks for all the fish.

38

The Certificates are not the same, and a person with ordinary skill in the arts of SSL Certificate technology would recognize that is not what I - or anyone else in this thread - have said.

Again, I strongly suggest you better educate yourself on SSL Certificate technology as your issues all appear to stem from fundamental misunderstandings of it and related concepts.

3 Likes
39

@patch-work The fact that openssl verify fails is to be expected. However, that does not mean OpenSSL (or other applications) as a client fails. OpenSSL newer than 1.0.2 (I.e., 1.1.x) works out of the box with the chain containing ISRG Root X1-signed-by-the-expired-DST Root CA X3. OpenSSL 1.0.2 can work, but can require some manipulation on the client host or server (see Old Let’s Encrypt Root Certificate Expiration and OpenSSL 1.0.2 - OpenSSL Blog).

In any case, this thread has become a broken record. Therefore, I'm closing it.

5 Likes