Mozilla's certificate bundle is out of date

Help
1

WARNING:

Mozilla's bundle of trusted CA includes the following out-of-date certificate. This holds for the latest OpenBSD's default (/etc/ssl/cert.pem), as well as the manually downloaded bundle from Mozilla (via curl's mk-ca-bundle.pl).

As a result, firefox is currently returning "Error code: SEC_ERROR_UNKNOWN_ISSUER" for all letsencrypted websites.

Digital Signature Trust Co.

=== /O=Digital Signature Trust Co./CN=DST Root CA X3
Certificate:
Data:
Version: 3 (0x2)
Serial Number:
44:af:b0:80:d6:a3:27:ba:89:30:39:86:2e:f8:40:6b
Signature Algorithm: sha1WithRSAEncryption
Validity
Not Before: Sep 30 21:12:19 2000 GMT
Not After : Sep 30 14:01:15 2021 GMT
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
X509v3 extensions:
X509v3 Basic Constraints: critical
CA:TRUE
X509v3 Key Usage: critical
Certificate Sign, CRL Sign
X509v3 Subject Key Identifier:
C4:A7:B1:A4:7B:2C:71:FA:DB:E1:4B:90:75:FF:C4:15:60:85:89:10
SHA1 Fingerprint=DA:C9:02:4F:54:D8:F6:DF:94:93:5F:B1:73:26:38:CA:6A:D7:7C:13
SHA256 Fingerprint=06:87:26:03:31:A7:24:03:D9:09:F1:05:E6:9B:CF:0D:32:E1:BD:24:93:FF:C6:D9:20:6D:11:BC:D6:77:07:39
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

1 Like
2

How did you retrieve Mozillas bundle using that? I just tried ./mk-ca-bundle.pl with the default options which uses the release bundle and I used -d nss which uses the NSS bundle. Neither bundles contain the DST Root CA X3 certificate in my case.

3 Likes
3

$ mk-ca-bundle.pl -f -i -l -b ./ca-bundle.crt
[...]
$ grep 'DST Root CA X3' ca-bundle.crt
DST Root CA X3

Besides, the new certificates from letsencrypt still include DST Root CA X3 as Issuer.

1 Like
4

Might have caught it before their recent update/bug fix https://twitter.com/bagder/status/1443685495360393217

3 Likes
5

The new certificates from letsencrypt still include DST Root CA X3 as Issuer.

1 Like
6

Yes, that's by design.

4 Likes
7

Well, it causes problems. If you read a live certificate chain, the last certificate in the chain is still DST Root CA X3. If you have the non-patched mozilla bundle, verification will fail because the trusted CA is out-of-date. If you use the patched version, verification still fails because you hit a non trusted CA.

2 Likes
8

Yes, that's by design.

It can't possibly be by design that you're still including a expired root in the chain, it breaks the whole chain

1 Like
9

@PhilV Please read Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt before making assumptions.

@patch-work Most clients will work fine with this default chain and will terminate the validation successfully when finding the R3-signed-by-ISRG Root X1. Only a few problematic clients such as OpenSSL 1.0.2 (or older I believe) are causing trouble. Also, see the link above.

4 Likes
10

when finding the R3-signed-by-ISRG Root X1

Osris, it happens to be the other way around: the last in the chain is ISRG Root X1, issued by DST Root CA X3 with a self-signed expired certificate.

Also the CRL is failing:

CRL ERROR: HTTP request failed with status code 404: http://crl.identrust.com/DSTROOTCAX3CRL.crl

Look at the certification paths here:

https://www.ssllabs.com/ssltest/analyze.html?d=community.letsencrypt.org&s=64.71.144.202&hideResults=on&latest

It is a fact that there are problems in the chain that need a loving hand.

11

It's true that some TLS clients won't stop at finding the R3-signed-by-ISRG Root X1 intermediate even when they have a trust anchor available in their root store. Indeed, some clients continue with the validation and end up at the expired DST Root CA X3. And some clients are fine with that (older Android) and some clients are NOT fine by that.

Please read the document I've linked above. Using the default chain as it is now was a deliberate choice as said before. As a server operator you have the choice to select the alternative chain without DST Root CA X3.

4 Likes
13

Osiris: "It's true that some TLS clients won't stop at finding the R3-signed-by-ISRG Root X1 intermediate even when they have a trust anchor available in their root store."

This is Mozilla's root store as retrieved by the patched mk-ca-bundle.pl:

> grep 'DST Root CA X3' ca-bundle.pem
[empty]

> grep ' R3' ca-bundle.pem
GlobalSign Root CA - R3
GTS Root R3

None of them is signed by ISRG:

subject=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
issuer=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign

subject=C = US, O = Google Trust Services LLC, CN = GTS Root R3
issuer=C = US, O = Google Trust Services LLC, CN = GTS Root R3

14

Sooo, you're saying, all is good now? The DST Root CA X3 is gone?

1 Like
15

I just said that what you said is wrong.

16

Osiris: "Sooo, you're saying, all is good now? The DST Root CA X3 is gone?"

The DST Root CA X3 is gone from the bundle, but the chain still fails validation.

This is the live chain:

level0.pem

subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R3
notBefore=Oct 4 12:30:45 2021 GMT
notAfter=Jan 2 12:30:44 2022 GMT

level1.pem

subject=C = US, O = Let's Encrypt, CN = R3
i>ssuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
serial=912B084ACF0C18A753F6D62E25A75F5A
notBefore=Sep 4 00:00:00 2020 GMT
notAfter=Sep 15 16:00:00 2025 GMT

level2.pem

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
serial=4001772137D4E942B8EE76AA3C640AB7
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT

level3.pem
This is DST Root CA X3, which is now empty.

If you check level0.pem against the bundle+level1.pem+level2.pem, it fails.

"R3" (level1.pem) does not occur in Mozilla's bundle.

Also, if I extract "ISRG Root X1" from Mozilla's bundle and compare it with the live certificate, I see they are different.

This is the live certificate:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
serial=4001772137D4E942B8EE76AA3C640AB7
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT

This is the cert inside Mozilla's bundle:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
serial=8210CFB0D240E3594463E0BB63828B00
notBefore=Jun 4 11:04:38 2015 GMT
notAfter=Jun 4 11:04:38 2035 GMT

17

Let's Encrypts R3 certificate is an intermediate. It is not included in any trust stores. The certificates you have listed have nothing to do with Let's Encrypt. You should check that ISRG Root X1 is present in your bundle.

We are aware that Let's Encrypts default chain contains a cross-signed version of ISRG Root X1 that leads up to DST Root CA X3. This is intentional. This workaround ensures Android compatibility, because Android devices do not care about DST Root CA X3's expiry.

Devices that do care can build a chain up to ISRG Root X1, which is a) also a root and b) is part of the verification chain. You can easily build a chain up to ISRG Root X1, ignoring the expired root certificate. Most modern TLS implementations do this.

We're aware that there are broken TLS libraries (old OpenSSL, GnuTLS, LibreSSL, WolfSSL...) that can't do this. For these libraries, solutions are either:

  • Patching the libraries to support better path building logics (usually this means simply upgrading them to recent versions)
  • Applying workarounds, such as removing DST Root CA X3 from the trust stores.

Having ISRG Root X1 in the trust store is a requirement in any case, except for Android devices.

PS: I forgot to mention that it is possible to request a chain that does not include the final cross-sign up to DST Root CA X3 (instead terminating at ISRG Root X1). This is what Let's Encrypt calls the alternate chain. This chain is not compatible with Android < 7.1, but is compatible with aforementioned broken libraries.

6 Likes
18

Supplementing the excellent information that @Nummer378 has just presented...

long/default chain chain:

  1. leaf certificate
  2. R3 signed by ISRG Root X1
  3. ISRG Root X1 signed by DST Root CA X3

short/alternate chain:

  1. leaf certificate
  2. R3 signed by ISRG Root X1

This ISRG Root X1 intermediate certificate is cross-signed by DST Root CA X3:

This ISRG Root X1 root certificate is self-signed:

  • Both ISRG Root X1 certificates contain the same public key.
  • For the long chain, many clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, they will proceed to the ISRG Root X1 intermediate certificate and look for the self-signed DST Root CA X3 trust anchor in their trust stores. If said trust anchor is found on an older Android device, the expiration of the self-signed DST Root CA X3 trust anchor will be ignored and the chain validation will succeed. Otherwise, the chain validation will fail.
  • For the short chain, clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, the chain validation will fail.
4 Likes
19

Let's Encrypts R3 certificate is an intermediate. It is not included in any trust stores.

Osiris suggested otherwise, and I proved him wrong with evidence of the fact.

The certificates you have listed have nothing to do with Let's Encrypt.

Both level0.pem (the leaf) and level1.pem (the intermediate) have strictly to do with Let's Encrypt.

level2+ are related to Let's Encrypt, as they are the issuers.

You should check that ISRG Root X1 is present in your bundle.

You seem to have an attention deficit, because I have just shown the existence of two different certificates for "ISRG Root X1", one occurs in Mozilla's bundle, the other is live.

We are aware that Let's Encrypts default chain contains a cross-signed version of ISRG Root X1 that leads up to DST Root CA X3. This is intentional. This workaround ensures Android compatibility, because Android devices do not care about DST Root CA X3's expiry.

I am concerned about certificate chain validation using Openssl and desktop Firefox.

Android compatibility is not my concern at this time.

We're aware that ...

Blah blah blah.... :slight_smile:

Show me a successful openssl chain validation instead, with its command line script.

20
  • Both ISRG Root X1 certificates contain the same public key.

I have no evidence of what you are saying.

This is Mozilla's:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
serial=8210CFB0D240E3594463E0BB63828B00
notBefore=Jun 4 11:04:38 2015 GMT
notAfter=Jun 4 11:04:38 2035 GMT

> head -2 ISRG_Root_X1.pem
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAwTzELMAkGA1UE

And this is the live:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
serial=4001772137D4E942B8EE76AA3C640AB7
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT

> head -2 live2.pem
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/

21

I did no such thing. Either I mistyped or you misinterpreted.

2 Likes