Mozilla's certificate bundle is out of date

Osiris · October 5, 2021, 11:17am

It's true that some TLS clients won't stop at finding the R3-signed-by-ISRG Root X1 intermediate even when they have a trust anchor available in their root store. Indeed, some clients continue with the validation and end up at the expired DST Root CA X3. And some clients are fine with that (older Android) and some clients are NOT fine by that.

Please read the document I've linked above. Using the default chain as it is now was a deliberate choice as said before. As a server operator you have the choice to select the alternative chain without DST Root CA X3.

patch-work · October 7, 2021, 5:20pm

Osiris: "It's true that some TLS clients won't stop at finding the R3-signed-by-ISRG Root X1 intermediate even when they have a trust anchor available in their root store."

This is Mozilla's root store as retrieved by the patched mk-ca-bundle.pl:

> grep 'DST Root CA X3' ca-bundle.pem
[empty]

> grep ' R3' ca-bundle.pem
GlobalSign Root CA - R3
GTS Root R3

None of them is signed by ISRG:

subject=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign
issuer=OU = GlobalSign Root CA - R3, O = GlobalSign, CN = GlobalSign

subject=C = US, O = Google Trust Services LLC, CN = GTS Root R3
issuer=C = US, O = Google Trust Services LLC, CN = GTS Root R3

Osiris · October 7, 2021, 5:22pm

Sooo, you're saying, all is good now? The DST Root CA X3 is gone?

patch-work · October 7, 2021, 5:23pm

I just said that what you said is wrong.

patch-work · October 7, 2021, 5:23pm

Osiris: "Sooo, you're saying, all is good now? The DST Root CA X3 is gone?"

The DST Root CA X3 is gone from the bundle, but the chain still fails validation.

This is the live chain:

level0.pem

subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R3
notBefore=Oct 4 12:30:45 2021 GMT
notAfter=Jan 2 12:30:44 2022 GMT

level1.pem

subject=C = US, O = Let's Encrypt, CN = R3
i>ssuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
serial=912B084ACF0C18A753F6D62E25A75F5A
notBefore=Sep 4 00:00:00 2020 GMT
notAfter=Sep 15 16:00:00 2025 GMT

level2.pem

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
serial=4001772137D4E942B8EE76AA3C640AB7
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT

level3.pem
This is DST Root CA X3, which is now empty.

If you check level0.pem against the bundle+level1.pem+level2.pem, it fails.

"R3" (level1.pem) does not occur in Mozilla's bundle.

Also, if I extract "ISRG Root X1" from Mozilla's bundle and compare it with the live certificate, I see they are different.

This is the live certificate:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
serial=4001772137D4E942B8EE76AA3C640AB7
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT

This is the cert inside Mozilla's bundle:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
serial=8210CFB0D240E3594463E0BB63828B00
notBefore=Jun 4 11:04:38 2015 GMT
notAfter=Jun 4 11:04:38 2035 GMT

Nummer378 · October 7, 2021, 5:33pm

Let's Encrypts R3 certificate is an intermediate. It is not included in any trust stores. The certificates you have listed have nothing to do with Let's Encrypt. You should check that ISRG Root X1 is present in your bundle.

We are aware that Let's Encrypts default chain contains a cross-signed version of ISRG Root X1 that leads up to DST Root CA X3. This is intentional. This workaround ensures Android compatibility, because Android devices do not care about DST Root CA X3's expiry.

Devices that do care can build a chain up to ISRG Root X1, which is a) also a root and b) is part of the verification chain. You can easily build a chain up to ISRG Root X1, ignoring the expired root certificate. Most modern TLS implementations do this.

We're aware that there are broken TLS libraries (old OpenSSL, GnuTLS, LibreSSL, WolfSSL...) that can't do this. For these libraries, solutions are either:

Patching the libraries to support better path building logics (usually this means simply upgrading them to recent versions)
Applying workarounds, such as removing DST Root CA X3 from the trust stores.

Having ISRG Root X1 in the trust store is a requirement in any case, except for Android devices.

PS: I forgot to mention that it is possible to request a chain that does not include the final cross-sign up to DST Root CA X3 (instead terminating at ISRG Root X1). This is what Let's Encrypt calls the alternate chain. This chain is not compatible with Android < 7.1, but is compatible with aforementioned broken libraries.

griffin · October 7, 2021, 7:34pm

Supplementing the excellent information that @Nummer378 has just presented...

long/default chain chain:

short/alternate chain:

leaf certificate
R3 signed by ISRG Root X1

This ISRG Root X1 intermediate certificate is cross-signed by DST Root CA X3:

This ISRG Root X1 root certificate is self-signed:

Both ISRG Root X1 certificates contain the same public key.
For the long chain, many clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, they will proceed to the ISRG Root X1 intermediate certificate and look for the self-signed DST Root CA X3 trust anchor in their trust stores. If said trust anchor is found on an older Android device, the expiration of the self-signed DST Root CA X3 trust anchor will be ignored and the chain validation will succeed. Otherwise, the chain validation will fail.
For the short chain, clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, the chain validation will fail.

patch-work · October 8, 2021, 9:03am

Let's Encrypts R3 certificate is an intermediate. It is not included in any trust stores.

Osiris suggested otherwise, and I proved him wrong with evidence of the fact.

The certificates you have listed have nothing to do with Let's Encrypt.

Both level0.pem (the leaf) and level1.pem (the intermediate) have strictly to do with Let's Encrypt.

level2+ are related to Let's Encrypt, as they are the issuers.

You should check that ISRG Root X1 is present in your bundle.

You seem to have an attention deficit, because I have just shown the existence of two different certificates for "ISRG Root X1", one occurs in Mozilla's bundle, the other is live.

We are aware that Let's Encrypts default chain contains a cross-signed version of ISRG Root X1 that leads up to DST Root CA X3. This is intentional. This workaround ensures Android compatibility, because Android devices do not care about DST Root CA X3's expiry.

I am concerned about certificate chain validation using Openssl and desktop Firefox.

Android compatibility is not my concern at this time.

We're aware that ...

Blah blah blah....

Show me a successful openssl chain validation instead, with its command line script.

patch-work · October 8, 2021, 9:13am

Both ISRG Root X1 certificates contain the same public key.

I have no evidence of what you are saying.

This is Mozilla's:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1
serial=8210CFB0D240E3594463E0BB63828B00
notBefore=Jun 4 11:04:38 2015 GMT
notAfter=Jun 4 11:04:38 2035 GMT

> head -2 ISRG_Root_X1.pem
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAwTzELMAkGA1UE

And this is the live:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3
serial=4001772137D4E942B8EE76AA3C640AB7
notBefore=Jan 20 19:14:03 2021 GMT
notAfter=Sep 30 18:14:03 2024 GMT

> head -2 live2.pem
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/

Osiris · October 8, 2021, 9:44am

I did no such thing. Either I mistyped or you misinterpreted.

patch-work · October 8, 2021, 9:56am

For the long chain, many clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, they will proceed to the ISRG Root X1 intermediate certificate and look for the self-signed DST Root CA X3 trust anchor in their trust stores. If said trust anchor is found on an older Android device, the expiration of the self-signed DST Root CA X3 trust anchor will be ignored and the chain validation will succeed. Otherwise, the chain validation will fail.

For the short chain, clients/browsers will work down the chain from the leaf certificate until they encounter the R3 intermediate certificate signed by ISRG Root X1 and look for the self-signed ISRG Root X1 trust anchor in their trust stores. If said trust anchor is found, the chain validation will succeed. Otherwise, the chain validation will fail.

What follows is the evidence that is available to me.

bundle.pem

This is Mozilla's bundle, generated by the patched mk-ca-bundle.pl, that is, without the 'DST Root CA X3 certificate.

This is the certificate chain:

> echo QUIT | openssl s_client \
-CAfile bundle.pem \
-connect [ip-address-of-example.com]:443 \
-servername [example.com] \
-showcerts \
>chain.txt

Inside chain.txt there are three certificates:

level0.pem:

subject=CN = example.com
issuer=C = US, O = Let's Encrypt, CN = R3

level1.pem:

subject=C = US, O = Let's Encrypt, CN = R3
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

level2.pem:

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=O = Digital Signature Trust Co., CN = DST Root CA X3

Is the chain valid? Let see...

> grep 'ISRG Root X1' bundle.pem
ISRG Root X1

subject=C = US, O = Internet Security Research Group, CN = ISRG Root X1
issuer=C = US, O = Internet Security Research Group, CN = ISRG Root X1

> cat level1.pem >>bundle.pem

>openssl verify -verbose -CAfile bundle.pem level0.pem
level0.pem: OK

OK

Let us add the live certificate to the bundle. The live certificate was returned by openssl's code above (chain.txt) is signed by "DST Root CA X3".

>cat level2.pem >>bundle.pem

>openssl verify -verbose -CAfile bundle.pem level0.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error level0.pem: verification failed

This fails because Mozilla's bundle no longer trusts "DST Root CA X3".

If "DST Root CA X3" is in the bundle, as currently the case with Firefox, the verification fails as described by the first post in this thread.

These last two sentences should help clarify the problem at hand.

In summary, the chain verification works only if you ignore the live certificate and use Mozilla's bundled one instead.

This whole problem will find solution when the live certificate aligns to the self-signed certificate in the bundle, that is, when openssl/libressl's command above will return the "ISRG Root X1" self-signed certificate.

On Firefox, I updated my version yesterday, and it works.

My Chromium-based browsers worked on the 4th of October.

I answered my own question.

Osiris · October 8, 2021, 10:30am

As repeated: including the ISRG Root X1-signed-by-DST Root CA X3 in the chain is by design. Please read Extending Android Device Compatibility for Let's Encrypt Certificates - Let's Encrypt. Server operators can choose between the default chain as stated or choose the alternative shorter chain without the ISRG Root X1-signed-by-DST Root CA X3. This might depends on the ACME client used.

Further more, as far as I can tell, DST Root CA X3 is not present in the Mozilla bundle:

osiris@erazer tmp $ wget https://raw.githubusercontent.com/curl/curl/master/lib/mk-ca-bundle.pl
--2021-10-08 12:25:59--  https://raw.githubusercontent.com/curl/curl/master/lib/mk-ca-bundle.pl
Resolving raw.githubusercontent.com... 185.199.110.133, 185.199.109.133, 185.199.111.133, ...
Connecting to raw.githubusercontent.com|185.199.110.133|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 21394 (21K) [text/plain]
Saving to: ‘mk-ca-bundle.pl’

mk-ca-bundle.pl               100%[=================================================>]  20.89K  --.-KB/s    in 0s      

2021-10-08 12:25:59 (42.1 MB/s) - ‘mk-ca-bundle.pl’ saved [21394/21394]

osiris@erazer tmp $ chmod +x mk-ca-bundle.pl 
osiris@erazer tmp $ ./mk-ca-bundle.pl -f -i -l -b ./ca-bundle.crt
==============================================================================
Script Version                   : 1.28
Perl Version                     : 5.034000
Operating System Name            : linux
Getopt::Std.pm Version           : 1.13
Encode::Encoding.pm Version      : 2.08
MIME::Base64.pm Version          : 3.16
LWP::UserAgent.pm Version        : 6.55
LWP.pm Version                   : 6.55
Digest::SHA.pm Version           : 6.02
==============================================================================
SHA256 of old file: 0
Downloading certdata.txt ...
Get certdata with curl!
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 1173k  100 1173k    0     0  83591      0  0:00:14  0:00:14 --:--:-- 87601
Downloaded certdata.txt
SHA256 of new file: c8f6733d1ff4e6a4769c182971a1234f95ae079247a9c439a13423fe8ba5c24f
Processing  'certdata.txt' ...
Done (127 CA certs processed, 25 skipped).
osiris@erazer tmp $ grep 'DST Root CA X3' ca-bundle.crt
osiris@erazer tmp $

The commands for mk-ca-bundle.pl are a literal copy/paste from your post earlier.

Also, it looks like this thread is going to be a broken record. If it actually becomes a broken record without any substantial and notable input, I'm enclined to close it.

patch-work · October 8, 2021, 10:36am

As repeated: including the ISRG Root X1-signed-by-DST Root CA X3 in the chain is by design.

Please read carefully my last post.

Osiris · October 8, 2021, 10:38am

I did read it. The only thing I noticed out of the ordinary is the claim that Mozillas bundle contains the DST Root CA X3, which I cannot verify.

Further more, failing verification with openssl verify is to be expected, as DST Root CA X3 has indeed expired.

eva2000 · October 8, 2021, 11:03am

Technically it did prior to September 30, 2021 update fix. That has been fixed if you update to latest version.

Osiris · October 8, 2021, 11:04am

OP was posted on Oct 4, 8:29 PM.

rg305 · October 8, 2021, 11:24am

Self-signed root:

Cross-signed root:

If one doesn't work for you, then just use the other.
If both don't work (well enough) for you, then switch to another ACME friendly (free) CA.
[which will, no doubt, be using a completely different trust path]

patch-work · October 8, 2021, 3:35pm

If you go to Mozilla's observatory, section "TLS Observatory", it says "This site uses an untrusted or invalid certificate.":

https://observatory.mozilla.org/analyze/community.letsencrypt.org#tls

If you further select "certificate explainer", ISRG Root X1 is shown as intermediate:

https://tls-observatory.services.mozilla.com/static/certsplainer.html?id=188677175

This shows that the problem is still felt around.

Update: Now shows as root, but the previous page still shows the error message.

Osiris · October 8, 2021, 3:43pm

That's probably due to the presence of the ISRG Root X1-signed-by-DST Root CA X3 intermediate certificate in the chain.

Sites specifically designed to check a certificate chain might indeed choke on the expired DST Root CA X3. That doesn't mean there's an actual problem though.

Please elaborate more about what actual problem you have. Because 30 posts later we've already established that many sites using Let's Encrypt certificates use the default certificate chain which contains the cross-signed ISRG Root X1-signed-by-DST Root CA X3. And that this is by design for specific Android compatibility. We've also already established that the Mozilla root store contained, but currently does not contain since the latest update, the now-expired DST Root CA X3.

Can we now please focus about the actual problem you're having with your Firefox?

jvanasco · October 8, 2021, 4:08pm

Reading the entirety of the thread straight through, the crux of these problems seems to be that you appear to lack a working knowledge of OpenSSL and Certificate technology, while believing that you possess it. This is apparent across many of the comments, but this one best illustrates it:

The public keys must be extracted from the certificates. You have had the proof all along, but you failed to look at it. Two of the easiest methods:

Display the public key in PEM format:

openssl x509 -pubkey -in /path/to/chain -noout

Display the cert in TEXT format, look at the "Subject Public Key Info" section
```
openssl x509 -text -in /path/to/chain -noout
```

I suggest you re-examine your understanding of SSL Certificates and OpenSSL. The other posters have exhibited great patience and virtue in trying to explain and re-explain the same points to you. With a better understanding of these concepts, you would likely have no problems.