Chain validation issues with YE/YR under Linux distributions

Help
1

Issue description

Couple of days ago, I noticed some issues at my automated curl requests. The issues appear after a renewal of some LE certificates. I was able to trace back the problems to an incomplete validation of my LE TLS certifications chain. A missing ISRG Root YR cert in my local CA trust store or within the provided cert chain causes the problem. This topic was also referenced in a post YE/YR are not in ca-certificates! by 532910.

I opened this new topic, because I think many people could come across this issue, because many(?) major linux distributions do not provide the ISRG Root YE/YR within ins ca-certificates packages. It's also possible to reproduce this situation with the test websites of the ISRG Root YE/YR certificates, provide by Let's Encrypt > Chains of Trust

  • ISRG Root YE
    [...]
    • Test websites: valid, revoked, expired
  • ISRG Root YR
    [...]
    • Test websites: valid, revoked, expired

(see also: The difficulty of making sure your website is broken - Let's Encrypt)

If these test websites are opened with a browser like firefox, everything is fine. A complete TLS chain validation is able.
(I am not complete sure, why e.g. firefox has no issues validating the chain, because ISRG Root YE/YR is neither inside the ff certificate store (or I missed it searching for it). FF may downloads the missing Root certificate on demand... I do not know ).

A incomplete validation behaivor can be seen using curl under different Linux distributions, because even the testing websites do not provide the complete certificate chain:

# openssl s_client -connect valid.yr.test-certs.letsencrypt.org:443 -showcerts </dev/null 2>/dev/null | grep -E "( s| i):" 
 0 s:
   i:C=US, O=Let's Encrypt, CN=YR1
 1 s:C=US, O=Let's Encrypt, CN=YR1
   i:C=US, O=ISRG, CN=Root YR

Debian 13.5 (trixie)

root@e26f2be3e817:/# cat /etc/os-release 
PRETTY_NAME="Debian GNU/Linux 13 (trixie)"
[...]
DEBIAN_VERSION_FULL=13.5
[...]

root@e26f2be3e817:/# dpkg -l | grep ca-certificates
ii  ca-certificates           20250419                       all          Common CA certificates

root@e26f2be3e817:/# dpkg -L ca-certificates | grep ISRG
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/usr/share/ca-certificates/mozilla/ISRG_Root_X2.crt

root@e26f2be3e817:/# dpkg -L ca-certificates | grep YR  
root@e26f2be3e817:/# dpkg -L ca-certificates | grep YE
# ^_ YE/YR are not in ca-certificates trust store

Broken (insecure) TLS request:

root@e26f2be3e817:/# curl https://valid.yr.test-certs.letsencrypt.org/
curl: (60) SSL certificate problem: unable to get local issuer certificate
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

root@e26f2be3e817:/# curl -v https://valid.yr.test-certs.letsencrypt.org/ 2>&1 | grep CAfile
*  CAfile: /etc/ssl/certs/ca-certificates.crt

Ubuntu (26.04 LTS)

root@e46e8254f45e:/# cat /etc/os-release 
PRETTY_NAME="Ubuntu 26.04 LTS"
[...]

root@e46e8254f45e:/# dpkg -l | grep ca-certificates
ii  ca-certificates                20260223                         all          Common CA certificates

root@e46e8254f45e:/# dpkg -L ca-certificates | grep ISRG
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/usr/share/ca-certificates/mozilla/ISRG_Root_X2.crt

root@e46e8254f45e:/# dpkg -L ca-certificates | grep YR
root@e46e8254f45e:/# dpkg -L ca-certificates | grep YE
# ^_ YE/YR are not in ca-certificates trust store

root@e46e8254f45e:/# find /usr/share/ca-certificates -iname "*isrg*"
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/usr/share/ca-certificates/mozilla/ISRG_Root_X2.crt

Broken (insecure) TLS request:

root@e46e8254f45e:/# curl https://valid.yr.test-certs.letsencrypt.org/
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

Fedora (44)

[root@8fa12baac27c /]# cat /etc/os-release 
[...]
PRETTY_NAME="Fedora Linux 44 (Container Image)"
[...]

[root@8fa12baac27c /]# dnf list installed ca-certificates
Updating and loading repositories:
Repositories loaded.
Installed packages (available for reinstall, available for upgrade)
ca-certificates.noarch 2025.2.80_v9.0.304-7.fc44 9f717d77be324992a27b71f74a042c00

[root@8fa12baac27c /]# dnf repoquery -l ca-certificates | grep ISRG
Updating and loading repositories:
 [...]
Repositories loaded.
/etc/pki/ca-trust/extracted/pem/directory-hash/ISRG_Root_X1.pem
/etc/pki/ca-trust/extracted/pem/directory-hash/ISRG_Root_X2.pem

[root@8fa12baac27c /]# dnf repoquery -l ca-certificates | grep YR
Updating and loading repositories:
Repositories loaded.
[root@8fa12baac27c /]# dnf repoquery -l ca-certificates | grep YE
Updating and loading repositories:
Repositories loaded.

# ^_ YE/YR are not in ca-certificates trust store

Broken (insecure) TLS request:

[root@8fa12baac27c /]# curl https://valid.yr.test-certs.letsencrypt.org/
curl: (60) SSL certificate OpenSSL verify result: unable to get local issuer certificate (20)
More details here: https://curl.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the webpage mentioned above.

[root@8fa12baac27c /]# curl -v https://valid.yr.test-certs.letsencrypt.org/ 2>&1 | grep CAfile
*   CAfile: /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
[root@8fa12baac27c /]# grep ISRG /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
# ISRG Root X1
# ISRG Root X2

Conclusion

I my opinion the ISRG Root YE/YR should be provided by the distribution ca-certificates packages as mentioned in YE/YR are not in ca-certificates! by 532910.

Providing two intermediates (e.g. YR1+Root YR) certificates for the fullchain valiadition is a "solution" until then.

2

it'd be normal to see it there:
things like https://valid.x1.test-certs.letsencrypt.org/ already using Y tier signed certificate at its leaf, just including cross-sign certificate from YE/YR to X2/X1.

3

That's entirely normal, since the new YE and YR Root Certificates haven't been submitted to any root programs yet, including Mozilla's Root Program, where the ca-certificates trust store is sourced from.

The "current" trust anchors are still X1 and X2, and all chains served by Let's Encrypt chain up to one or both of these roots:

For ECC certificates: EE <= YEn <= YE <= X2 <= X1 (the last cross-cert being optional)
For RSA certificates: EE <= YRn <= YR <= X1

5

This is not true, we are getting new certificates signed by YR2 and things are broken as Root YR is not trusted anywhere.

le2
le2853×526 22.6 KB

6

The Root YR does not have to be in the trust anchor store, because the TLS server supposed to send it (actually a cross-signed version) to the TLS client. Then, this certificate chains up to a trusted root certificate X2, or definitely to X1 as the client searches for a valid trust chain.

7

Can you point me to any LE documentation saying this? Thanks.

Note that LE is NOT sending Root YR in default chain.

8

Here it is: Chains of Trust - Let's Encrypt

9

That's why you have the YR <= X1 cross-cert in the chain.

YR is not a trust anchor (yet), X1 is. But soon there will be disjoint sets of clients trusting either YR or X1 (not both). That's why you must have both in the chain.