RHEL/CentOS 6 OpenSSL client compatibility after DST Root CA X3 expiration

Help
1

As announced (OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates) expiration of DST Root CA X3 causing issues for clients with OpenSSL < 1.1.0.

As there are still some very old Centos/RHEL 6 Servers (openssl-1.0.1e-58.el6_10.x86_64) out there (especially some of our VM Hosting/Housing Customers still resist upgrading some of their legacy system) and today some of those Customers contacted our Support I'm puzzled on how to remove the old X3 (ISRG Root X1 is already available in the last ca-certificates rpm) from those Centos/RHEL 6 Servers ... (after suggesting that they should upgrade to newer OS of course ... :wink: )

Any ideas on that?

Thank you, bye from Austria
Andy

4 Likes
2

My latest info was that CentOS/RHEL 6 ships OpenSSL 1.0.1, while 7 ships OpenSSL 1.0.2.

The workarounds known for OpenSSL (removing the old root is one of them) only work on OpenSSL 1.0.2. Unless RHEL 6 has backported patches*, I'm not aware of anything you can do for those.

The CentOS/RHEL 7 discussion was here: RHEL/CentOS 7 OpenSSL client compatibility after new chain - #3 by ghen

*There were no patches for RHEL 6 back in May this year, but I don't know if the situation has changed since then.

3 Likes
3

Note that there is no such thing as a "not expired X3". In principle, Let's Encrypt uses its own ISRG Root X1 root certificate. The only reason the LE default certificate chain still chains up to DST Root CA X3 is for older Android support. But that does not mean there's going to be a new DST Root CA X3: it's gone and will never return.

You probably need to add ISRG Root X1 to the root stores.

1 Like
4

Have same issue. How can we install new root cert on Centos 6?

1 Like
5

If it's unpatched OpenSSL 1.0.1, this won't help. (If patches were available, I would hope they also include ISRG Root X1?)

1 Like
6

of course ... my fault ... corrected in posting ... thx! :slight_smile:

2 Likes
7

Removing the X3 from server fixed the issue here:

First you need to update CA certificates to add the new X1 cert.

  • yum update ca-certificates

Then look for the old X3

  • sudo nano /etc/ssl/certs/ca-bundle.crt

Search for "DST Root CA X3" in this file and delete the X3 certificate.
Save file and be happy :slight_smile:

1 Like
8

already tried to manually delete the X3 - but had no effect on Centos 6 Test-Server ...

1 Like
9

probably only if openssl also updated somehow? as on OpenSSL 1.0.1e-fips which is latest for Centos 6.10 it won't help

1 Like
10

Even if DST Root CA X3 is removed, as suggested in this thread?

1 Like
11

Yes, for me it didn't help

1 Like
12

Ok, that's unfortunate.. I stand corrected.

1 Like
13

Yeah on CentOS 6 with openssl 1.0.1 didn't work like CentOS 7 with openssl 1.0.2 which works

But on CentOS 6 with openssl 1.0.1 test against valid-isrgrootx1.letsencrypt.org worked for openssl, curl and wget but test against community.letsencrypt.org didn't work for openssl or wget but works on curl.

domain=valid-isrgrootx1.letsencrypt.org

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
 0 s:/CN=valid-isrgrootx1.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1

curl -Iv https://$domain
* About to connect() to valid-isrgrootx1.letsencrypt.org port 443 (#0)
*   Trying 52.9.173.94... connected
* Connected to valid-isrgrootx1.letsencrypt.org (52.9.173.94) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=valid-isrgrootx1.letsencrypt.org
*       start date: Aug 04 15:00:08 2021 GMT
*       expire date: Nov 02 15:00:06 2021 GMT
*       common name: valid-isrgrootx1.letsencrypt.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: valid-isrgrootx1.letsencrypt.org
> Accept: */*

wget -O /dev/null -vS https://$domain
--2021-09-30 17:42:06--  https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org... 52.9.173.94
Connecting to valid-isrgrootx1.letsencrypt.org|52.9.173.94|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx
  Date: Thu, 30 Sep 2021 21:42:06 GMT
  Content-Type: text/html
  Content-Length: 4067
  Last-Modified: Mon, 09 Aug 2021 23:45:57 GMT
  Connection: keep-alive
  Vary: Accept-Encoding
  ETag: "6111be35-fe3"
  Strict-Transport-Security: max-age=604800
  X-XSS-Protection: 1; mode=block
  Accept-Ranges: bytes
Length: 4067 (4.0K) [text/html]
Saving to: ‘/dev/null’

/dev/null                                           100%[=================================================================================================================>]   3.97K  --.-KB/s    in 0s      

2021-09-30 17:42:06 (199 MB/s) - ‘/dev/null’ saved [4067/4067]

against community.letsencrypt.org

domain=community.letsencrypt.org

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:0

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
 0 s:/CN=community.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

curl -Iv https://$domain
* About to connect() to community.letsencrypt.org port 443 (#0)
*   Trying 64.71.144.202... connected
* Connected to community.letsencrypt.org (64.71.144.202) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=community.letsencrypt.org
*       start date: Sep 13 00:00:28 2021 GMT
*       expire date: Dec 12 00:00:27 2021 GMT
*       common name: community.letsencrypt.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: community.letsencrypt.org
> Accept: */*

wget -O /dev/null -vS https://$domain
--2021-09-30 17:45:06--  https://community.letsencrypt.org/
Resolving community.letsencrypt.org... 64.71.144.202, 2602:fd3f:3:ff01::ca
Connecting to community.letsencrypt.org|64.71.144.202|:443... connected.
ERROR: cannot verify community.letsencrypt.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Issued certificate has expired.
To connect to community.letsencrypt.org insecurely, use `--no-check-certificate'.

echo -n | strace openssl s_client -servername $domain -showcerts -connect $domain:443 |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 3

strace curl -Iv https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 10

strace wget -O /dev/null -vS https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 4

openssl, wget and curl all read the same file

ls -lah /etc/pki/tls/cert.pem
lrwxrwxrwx 1 root root 19 Dec  1  2020 /etc/pki/tls/cert.pem -> certs/ca-bundle.crt

Looks like blacklisting the DST isn''t enough on CentOS 6

cat /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----


grep 'DST Root CA X3' /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.trust.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
/etc/pki/tls/certs/ca-bundle.trust.crt:# DST Root CA X3
/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt:# DST Root CA X3

seems to be picked up from /etc/pki/tls/cert.pem

grep DST /etc/pki/tls/cert.pem
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k

yum -q provides /etc/pki/tls/cert.pem
ca-certificates-2018.2.22-65.1.el6.noarch : The Mozilla CA root certificate bundle
Repo        : base
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2019.2.32-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : installed
Matched from:
Other       : Provides-match: /etc/pki/tls/cert.pem

edit: ah Redhat issued new ca-certificate packages for RH 8, 7 Let's Encrypt DST X3 Root Certificate Expiration - Red Hat Customer Portal

Updated System Certificate Packages

We have released updated ca-certificates packages for RHEL6 ELS, RHEL7 and RHEL8.

These can be updated via the normal yum or dnf update commands.

There is no updated package for the non-ELS RHEL6. The client workaround below can be used to remove the expired certificate from the system bundle.

Updated CentOS 6 x86 yum packages

x86_64
ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm	SHA-256: cd59b687781bf3795e1a2ba3d19b1eba29dc08505ad0c4ae5ae8a1a3b1369ad9
nss-3.44.0-11.el6_10.i686.rpm	SHA-256: 7b1ddfa75b8af481a2b95b5284b266969a646776795af2f0f6239a639a4814bf
nss-3.44.0-11.el6_10.x86_64.rpm	SHA-256: a8ddde535581e3bb5bd059c02f62cee9e352e568fcd091e33016d1e83e8ec836
nss-debuginfo-3.44.0-11.el6_10.i686.rpm	SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.i686.rpm	SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm	SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm	SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-devel-3.44.0-11.el6_10.i686.rpm	SHA-256: 1512614cc1e0c7874d6f7f397eac1b9fc7a3ccdc28e285e463f93323c5e681f6
nss-devel-3.44.0-11.el6_10.x86_64.rpm	SHA-256: 66b09eb27e0492b1d50604a98039559c9d849db6cb0694ad0db99a07eda1e8c4
nss-pkcs11-devel-3.44.0-11.el6_10.i686.rpm	SHA-256: 0e8cad432a9c2d81e342cd126fb92f12db44339ec099290e56059adb622437cf
nss-pkcs11-devel-3.44.0-11.el6_10.x86_64.rpm	SHA-256: f48020e682d84993ad9f38628b3a670bb63232b281d463578c7eca86a0d4b2c8
nss-sysinit-3.44.0-11.el6_10.x86_64.rpm	SHA-256: 717aca2880a7207f34a3fba1173e1d79081e5260c9432439c906dddb8b584dd4
nss-tools-3.44.0-11.el6_10.x86_64.rpm	SHA-256: e55a3fec6fcf0493f0a3b0e32f81cc9690d4f9b38e315c1ec53869c82db4a4a

and below so the blacklist didn't work because you still needed a CentOS 6 ca-certificates YUM update to remove the reference to DST Root CA X3

Client Connection Workaround

To work around the openssl client problem on RHEL 6 first ensure your ca-certificates package is updated to the most recently available in your RHEL6 channels ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm .

Then to remove the expired root CA from the system trust store,

  • Create an exclusion file:

Raw

# perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}' </etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem
  • Update the system trust store:

Raw

# update-ca-trust extract

that is for client workaround and they also have a longer guide for server workaround to fix the chain

3 Likes
14

Based on my tests I did earlier this year, even if you manage to remove DST Root CA X3 from the trust store, this won't solve the problem (on CentOS 6/OpenSSL 1.0.1). OpenSSL will then start printing "unable to get local issuer certificate", doesn't matter what you add or remove from the trust store.

The workaround with removing the expired root requires OpenSSL's partial chain feature, which is only available in 1.0.2 and up. If RHEL has made patches to OpenSSL 1.0.1 to backport that feature, then you have a chance.

2 Likes
15

I updated my previous post, reason removing DST isn't enough is you also need updated ca-certificates package for CentOS 6 it seems = RHEL 6 ELS - ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm

1 Like
16

Can you confirm that this resolves the connection issues on OpenSSL 1.0.1/CentOS 6? I don't have access to the patches Red Hat has made to the libraries, so I can't check.

1 Like
17

yeah I don't have access to ELS CentOS 6 either heh so issues I outlined above still apply openssl and wget don't work but curl does

Right now looks like you'd need to rely on letsencrypt sites like community.letsencrypt.org to fix their server side chains.

3 Likes
18

I manually built and installed a version of ca-certificates for RHEL6 and this isn't enough. OpenSSL 1.0.1e is the real problem. Blacklisting the certificate manually produces the same result as updating the ca-certificates package to the latest version from ELS (or backporting from EL7, as I did), which is to remove if from the ca-bundle file. RH has not released any updated version of OpenSSL for RHEL6 so there is currently no fix for the root cause.

For servers I manage that my CentOS 6 server needs to connect to, I used certbot with the --preferred-chain "ISRG Root X1" option to remove the DST X3 cross-signature from the chain. certbot version 1.12 or later is needed for this option to work. It breaks the special Android compatibility that Let's Encrypt set up, but it makes it so my CentOS 6 server can connect to them without OpenSSL becoming confused. This only works if you can control the remote server your CentOS 6 is connecting to, of course.

4 Likes
19

Yeah that's the only way now fixing server side chain of the site you're trying to connect to to allow CentOS 6 clients like wget and openssl to work properly. Unfortunately, if you don't control the remote site, you're out of luck heh.

I use acme.sh client for issuance so preferred chain flag is supported on command line right now. Hoping it can be added a system default flag Setting Preferred Chain system wide? · Issue #3717 · acmesh-official/acme.sh · GitHub by @Neilpang :slight_smile:

1 Like
20

Thanks for the confirmation, this matches my expectations. Sadly this is not great for those running these old versions.

So my initial statement still holds, right now there is not much that can be done for CentOS/RHEL 6. I strongly recommend upgrading, if that's possible in any way shape or form.

Otherwise, I don't know if there's still someone willing to support this OS, but if there is: It may be possible to backport the partial/alternate chain behaviour of 1.0.2 to 1.0.1*, which could enable "remove-the-old-root" workarounds. This requires patching OpenSSL though.

Using LE's short alternate chain is definetly a fix, if all servers are under your control and old Android compatibility is not required.

*I just made a trip to the source code. The basic commit that enables the 1.0.2 workaround was this one, but you can't backport that directly, because a) the code on 1.0.1 is slightly different and b) there were issues with that code, later fixed in followup commits (one, two, three + at least three more). May be easier to just update everything to 1.0.2...

3 Likes