RHEL/CentOS 6 OpenSSL client compatibility after DST Root CA X3 expiration

already tried to manually delete the X3 - but had no effect on Centos 6 Test-Server ...

1 Like

probably only if openssl also updated somehow? as on OpenSSL 1.0.1e-fips which is latest for Centos 6.10 it won't help

1 Like

Even if DST Root CA X3 is removed, as suggested in this thread?

1 Like

Yes, for me it didn't help

1 Like

Ok, that's unfortunate.. I stand corrected.

1 Like

Yeah on CentOS 6 with openssl 1.0.1 didn't work like CentOS 7 with openssl 1.0.2 which works

But on CentOS 6 with openssl 1.0.1 test against valid-isrgrootx1.letsencrypt.org worked for openssl, curl and wget but test against community.letsencrypt.org didn't work for openssl or wget but works on curl.

domain=valid-isrgrootx1.letsencrypt.org

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
 0 s:/CN=valid-isrgrootx1.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1

curl -Iv https://$domain
* About to connect() to valid-isrgrootx1.letsencrypt.org port 443 (#0)
*   Trying 52.9.173.94... connected
* Connected to valid-isrgrootx1.letsencrypt.org (52.9.173.94) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=valid-isrgrootx1.letsencrypt.org
*       start date: Aug 04 15:00:08 2021 GMT
*       expire date: Nov 02 15:00:06 2021 GMT
*       common name: valid-isrgrootx1.letsencrypt.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: valid-isrgrootx1.letsencrypt.org
> Accept: */*

wget -O /dev/null -vS https://$domain
--2021-09-30 17:42:06--  https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org... 52.9.173.94
Connecting to valid-isrgrootx1.letsencrypt.org|52.9.173.94|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx
  Date: Thu, 30 Sep 2021 21:42:06 GMT
  Content-Type: text/html
  Content-Length: 4067
  Last-Modified: Mon, 09 Aug 2021 23:45:57 GMT
  Connection: keep-alive
  Vary: Accept-Encoding
  ETag: "6111be35-fe3"
  Strict-Transport-Security: max-age=604800
  X-XSS-Protection: 1; mode=block
  Accept-Ranges: bytes
Length: 4067 (4.0K) [text/html]
Saving to: ‘/dev/null’

/dev/null                                           100%[=================================================================================================================>]   3.97K  --.-KB/s    in 0s      

2021-09-30 17:42:06 (199 MB/s) - ‘/dev/null’ saved [4067/4067]

against community.letsencrypt.org

domain=community.letsencrypt.org

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:0

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
 0 s:/CN=community.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

curl -Iv https://$domain
* About to connect() to community.letsencrypt.org port 443 (#0)
*   Trying 64.71.144.202... connected
* Connected to community.letsencrypt.org (64.71.144.202) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=community.letsencrypt.org
*       start date: Sep 13 00:00:28 2021 GMT
*       expire date: Dec 12 00:00:27 2021 GMT
*       common name: community.letsencrypt.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: community.letsencrypt.org
> Accept: */*

wget -O /dev/null -vS https://$domain
--2021-09-30 17:45:06--  https://community.letsencrypt.org/
Resolving community.letsencrypt.org... 64.71.144.202, 2602:fd3f:3:ff01::ca
Connecting to community.letsencrypt.org|64.71.144.202|:443... connected.
ERROR: cannot verify community.letsencrypt.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Issued certificate has expired.
To connect to community.letsencrypt.org insecurely, use `--no-check-certificate'.
echo -n | strace openssl s_client -servername $domain -showcerts -connect $domain:443 |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 3

strace curl -Iv https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 10

strace wget -O /dev/null -vS https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 4

openssl, wget and curl all read the same file

ls -lah /etc/pki/tls/cert.pem
lrwxrwxrwx 1 root root 19 Dec  1  2020 /etc/pki/tls/cert.pem -> certs/ca-bundle.crt

Looks like blacklisting the DST isn''t enough on CentOS 6

cat /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----

grep 'DST Root CA X3' /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.trust.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
/etc/pki/tls/certs/ca-bundle.trust.crt:# DST Root CA X3
/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt:# DST Root CA X3

seems to be picked up from /etc/pki/tls/cert.pem

grep DST /etc/pki/tls/cert.pem
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k
yum -q provides /etc/pki/tls/cert.pem
ca-certificates-2018.2.22-65.1.el6.noarch : The Mozilla CA root certificate bundle
Repo        : base
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2019.2.32-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : installed
Matched from:
Other       : Provides-match: /etc/pki/tls/cert.pem

edit: ah Redhat issued new ca-certificate packages for RH 8, 7 Let's Encrypt DST X3 Root Certificate Expiration - Red Hat Customer Portal

Updated System Certificate Packages

We have released updated ca-certificates packages for RHEL6 ELS, RHEL7 and RHEL8.

These can be updated via the normal yum or dnf update commands.

There is no updated package for the non-ELS RHEL6. The client workaround below can be used to remove the expired certificate from the system bundle.

Updated CentOS 6 x86 yum packages

x86_64
ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm	SHA-256: cd59b687781bf3795e1a2ba3d19b1eba29dc08505ad0c4ae5ae8a1a3b1369ad9
nss-3.44.0-11.el6_10.i686.rpm	SHA-256: 7b1ddfa75b8af481a2b95b5284b266969a646776795af2f0f6239a639a4814bf
nss-3.44.0-11.el6_10.x86_64.rpm	SHA-256: a8ddde535581e3bb5bd059c02f62cee9e352e568fcd091e33016d1e83e8ec836
nss-debuginfo-3.44.0-11.el6_10.i686.rpm	SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.i686.rpm	SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm	SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm	SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-devel-3.44.0-11.el6_10.i686.rpm	SHA-256: 1512614cc1e0c7874d6f7f397eac1b9fc7a3ccdc28e285e463f93323c5e681f6
nss-devel-3.44.0-11.el6_10.x86_64.rpm	SHA-256: 66b09eb27e0492b1d50604a98039559c9d849db6cb0694ad0db99a07eda1e8c4
nss-pkcs11-devel-3.44.0-11.el6_10.i686.rpm	SHA-256: 0e8cad432a9c2d81e342cd126fb92f12db44339ec099290e56059adb622437cf
nss-pkcs11-devel-3.44.0-11.el6_10.x86_64.rpm	SHA-256: f48020e682d84993ad9f38628b3a670bb63232b281d463578c7eca86a0d4b2c8
nss-sysinit-3.44.0-11.el6_10.x86_64.rpm	SHA-256: 717aca2880a7207f34a3fba1173e1d79081e5260c9432439c906dddb8b584dd4
nss-tools-3.44.0-11.el6_10.x86_64.rpm	SHA-256: e55a3fec6fcf0493f0a3b0e32f81cc9690d4f9b38e315c1ec53869c82db4a4a

and below so the blacklist didn't work because you still needed a CentOS 6 ca-certificates YUM update to remove the reference to DST Root CA X3

Client Connection Workaround

To work around the openssl client problem on RHEL 6 first ensure your ca-certificates package is updated to the most recently available in your RHEL6 channels ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm .

Then to remove the expired root CA from the system trust store,

  • Create an exclusion file:

Raw

# perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}' </etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem
  • Update the system trust store:

Raw

# update-ca-trust extract

that is for client workaround and they also have a longer guide for server workaround to fix the chain

3 Likes

Based on my tests I did earlier this year, even if you manage to remove DST Root CA X3 from the trust store, this won't solve the problem (on CentOS 6/OpenSSL 1.0.1). OpenSSL will then start printing "unable to get local issuer certificate", doesn't matter what you add or remove from the trust store.

The workaround with removing the expired root requires OpenSSL's partial chain feature, which is only available in 1.0.2 and up. If RHEL has made patches to OpenSSL 1.0.1 to backport that feature, then you have a chance.

2 Likes

I updated my previous post, reason removing DST isn't enough is you also need updated ca-certificates package for CentOS 6 it seems = RHEL 6 ELS - ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm

1 Like

Can you confirm that this resolves the connection issues on OpenSSL 1.0.1/CentOS 6? I don't have access to the patches Red Hat has made to the libraries, so I can't check.

1 Like

yeah I don't have access to ELS CentOS 6 either heh so issues I outlined above still apply openssl and wget don't work but curl does

Right now looks like you'd need to rely on letsencrypt sites like community.letsencrypt.org to fix their server side chains.

3 Likes

I manually built and installed a version of ca-certificates for RHEL6 and this isn't enough. OpenSSL 1.0.1e is the real problem. Blacklisting the certificate manually produces the same result as updating the ca-certificates package to the latest version from ELS (or backporting from EL7, as I did), which is to remove if from the ca-bundle file. RH has not released any updated version of OpenSSL for RHEL6 so there is currently no fix for the root cause.

For servers I manage that my CentOS 6 server needs to connect to, I used certbot with the --preferred-chain "ISRG Root X1" option to remove the DST X3 cross-signature from the chain. certbot version 1.12 or later is needed for this option to work. It breaks the special Android compatibility that Let's Encrypt set up, but it makes it so my CentOS 6 server can connect to them without OpenSSL becoming confused. This only works if you can control the remote server your CentOS 6 is connecting to, of course.

4 Likes

Yeah that's the only way now fixing server side chain of the site you're trying to connect to to allow CentOS 6 clients like wget and openssl to work properly. Unfortunately, if you don't control the remote site, you're out of luck heh.

I use acme.sh client for issuance so preferred chain flag is supported on command line right now. Hoping it can be added a system default flag Setting Preferred Chain system wide? · Issue #3717 · acmesh-official/acme.sh · GitHub by @Neilpang :slight_smile:

1 Like

Thanks for the confirmation, this matches my expectations. Sadly this is not great for those running these old versions.

So my initial statement still holds, right now there is not much that can be done for CentOS/RHEL 6. I strongly recommend upgrading, if that's possible in any way shape or form.

Otherwise, I don't know if there's still someone willing to support this OS, but if there is: It may be possible to backport the partial/alternate chain behaviour of 1.0.2 to 1.0.1*, which could enable "remove-the-old-root" workarounds. This requires patching OpenSSL though.

Using LE's short alternate chain is definetly a fix, if all servers are under your control and old Android compatibility is not required.

*I just made a trip to the source code. The basic commit that enables the 1.0.2 workaround was this one, but you can't backport that directly, because a) the code on 1.0.1 is slightly different and b) there were issues with that code, later fixed in followup commits (one, two, three + at least three more). May be easier to just update everything to 1.0.2...

3 Likes

It's my understanding that OpenSSL 1.0.1 and 1.0.2 are intended to be ABI compatible. I'm currently doing some experimentation with rebuilding openssl-1.0.2k-21 (latest for CentOS 7) for CentOS 6. If that proves to work, it could be an option for anyone that needs this to work for a while longer. It needs to be combined with a ca-certificates update to version 2021.2.50 or removal of the DST cert from the trust store to have any chance of working. I'll try to report back with results soon.

5 Likes

I was able to successfully build the package after modifying a few things, and it installed without an issue. Programs built against openssl still run, and wget and openssl s_client work correctly and validate the certificate. This success requires you also have either removed the DST Root CA X3 certificate or have the EL6 ELS version of ca-certificates installed.

I picked up the source RPM from https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm

To build, I had to change ALL references in three patches from secure_getenv() to getenv():

  • openssl-1.0.2a-env-zlib.patch
  • openssl-1.0.2j-deprecate-algos.patch
  • openssl-1.0.2a-fips-ctor.patch

I also had to comment out this line (216) from the openssl.spec file:
%patch68 -p1 -b .secure-getenv

These references to secure_getenv() can't be used because glibc in CentOS 6 doesn't have the function. RH blindly added them without checking for support because they knew it was safe to assume CentOS 7 would have it.

Your mileage may vary, of course. Anything that depends on the specific 1.0.1e version might break, or might have incorrect expectations about what security features are available, leading to unexpected bad stuff, but so far I haven't had it happen to me. I'm not giving out any warranties or prebuilt packages, but hopefully this is enough to get people pointed in the right direction.

Edit: Just in case it may also help: To produce an updated version of ca-certificates, you can do something like this. Take these two packages:

Extract certdata.txt from the el7 SRPM and replace the file at SOURCES/certdata.txt for the el6 rpm build. Update the EL6 RPM spec file Version line to 2021.2.50, and build that package.

7 Likes

thanks @kstange was trying to find a way of just rebuild the ca-certificate cert files themselves - not sure if any info at GitHub - rjrelyea/ca-certificate-scripts: script to manage ca-certificates. helps

looks like cpanel users getting hit by this too https://forums.cpanel.net/threads/cpanel-33077-letsencrypt-transition-to-isrgs-root-important.673981/post-2877037

cloudlinux ELS 6 has updated packages for ca-certificates and seems to work https://forums.cpanel.net/threads/cpanel-33077-letsencrypt-transition-to-isrgs-root-important.673981/post-2877329 and DST Root CA X3 Expiration and Let's Encrypt – cPanel

2 Likes

Coukd you please share the package?

1 Like

I successfully build both packages, installec ca-certificates without any problems, but i have problems to install openssl-* since it is now splitting into many rpms like openssl-libs, openssl-static and so on...

How did you manage to install new openssl-1.0.2 packages without RPM DB conflits/problems?
I have bunch of errors like this:
https://paste.centos.org/view/82414818

Which is strange, because libcrypto.so.10 is in openssl-devel-1.0.2 for 100%

1 Like

I was able to build and install openssl 1.0.2k-8 package. I can provide you with SRPM if you'd like. It wasn't build for Centos, but RHEL based OS. Shouldn't be any different.

1 Like

rpm --test -Uvf openssl-1.0.2k-21.el6.x86_64.rpm openssl-libs-1.0.2k-21.el6.x86_64.rpm

If ok, delete --test param

2 Likes