RHEL/CentOS 6 OpenSSL client compatibility after DST Root CA X3 expiration

Help
13

Yeah on CentOS 6 with openssl 1.0.1 didn't work like CentOS 7 with openssl 1.0.2 which works

But on CentOS 6 with openssl 1.0.1 test against valid-isrgrootx1.letsencrypt.org worked for openssl, curl and wget but test against community.letsencrypt.org didn't work for openssl or wget but works on curl.

domain=valid-isrgrootx1.letsencrypt.org

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
 0 s:/CN=valid-isrgrootx1.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1

curl -Iv https://$domain
* About to connect() to valid-isrgrootx1.letsencrypt.org port 443 (#0)
*   Trying 52.9.173.94... connected
* Connected to valid-isrgrootx1.letsencrypt.org (52.9.173.94) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=valid-isrgrootx1.letsencrypt.org
*       start date: Aug 04 15:00:08 2021 GMT
*       expire date: Nov 02 15:00:06 2021 GMT
*       common name: valid-isrgrootx1.letsencrypt.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: valid-isrgrootx1.letsencrypt.org
> Accept: */*

wget -O /dev/null -vS https://$domain
--2021-09-30 17:42:06--  https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org... 52.9.173.94
Connecting to valid-isrgrootx1.letsencrypt.org|52.9.173.94|:443... connected.
HTTP request sent, awaiting response... 
  HTTP/1.1 200 OK
  Server: nginx
  Date: Thu, 30 Sep 2021 21:42:06 GMT
  Content-Type: text/html
  Content-Length: 4067
  Last-Modified: Mon, 09 Aug 2021 23:45:57 GMT
  Connection: keep-alive
  Vary: Accept-Encoding
  ETag: "6111be35-fe3"
  Strict-Transport-Security: max-age=604800
  X-XSS-Protection: 1; mode=block
  Accept-Ranges: bytes
Length: 4067 (4.0K) [text/html]
Saving to: ‘/dev/null’

/dev/null                                           100%[=================================================================================================================>]   3.97K  --.-KB/s    in 0s      

2021-09-30 17:42:06 (199 MB/s) - ‘/dev/null’ saved [4067/4067]

against community.letsencrypt.org

domain=community.letsencrypt.org

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:0

echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
 0 s:/CN=community.letsencrypt.org
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3

curl -Iv https://$domain
* About to connect() to community.letsencrypt.org port 443 (#0)
*   Trying 64.71.144.202... connected
* Connected to community.letsencrypt.org (64.71.144.202) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: CN=community.letsencrypt.org
*       start date: Sep 13 00:00:28 2021 GMT
*       expire date: Dec 12 00:00:27 2021 GMT
*       common name: community.letsencrypt.org
*       issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: community.letsencrypt.org
> Accept: */*

wget -O /dev/null -vS https://$domain
--2021-09-30 17:45:06--  https://community.letsencrypt.org/
Resolving community.letsencrypt.org... 64.71.144.202, 2602:fd3f:3:ff01::ca
Connecting to community.letsencrypt.org|64.71.144.202|:443... connected.
ERROR: cannot verify community.letsencrypt.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  Issued certificate has expired.
To connect to community.letsencrypt.org insecurely, use `--no-check-certificate'.

echo -n | strace openssl s_client -servername $domain -showcerts -connect $domain:443 |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 3

strace curl -Iv https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 10

strace wget -O /dev/null -vS https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 4

openssl, wget and curl all read the same file

ls -lah /etc/pki/tls/cert.pem
lrwxrwxrwx 1 root root 19 Dec  1  2020 /etc/pki/tls/cert.pem -> certs/ca-bundle.crt

Looks like blacklisting the DST isn''t enough on CentOS 6

cat /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----


grep 'DST Root CA X3' /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.trust.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
/etc/pki/tls/certs/ca-bundle.trust.crt:# DST Root CA X3
/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt:# DST Root CA X3

seems to be picked up from /etc/pki/tls/cert.pem

grep DST /etc/pki/tls/cert.pem
        Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
        Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k

yum -q provides /etc/pki/tls/cert.pem
ca-certificates-2018.2.22-65.1.el6.noarch : The Mozilla CA root certificate bundle
Repo        : base
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2019.2.32-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : updates
Matched from:
Filename    : /etc/pki/tls/cert.pem



ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo        : installed
Matched from:
Other       : Provides-match: /etc/pki/tls/cert.pem

edit: ah Redhat issued new ca-certificate packages for RH 8, 7 Let's Encrypt DST X3 Root Certificate Expiration - Red Hat Customer Portal

Updated System Certificate Packages

We have released updated ca-certificates packages for RHEL6 ELS, RHEL7 and RHEL8.

These can be updated via the normal yum or dnf update commands.

There is no updated package for the non-ELS RHEL6. The client workaround below can be used to remove the expired certificate from the system bundle.

Updated CentOS 6 x86 yum packages

x86_64
ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm	SHA-256: cd59b687781bf3795e1a2ba3d19b1eba29dc08505ad0c4ae5ae8a1a3b1369ad9
nss-3.44.0-11.el6_10.i686.rpm	SHA-256: 7b1ddfa75b8af481a2b95b5284b266969a646776795af2f0f6239a639a4814bf
nss-3.44.0-11.el6_10.x86_64.rpm	SHA-256: a8ddde535581e3bb5bd059c02f62cee9e352e568fcd091e33016d1e83e8ec836
nss-debuginfo-3.44.0-11.el6_10.i686.rpm	SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.i686.rpm	SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm	SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm	SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-devel-3.44.0-11.el6_10.i686.rpm	SHA-256: 1512614cc1e0c7874d6f7f397eac1b9fc7a3ccdc28e285e463f93323c5e681f6
nss-devel-3.44.0-11.el6_10.x86_64.rpm	SHA-256: 66b09eb27e0492b1d50604a98039559c9d849db6cb0694ad0db99a07eda1e8c4
nss-pkcs11-devel-3.44.0-11.el6_10.i686.rpm	SHA-256: 0e8cad432a9c2d81e342cd126fb92f12db44339ec099290e56059adb622437cf
nss-pkcs11-devel-3.44.0-11.el6_10.x86_64.rpm	SHA-256: f48020e682d84993ad9f38628b3a670bb63232b281d463578c7eca86a0d4b2c8
nss-sysinit-3.44.0-11.el6_10.x86_64.rpm	SHA-256: 717aca2880a7207f34a3fba1173e1d79081e5260c9432439c906dddb8b584dd4
nss-tools-3.44.0-11.el6_10.x86_64.rpm	SHA-256: e55a3fec6fcf0493f0a3b0e32f81cc9690d4f9b38e315c1ec53869c82db4a4a

and below so the blacklist didn't work because you still needed a CentOS 6 ca-certificates YUM update to remove the reference to DST Root CA X3

Client Connection Workaround

To work around the openssl client problem on RHEL 6 first ensure your ca-certificates package is updated to the most recently available in your RHEL6 channels ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm .

Then to remove the expired root CA from the system trust store,

  • Create an exclusion file:

Raw

# perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}' </etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem
  • Update the system trust store:

Raw

# update-ca-trust extract

that is for client workaround and they also have a longer guide for server workaround to fix the chain

3 Likes