already tried to manually delete the X3 - but had no effect on Centos 6 Test-Server ...
probably only if openssl also updated somehow? as on OpenSSL 1.0.1e-fips which is latest for Centos 6.10 it won't help
Even if DST Root CA X3 is removed, as suggested in this thread?
Yes, for me it didn't help
Ok, that's unfortunate.. I stand corrected.
Yeah on CentOS 6 with openssl 1.0.1 didn't work like CentOS 7 with openssl 1.0.2 which works
But on CentOS 6 with openssl 1.0.1 test against valid-isrgrootx1.letsencrypt.org
worked for openssl, curl and wget but test against community.letsencrypt.org
didn't work for openssl or wget but works on curl.
domain=valid-isrgrootx1.letsencrypt.org
echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
0 s:/CN=valid-isrgrootx1.letsencrypt.org
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
curl -Iv https://$domain
* About to connect() to valid-isrgrootx1.letsencrypt.org port 443 (#0)
* Trying 52.9.173.94... connected
* Connected to valid-isrgrootx1.letsencrypt.org (52.9.173.94) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=valid-isrgrootx1.letsencrypt.org
* start date: Aug 04 15:00:08 2021 GMT
* expire date: Nov 02 15:00:06 2021 GMT
* common name: valid-isrgrootx1.letsencrypt.org
* issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: valid-isrgrootx1.letsencrypt.org
> Accept: */*
wget -O /dev/null -vS https://$domain
--2021-09-30 17:42:06-- https://valid-isrgrootx1.letsencrypt.org/
Resolving valid-isrgrootx1.letsencrypt.org... 52.9.173.94
Connecting to valid-isrgrootx1.letsencrypt.org|52.9.173.94|:443... connected.
HTTP request sent, awaiting response...
HTTP/1.1 200 OK
Server: nginx
Date: Thu, 30 Sep 2021 21:42:06 GMT
Content-Type: text/html
Content-Length: 4067
Last-Modified: Mon, 09 Aug 2021 23:45:57 GMT
Connection: keep-alive
Vary: Accept-Encoding
ETag: "6111be35-fe3"
Strict-Transport-Security: max-age=604800
X-XSS-Protection: 1; mode=block
Accept-Ranges: bytes
Length: 4067 (4.0K) [text/html]
Saving to: ‘/dev/null’
/dev/null 100%[=================================================================================================================>] 3.97K --.-KB/s in 0s
2021-09-30 17:42:06 (199 MB/s) - ‘/dev/null’ saved [4067/4067]
against community.letsencrypt.org
domain=community.letsencrypt.org
echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:0
echo -n | openssl s_client -servername $domain -showcerts -connect $domain:443 2>&1 | egrep ' s:\/| i:\/'
0 s:/CN=community.letsencrypt.org
i:/C=US/O=Let's Encrypt/CN=R3
1 s:/C=US/O=Let's Encrypt/CN=R3
i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
i:/O=Digital Signature Trust Co./CN=DST Root CA X3
curl -Iv https://$domain
* About to connect() to community.letsencrypt.org port 443 (#0)
* Trying 64.71.144.202... connected
* Connected to community.letsencrypt.org (64.71.144.202) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
* subject: CN=community.letsencrypt.org
* start date: Sep 13 00:00:28 2021 GMT
* expire date: Dec 12 00:00:27 2021 GMT
* common name: community.letsencrypt.org
* issuer: CN=R3,O=Let's Encrypt,C=US
> HEAD / HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.44 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: community.letsencrypt.org
> Accept: */*
wget -O /dev/null -vS https://$domain
--2021-09-30 17:45:06-- https://community.letsencrypt.org/
Resolving community.letsencrypt.org... 64.71.144.202, 2602:fd3f:3:ff01::ca
Connecting to community.letsencrypt.org|64.71.144.202|:443... connected.
ERROR: cannot verify community.letsencrypt.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
Issued certificate has expired.
To connect to community.letsencrypt.org insecurely, use `--no-check-certificate'.
echo -n | strace openssl s_client -servername $domain -showcerts -connect $domain:443 |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 3
strace curl -Iv https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/certs/ca-bundle.crt", O_RDONLY) = 10
strace wget -O /dev/null -vS https://$domain |& grep open | egrep '\.pem|\.crt'
open("/etc/pki/tls/cert.pem", O_RDONLY) = 4
openssl, wget and curl all read the same file
ls -lah /etc/pki/tls/cert.pem
lrwxrwxrwx 1 root root 19 Dec 1 2020 /etc/pki/tls/cert.pem -> certs/ca-bundle.crt
Looks like blacklisting the DST isn''t enough on CentOS 6
cat /etc/pki/ca-trust/source/blacklist/DST-Root-CA-X3.pem
-----BEGIN CERTIFICATE-----
MIIDSjCCAjKgAwIBAgIQRK+wgNajJ7qJMDmGLvhAazANBgkqhkiG9w0BAQUFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTAwMDkzMDIxMTIxOVoXDTIxMDkzMDE0MDExNVow
PzEkMCIGA1UEChMbRGlnaXRhbCBTaWduYXR1cmUgVHJ1c3QgQ28uMRcwFQYDVQQD
Ew5EU1QgUm9vdCBDQSBYMzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEB
AN+v6ZdQCINXtMxiZfaQguzH0yxrMMpb7NnDfcdAwRgUi+DoM3ZJKuM/IUmTrE4O
rz5Iy2Xu/NMhD2XSKtkyj4zl93ewEnu1lcCJo6m67XMuegwGMoOifooUMM0RoOEq
OLl5CjH9UL2AZd+3UWODyOKIYepLYYHsUmu5ouJLGiifSKOeDNoJjj4XLh7dIN9b
xiqKqy69cK3FCxolkHRyxXtqqzTWMIn/5WgTe1QLyNau7Fqckh49ZLOMxt+/yUFw
7BZy1SbsOFU5Q9D8/RhcQPGX69Wam40dutolucbY38EVAjqr2m7xPi71XAicPNaD
aeQQmxkqtilX4+U9m5/wAl0CAwEAAaNCMEAwDwYDVR0TAQH/BAUwAwEB/zAOBgNV
HQ8BAf8EBAMCAQYwHQYDVR0OBBYEFMSnsaR7LHH62+FLkHX/xBVghYkQMA0GCSqG
SIb3DQEBBQUAA4IBAQCjGiybFwBcqR7uKGY3Or+Dxz9LwwmglSBd49lZRNI+DT69
ikugdB/OEIKcdBodfpga3csTS7MgROSR6cz8faXbauX+5v3gTt23ADq1cEmv8uXr
AvHRAosZy5Q6XkjEGB5YGV8eAlrwDPGxrancWYaLbumR9YbK+rlmM6pZW87ipxZz
R8srzJmwN0jP41ZL9c8PDHIyh8bwRLtTcm1D9SZImlJnt1ir/md2cXjbDaJWFBM5
JDGFoqgCWjBH4d1QB7wCCZAA62RjYJsWvIjJEubSfZGL+T0yjWW06XyxV3bqxbYo
Ob8VZRzI9neWagqNdwvYkQsEjgfbKbYK7p2CNTUQ
-----END CERTIFICATE-----
grep 'DST Root CA X3' /etc/pki/tls/certs/ca-bundle.crt /etc/pki/tls/certs/ca-bundle.trust.crt /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem /etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt
/etc/pki/tls/certs/ca-bundle.trust.crt:# DST Root CA X3
/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt:# DST Root CA X3
seems to be picked up from /etc/pki/tls/cert.pem
grep DST /etc/pki/tls/cert.pem
Issuer: O=Digital Signature Trust Co., CN=DST Root CA X3
Subject: O=Digital Signature Trust Co., CN=DST Root CA X3
/X1PzaBB4DSTv8vihpw3kpBWHNzrKQXlxJ7HNd+KDM3FIUPpqojlNcAZQmNaAl6k
yum -q provides /etc/pki/tls/cert.pem
ca-certificates-2018.2.22-65.1.el6.noarch : The Mozilla CA root certificate bundle
Repo : base
Matched from:
Filename : /etc/pki/tls/cert.pem
ca-certificates-2019.2.32-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo : updates
Matched from:
Filename : /etc/pki/tls/cert.pem
ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo : updates
Matched from:
Filename : /etc/pki/tls/cert.pem
ca-certificates-2020.2.41-65.1.el6_10.noarch : The Mozilla CA root certificate bundle
Repo : installed
Matched from:
Other : Provides-match: /etc/pki/tls/cert.pem
edit: ah Redhat issued new ca-certificate packages for RH 8, 7 Let's Encrypt DST X3 Root Certificate Expiration - Red Hat Customer Portal
Updated System Certificate Packages
We have released updated ca-certificates packages for RHEL6 ELS, RHEL7 and RHEL8.
- RHEL 8 - ca-certificates-2021.2.50-80.0.el8_4.noarch.rpm - RHBA-2021:3573
- RHEL 7 - ca-certificates-2021.2.50-72.el7_9.noarch.rpm - RHBA-2021:3649
- RHEL 6 ELS - ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm - RHBA-2021:3649
These can be updated via the normal yum or dnf update commands.
There is no updated package for the non-ELS RHEL6. The client workaround below can be used to remove the expired certificate from the system bundle.
Updated CentOS 6 x86 yum packages
x86_64
ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm SHA-256: cd59b687781bf3795e1a2ba3d19b1eba29dc08505ad0c4ae5ae8a1a3b1369ad9
nss-3.44.0-11.el6_10.i686.rpm SHA-256: 7b1ddfa75b8af481a2b95b5284b266969a646776795af2f0f6239a639a4814bf
nss-3.44.0-11.el6_10.x86_64.rpm SHA-256: a8ddde535581e3bb5bd059c02f62cee9e352e568fcd091e33016d1e83e8ec836
nss-debuginfo-3.44.0-11.el6_10.i686.rpm SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.i686.rpm SHA-256: 17b5d29d589aaa01638e52e9d5b7aa9e6b71eebabc84e524b7cd4557e73cb76d
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-debuginfo-3.44.0-11.el6_10.x86_64.rpm SHA-256: dded258c107ad645ae62708c051446a86f8cb33f1693e005a2aaf8e16f65f025
nss-devel-3.44.0-11.el6_10.i686.rpm SHA-256: 1512614cc1e0c7874d6f7f397eac1b9fc7a3ccdc28e285e463f93323c5e681f6
nss-devel-3.44.0-11.el6_10.x86_64.rpm SHA-256: 66b09eb27e0492b1d50604a98039559c9d849db6cb0694ad0db99a07eda1e8c4
nss-pkcs11-devel-3.44.0-11.el6_10.i686.rpm SHA-256: 0e8cad432a9c2d81e342cd126fb92f12db44339ec099290e56059adb622437cf
nss-pkcs11-devel-3.44.0-11.el6_10.x86_64.rpm SHA-256: f48020e682d84993ad9f38628b3a670bb63232b281d463578c7eca86a0d4b2c8
nss-sysinit-3.44.0-11.el6_10.x86_64.rpm SHA-256: 717aca2880a7207f34a3fba1173e1d79081e5260c9432439c906dddb8b584dd4
nss-tools-3.44.0-11.el6_10.x86_64.rpm SHA-256: e55a3fec6fcf0493f0a3b0e32f81cc9690d4f9b38e315c1ec53869c82db4a4a
and below so the blacklist didn't work because you still needed a CentOS 6 ca-certificates YUM update to remove the reference to DST Root CA X3
Client Connection Workaround
To work around the openssl client problem on RHEL 6 first ensure your ca-certificates package is updated to the most recently available in your RHEL6 channels ca-certificates-2020.2.41-65.1.el6_10.noarch.rpm .
Then to remove the expired root CA from the system trust store,
- Create an exclusion file:
# perl -e 'while(<>){last if $_ =~ m/DST Root CA X3/;}print $_;while(<>){last if length($_)==1;print $_}' </etc/pki/tls/certs/ca-bundle.crt > /etc/pki/ca-trust/source/blacklist/DST_Root_CA_X3.pem
- Update the system trust store:
# update-ca-trust extract
that is for client workaround and they also have a longer guide for server workaround to fix the chain
Based on my tests I did earlier this year, even if you manage to remove DST Root CA X3 from the trust store, this won't solve the problem (on CentOS 6/OpenSSL 1.0.1). OpenSSL will then start printing "unable to get local issuer certificate", doesn't matter what you add or remove from the trust store.
The workaround with removing the expired root requires OpenSSL's partial chain feature, which is only available in 1.0.2 and up. If RHEL has made patches to OpenSSL 1.0.1 to backport that feature, then you have a chance.
I updated my previous post, reason removing DST isn't enough is you also need updated ca-certificates package for CentOS 6 it seems = RHEL 6 ELS - ca-certificates-2021.2.50-60.1.el6_10.noarch.rpm
Can you confirm that this resolves the connection issues on OpenSSL 1.0.1/CentOS 6? I don't have access to the patches Red Hat has made to the libraries, so I can't check.
yeah I don't have access to ELS CentOS 6 either heh so issues I outlined above still apply openssl and wget don't work but curl does
Right now looks like you'd need to rely on letsencrypt sites like community.letsencrypt.org
to fix their server side chains.
I manually built and installed a version of ca-certificates for RHEL6 and this isn't enough. OpenSSL 1.0.1e is the real problem. Blacklisting the certificate manually produces the same result as updating the ca-certificates package to the latest version from ELS (or backporting from EL7, as I did), which is to remove if from the ca-bundle file. RH has not released any updated version of OpenSSL for RHEL6 so there is currently no fix for the root cause.
For servers I manage that my CentOS 6 server needs to connect to, I used certbot with the --preferred-chain "ISRG Root X1"
option to remove the DST X3 cross-signature from the chain. certbot version 1.12 or later is needed for this option to work. It breaks the special Android compatibility that Let's Encrypt set up, but it makes it so my CentOS 6 server can connect to them without OpenSSL becoming confused. This only works if you can control the remote server your CentOS 6 is connecting to, of course.
Yeah that's the only way now fixing server side chain of the site you're trying to connect to to allow CentOS 6 clients like wget and openssl to work properly. Unfortunately, if you don't control the remote site, you're out of luck heh.
I use acme.sh client for issuance so preferred chain flag is supported on command line right now. Hoping it can be added a system default flag Setting Preferred Chain system wide? · Issue #3717 · acmesh-official/acme.sh · GitHub by @Neilpang
Thanks for the confirmation, this matches my expectations. Sadly this is not great for those running these old versions.
So my initial statement still holds, right now there is not much that can be done for CentOS/RHEL 6. I strongly recommend upgrading, if that's possible in any way shape or form.
Otherwise, I don't know if there's still someone willing to support this OS, but if there is: It may be possible to backport the partial/alternate chain behaviour of 1.0.2 to 1.0.1*, which could enable "remove-the-old-root" workarounds. This requires patching OpenSSL though.
Using LE's short alternate chain is definetly a fix, if all servers are under your control and old Android compatibility is not required.
*I just made a trip to the source code. The basic commit that enables the 1.0.2 workaround was this one, but you can't backport that directly, because a) the code on 1.0.1 is slightly different and b) there were issues with that code, later fixed in followup commits (one, two, three + at least three more). May be easier to just update everything to 1.0.2...
It's my understanding that OpenSSL 1.0.1 and 1.0.2 are intended to be ABI compatible. I'm currently doing some experimentation with rebuilding openssl-1.0.2k-21 (latest for CentOS 7) for CentOS 6. If that proves to work, it could be an option for anyone that needs this to work for a while longer. It needs to be combined with a ca-certificates update to version 2021.2.50 or removal of the DST cert from the trust store to have any chance of working. I'll try to report back with results soon.
I was able to successfully build the package after modifying a few things, and it installed without an issue. Programs built against openssl still run, and wget and openssl s_client work correctly and validate the certificate. This success requires you also have either removed the DST Root CA X3 certificate or have the EL6 ELS version of ca-certificates installed.
I picked up the source RPM from https://vault.centos.org/7.9.2009/updates/Source/SPackages/openssl-1.0.2k-21.el7_9.src.rpm
To build, I had to change ALL references in three patches from secure_getenv() to getenv():
- openssl-1.0.2a-env-zlib.patch
- openssl-1.0.2j-deprecate-algos.patch
- openssl-1.0.2a-fips-ctor.patch
I also had to comment out this line (216) from the openssl.spec file:
%patch68 -p1 -b .secure-getenv
These references to secure_getenv() can't be used because glibc in CentOS 6 doesn't have the function. RH blindly added them without checking for support because they knew it was safe to assume CentOS 7 would have it.
Your mileage may vary, of course. Anything that depends on the specific 1.0.1e version might break, or might have incorrect expectations about what security features are available, leading to unexpected bad stuff, but so far I haven't had it happen to me. I'm not giving out any warranties or prebuilt packages, but hopefully this is enough to get people pointed in the right direction.
Edit: Just in case it may also help: To produce an updated version of ca-certificates, you can do something like this. Take these two packages:
- https://vault.centos.org/7.9.2009/updates/Source/SPackages/ca-certificates-2021.2.50-72.el7_9.src.rpm
- https://vault.centos.org/6.10/updates/Source/SPackages/ca-certificates-2020.2.41-65.1.el6_10.src.rpm
Extract certdata.txt from the el7 SRPM and replace the file at SOURCES/certdata.txt for the el6 rpm build. Update the EL6 RPM spec file Version line to 2021.2.50, and build that package.
thanks @kstange was trying to find a way of just rebuild the ca-certificate cert files themselves - not sure if any info at GitHub - rjrelyea/ca-certificate-scripts: script to manage ca-certificates. helps
looks like cpanel users getting hit by this too https://forums.cpanel.net/threads/cpanel-33077-letsencrypt-transition-to-isrgs-root-important.673981/post-2877037
cloudlinux ELS 6 has updated packages for ca-certificates and seems to work https://forums.cpanel.net/threads/cpanel-33077-letsencrypt-transition-to-isrgs-root-important.673981/post-2877329 and DST Root CA X3 Expiration and Let's Encrypt – cPanel
Coukd you please share the package?
I successfully build both packages, installec ca-certificates without any problems, but i have problems to install openssl-* since it is now splitting into many rpms like openssl-libs, openssl-static and so on...
How did you manage to install new openssl-1.0.2 packages without RPM DB conflits/problems?
I have bunch of errors like this:
https://paste.centos.org/view/82414818
Which is strange, because libcrypto.so.10 is in openssl-devel-1.0.2 for 100%
I was able to build and install openssl 1.0.2k-8 package. I can provide you with SRPM if you'd like. It wasn't build for Centos, but RHEL based OS. Shouldn't be any different.
rpm --test -Uvf openssl-1.0.2k-21.el6.x86_64.rpm openssl-libs-1.0.2k-21.el6.x86_64.rpm
If ok, delete --test param