RHEL/CentOS 7 OpenSSL client compatibility after new chain

As was announced here and on OpenSSL Client Compatibility Changes for Let’s Encrypt Certificates there will be a change in the certificate chain on June 2021 removing compatibilty for clients with OpenSSL < 1.1.0.

Many platforms are relying on Enterprise Linux distributions with older versions of OpenSSL, but are still supported and expected to run for many years to come.

Which fix that was released in OpenSSL 1.1.0 is required for compatiblity with the new chain? I would like to check if this fix was backported to OpenSSL 1.0.2k in RHEL/CentOS 7.

1 Like

Based on this m.d.s.p discussion , I would guess it's this change. It doesn't look like it's been backported, based on the openssl-1.0.2k-21.el7_9 SRPM from CentOS 7.

2 Likes

CentOS 7 has the optional X509_V_FLAG_TRUSTED_FIRST feature present in openssl 1.0.2 (exposed as -trusted_first flag for the openssl command line tools) but not enabled by default. Your application needs to enable it explicitly.

CentOS 8 with openssl 1.1.1 has it enabled by default.

4 Likes

I suggest migrating to CentOS 8 as soon as possible

Unfortunately, CentOS 8 is going EOL at the end of this year, whereas CentOS 7 will be kicking around until 2024.

At least for CentOS 7, there's an enormous number of cPanel servers which are probably going to stay around until the bitter end, which makes me a little worried about the impact. Those can't be upgraded and it's a big effort to migrate customers off them.

Edit: Using @Nummer378's great test cases here, it seems that an up-to-date CentOS 7 can't verify the chain:

[root@plugindev ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)

[root@plugindev ~]# rpm -q openssl
openssl-1.0.2k-21.el7_9.x86_64

[root@plugindev ~]# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile certs-combined.pem
verify depth is 1
CONNECTED(00000003)
depth=3 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
verify error:num=10:certificate has expired
notAfter=Jan 30 14:01:15 2021 GMT
140169193727888:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:s3_clnt.c:1264:
3 Likes

seems that every time I run across CentOS there are more headaches

at present my webserver is using Ubuntu 20.0.4.1 which has about the same as CentOS 7 according to your links. AFAIK Canonical is not changing their strategy but the crystal ball is still a tad foggy

Given the uncertainty it may be prudent to consider another distribution should it be necessary

It does pass verification with the -trusted_first option on CentOS 7:

$ rpm -q openssl
openssl-1.0.2k-21.el7_9.x86_64

$ openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile certs-combined.pem -trusted_first
verify depth is 1
CONNECTED(00000003)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = expired-root-ca-test.germancoding.com
verify return:1

I wonder why X509_V_FLAG_TRUSTED_FIRST is not enabled by default in the OpenSSL packages from RHEL/CentOS 7.

Edit: from my limited understanding it seems that cURL already enables X509_V_FLAG_TRUSTED_FIRST by default.

2 Likes

This is default OpenSSL behaviour. -trusted_first is equivalent to setting X509_V_FLAG_TRUSTED_FIRST in code.

X509_V_FLAG_TRUSTED_FIRST was first introduced to OpenSSL in version 1.0.2, but not enabled by default. Applications need to manually set this flag. This may not be possible for everyone.

Since OpenSSL 1.1.0, the flag is set by default and no longer requires action by the application.

1 Like

Bug report with request to backport this to RHEL 7: https://bugzilla.redhat.com/show_bug.cgi?id=1842174

Workaround documented by RedHat is to remove the expired root from the trust store: Sectigo Root and Intermediate Certificate Expiry - May 2020 - Red Hat Customer Portal.

3 Likes

If you can't access the article, it comes down to:
Put the expired CA certificate(s) in /etc/pki/ca-trust/source/blacklist/, and invoke update-ca-trust(8) to rebuild the trust store. Verify with trust list.

1 Like

I have little hope that this workaround works in Let's Encrypt case, as Sectigos chains were different back then (they also had problems with expired intermediates, lots of cross signs all over the place).

I just tested with the oldest OpenSSL I had lying around (simulating that DST Root CA X3 has been manually removed from the trust store):

# openssl version
OpenSSL 1.0.1j 15 Oct 2014

# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -CAfile letsencrypt-stg-root-x1.pem

verify depth is 1
CONNECTED(00000003)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify error:num=20:unable to get local issuer certificate
verify return:0
140518814185152:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1930:

Note that on these old versions you need to use -CAfile instead of the -verifyCAfile I used in my other examples.

What happens here is that without DST Root CA X3, OpenSSL cannot build a chain at all. We tell it to validate with ISRG Root X1, but it doesn't work, as the server chain never contains ISRG Root X1 as root, only as intermediate. These old pre-1.1 versions apparently do not allow partial chains, so it always wants to validate up to DST Root CA X3 (unless TRUSTED_FIRST is present, which has already been discussed). If DST Root CA X3 is in the trust store, it fails because it's expired. If not, it fails because it can't find it.

Someone with CentOS 7 + OpenSSL 1.0.2 needs to verify if the same applies on that version, as my test above is even older than 1.0.2. I currently don't have a 1.0.2 version at hand, but I believe they behave similar.

1 Like

I can reproduce that failure with OpenSSL 1.0.1 on CentOS 6, but not with OpenSSL 1.0.2 on CentOS 7:

$ cat /etc/redhat-release 
CentOS Linux release 7.9.2009 (Core)

$ openssl version
OpenSSL 1.0.2k-fips  26 Jan 2017

$ wget https://raw.githubusercontent.com/letsencrypt/website/master/static/certs/staging/letsencrypt-stg-root-x1.pem

$ openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -CAfile letsencrypt-stg-root-x1.pem
verify depth is 1
CONNECTED(00000003)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify return:1
depth=1 C = US, O = (STAGING) Let's Encrypt, CN = (STAGING) Artificial Apricot R3
verify return:1
depth=0 CN = expired-root-ca-test.germancoding.com
verify return:1
---
Certificate chain
 0 s:/CN=expired-root-ca-test.germancoding.com
   i:/C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Artificial Apricot R3
 1 s:/C=US/O=(STAGING) Let's Encrypt/CN=(STAGING) Artificial Apricot R3
   i:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Pretend Pear X1
 2 s:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Pretend Pear X1
   i:/C=US/O=(STAGING) Internet Security Research Group/CN=(STAGING) Doctored Durian Root CA X3
---
3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Seems this works for CentOS 7 and OpenSSL 1.0.2 at least. For CentOS 6 and OpenSSL 1.0.1 it doesn't seem to work though. Which seems to match @Nummer378 experience.

4 Likes

Yes, this is now a documented workaround for OpenSSL 1.0.2 only:

https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

(1.1+ is fixed anyway, doesn't work on 1.0.1)

5 Likes

I would hope RHEL 7 / CentOS 7 will push an updated ca-certificates package soon after Sep 30, with the expired DST Root CA X3 removed, fixing the problem without further manual intervention.

4 Likes

Does it ever happen that quickly?
If you want to remove it, you can always do so yourself; But I don't think they update it that often.
See:

-----BEGIN CERTIFICATE-----
MIIF0DCCBLigAwIBAgIEOrZQizANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJC
TTEZMBcGA1UEChMQUXVvVmFkaXMgTGltaXRlZDElMCMGA1UECxMcUm9vdCBDZXJ0
aWZpY2F0aW9uIEF1dGhvcml0eTEuMCwGA1UEAxMlUXVvVmFkaXMgUm9vdCBDZXJ0
aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wMTAzMTkxODMzMzNaFw0yMTAzMTcxODMz
MzNaMH8xCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMSUw
IwYDVQQLExxSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MS4wLAYDVQQDEyVR
dW9WYWRpcyBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv2G1lVO6V/z68mcLOhrfEYBklbTRvM16z/Yp
li4kVEAkOPcahdxYTMukJ0KX0J+DisPkBgNbAKVRHnAEdOLB1Dqr1607BxgFjv2D
rOpm2RgbaIr1VxqYuvXtdj182d6UajtLF8HVj71lODqV0D1VNk7feVcxKh7YWWVJ
WCCYfqtffp/p1k3sg3Spx2zY7ilKhSoGFPlU5tPaZQeLYzcS19Dsw3sgQUSj7cug
F+FxZc4dZjH3dgEZyH0DWLaVSR2mEiboxgx24ONmy+pdpibu5cxfvWenAScOospU
xbF6lR1xHkopigPcakXBpBlebzbNw6Kwt/5cOOJSvPhEQ+aQuwIDAQABo4ICUjCC
Ak4wPQYIKwYBBQUHAQEEMTAvMC0GCCsGAQUFBzABhiFodHRwczovL29jc3AucXVv
dmFkaXNvZmZzaG9yZS5jb20wDwYDVR0TAQH/BAUwAwEB/zCCARoGA1UdIASCAREw
ggENMIIBCQYJKwYBBAG+WAABMIH7MIHUBggrBgEFBQcCAjCBxxqBxFJlbGlhbmNl
IG9uIHRoZSBRdW9WYWRpcyBSb290IENlcnRpZmljYXRlIGJ5IGFueSBwYXJ0eSBh
c3N1bWVzIGFjY2VwdGFuY2Ugb2YgdGhlIHRoZW4gYXBwbGljYWJsZSBzdGFuZGFy
ZCB0ZXJtcyBhbmQgY29uZGl0aW9ucyBvZiB1c2UsIGNlcnRpZmljYXRpb24gcHJh
Y3RpY2VzLCBhbmQgdGhlIFF1b1ZhZGlzIENlcnRpZmljYXRlIFBvbGljeS4wIgYI
KwYBBQUHAgEWFmh0dHA6Ly93d3cucXVvdmFkaXMuYm0wHQYDVR0OBBYEFItLbe3T
KbkGGew5Oanwl4Rqy+/fMIGuBgNVHSMEgaYwgaOAFItLbe3TKbkGGew5Oanwl4Rq
y+/foYGEpIGBMH8xCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1p
dGVkMSUwIwYDVQQLExxSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MS4wLAYD
VQQDEyVRdW9WYWRpcyBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggQ6tlCL
MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOCAQEAitQUtf70mpKnGdSk
fnIYj9lofFIk3WdvOXrEql494liwTXCYhGHoG+NpGA7O+0dQoE7/8CQfvbLO9Sf8
7C9TqnN7Az10buYWnuulLsS/VidQK2K6vkscPFVcQR0kvoIgR13VRH56FmjffU1R
cHhXHTMe/QKZnAzNCgVPx7uOpHX6Sm2xgI4JVrmcGmD+XcHXetwReNDWXcG31a0y
mQM6isxUJTkxgXsTIlG6Rmyhu576BGxJJnSP0nPrzDCi5upZIof4l/UO/erMkqQW
xFIY6iHOsfHmhIHluqmGKPJDWl0Snawe2ajlCmqnf6CHKc/yiU3U7MXi5nrQNiOK
SnQ2+Q==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDIDCCAgigAwIBAgIBHTANBgkqhkiG9w0BAQUFADA5MQswCQYDVQQGEwJGSTEP
MA0GA1UEChMGU29uZXJhMRkwFwYDVQQDExBTb25lcmEgQ2xhc3MyIENBMB4XDTAx
MDQwNjA3Mjk0MFoXDTIxMDQwNjA3Mjk0MFowOTELMAkGA1UEBhMCRkkxDzANBgNV
BAoTBlNvbmVyYTEZMBcGA1UEAxMQU29uZXJhIENsYXNzMiBDQTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAJAXSjWdyvANlsdE+hY3/Ei9vX+ALTU74W+o
Z6m/AxxNjG8yR9VBaKQTBME1DJqEQ/xcHf+Js+gXGM2RX/uJ4+q/Tl18GybTdXnt
5oTjV+WtKcT0OijnpXuENmmz/V52vaMtmdOQTiMofRhj8VQ7Jp12W5dCsv+u8E7s
3TmVToMGf+dJQMjFAbJUWmYdPfz56TwKnoG4cPABi+QjVHzIrviQHgCWctRUz2Ej
vOr7nQKV0ba5cTppCD8PtOFCx4j1P5iop7oc4HFx71hXgVB6XGt0Rg6DA5jDjqhu
8nYybieDwnPz3BjotJPqdURrBGAgcVeHnfO+oJAjPYok4doh28MCAwEAAaMzMDEw
DwYDVR0TAQH/BAUwAwEB/zARBgNVHQ4ECgQISqCqWITTXjwwCwYDVR0PBAQDAgEG
MA0GCSqGSIb3DQEBBQUAA4IBAQBazof5FnIVV0sd2ZvnoiYw7JNn39Yt0jSv9zil
zqsWuasvfDXLrNAPtEwr/IDva4yRXzZ299uzGxnq9LIR/WFxRL8oszodv7ND6J+/
3DEIcbCdjdY0RzKQxmUk96BKfARzjzlvF4xytb1LyHr4e4PDKE6cCepnP7JnBBvD
FNr450kkkdAdavphOe9r5yF1BgfYErQhIHBCcYHaPJo2vqZbDWpsmh+Re/n570K6
Tk6ezAyNlNzZRZxe7EJQY670XcSxEtzKO6gunRRaBXW37Ndj4ro1tgQIkejanZz2
ZrUYrAqmVCY0M9IbwdR/GjqOC6oybtv8TyWf2TLHllpwrN9M
-----END CERTIFICATE-----

Some examples of expired certs that remain in the ca-certificates.crt file.

3 Likes

Retaining expired certificates in trust stores is sometimes required, in order to verify (timestamped) signatures from the past, when the certificate was not yet expired. This is especially important with certificates used for code signing, as those signatures do not always have expiry dates (but the roots they're signed with do). All of this depends on what the trust store is supposed to do, and how it integrates with path validators.

Regarding the ca-certificates package, their changelog (at Debian) suggests they do actually remove expired certificates*, but sometimes it does take them a while to do so. The package seems to be only getting updates once a year lately. The RHEL/CentOS package with the same name may handle things differently**.

*which makes sense considering that the package was never intended for anything non-TLS.

**Found the CentOS changelog here, looks similar to what Debian does.

4 Likes

Indeed, openssl.org is what I followed :slight_smile:

Yeah one can only hope!

Nice find!

4 Likes