Many platforms are relying on Enterprise Linux distributions with older versions of OpenSSL, but are still supported and expected to run for many years to come.
Which fix that was released in OpenSSL 1.1.0 is required for compatiblity with the new chain? I would like to check if this fix was backported to OpenSSL 1.0.2k in RHEL/CentOS 7.
Based on this m.d.s.p discussion , I would guess it's this change. It doesn't look like it's been backported, based on the openssl-1.0.2k-21.el7_9 SRPM from CentOS 7.
CentOS 7 has the optional X509_V_FLAG_TRUSTED_FIRST feature present in openssl 1.0.2 (exposed as -trusted_first flag for the openssl command line tools) but not enabled by default. Your application needs to enable it explicitly.
CentOS 8 with openssl 1.1.1 has it enabled by default.
At least for CentOS 7, there's an enormous number of cPanel servers which are probably going to stay around until the bitter end, which makes me a little worried about the impact. Those can't be upgraded and it's a big effort to migrate customers off them.
Edit: Using @Nummer378's great test cases here, it seems that an up-to-date CentOS 7 can't verify the chain:
[root@plugindev ~]# cat /etc/centos-release
CentOS Linux release 7.9.2009 (Core)
[root@plugindev ~]# rpm -q openssl
openssl-1.0.2k-21.el7_9.x86_64
[root@plugindev ~]# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -verifyCAfile certs-combined.pem
verify depth is 1
CONNECTED(00000003)
depth=3 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Doctored Durian Root CA X3
verify error:num=10:certificate has expired
notAfter=Jan 30 14:01:15 2021 GMT
140169193727888:error:14090086:SSL routines:ssl3_get_server_certificate:certificate verify failed:s3_clnt.c:1264:
seems that every time I run across CentOS there are more headaches
at present my webserver is using Ubuntu 20.0.4.1 which has about the same as CentOS 7 according to your links. AFAIK Canonical is not changing their strategy but the crystal ball is still a tad foggy
Given the uncertainty it may be prudent to consider another distribution should it be necessary
This is default OpenSSL behaviour. -trusted_first is equivalent to setting X509_V_FLAG_TRUSTED_FIRST in code.
X509_V_FLAG_TRUSTED_FIRST was first introduced to OpenSSL in version 1.0.2, but not enabled by default. Applications need to manually set this flag. This may not be possible for everyone.
Since OpenSSL 1.1.0, the flag is set by default and no longer requires action by the application.
If you can't access the article, it comes down to:
Put the expired CA certificate(s) in /etc/pki/ca-trust/source/blacklist/, and invoke update-ca-trust(8) to rebuild the trust store. Verify with trust list.
I have little hope that this workaround works in Let's Encrypt case, as Sectigos chains were different back then (they also had problems with expired intermediates, lots of cross signs all over the place).
I just tested with the oldest OpenSSL I had lying around (simulating that DST Root CA X3 has been manually removed from the trust store):
# openssl version
OpenSSL 1.0.1j 15 Oct 2014
# openssl s_client -connect expired-root-ca-test.germancoding.com:443 -servername expired-root-ca-test.germancoding.com -verify 1 -CAfile letsencrypt-stg-root-x1.pem
verify depth is 1
CONNECTED(00000003)
depth=2 C = US, O = (STAGING) Internet Security Research Group, CN = (STAGING) Pretend Pear X1
verify error:num=20:unable to get local issuer certificate
verify return:0
140518814185152:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1930:
Note that on these old versions you need to use -CAfile instead of the -verifyCAfile I used in my other examples.
What happens here is that without DST Root CA X3, OpenSSL cannot build a chain at all. We tell it to validate with ISRG Root X1, but it doesn't work, as the server chain never contains ISRG Root X1 as root, only as intermediate. These old pre-1.1 versions apparently do not allow partial chains, so it always wants to validate up to DST Root CA X3 (unless TRUSTED_FIRST is present, which has already been discussed). If DST Root CA X3 is in the trust store, it fails because it's expired. If not, it fails because it can't find it.
Someone with CentOS 7 + OpenSSL 1.0.2 needs to verify if the same applies on that version, as my test above is even older than 1.0.2. I currently don't have a 1.0.2 version at hand, but I believe they behave similar.
Seems this works for CentOS 7 and OpenSSL 1.0.2 at least. For CentOS 6 and OpenSSL 1.0.1 it doesn't seem to work though. Which seems to match @Nummer378 experience.
I would hope RHEL 7 / CentOS 7 will push an updated ca-certificates package soon after Sep 30, with the expired DST Root CA X3 removed, fixing the problem without further manual intervention.
Retaining expired certificates in trust stores is sometimes required, in order to verify (timestamped) signatures from the past, when the certificate was not yet expired. This is especially important with certificates used for code signing, as those signatures do not always have expiry dates (but the roots they're signed with do). All of this depends on what the trust store is supposed to do, and how it integrates with path validators.
Regarding the ca-certificates package, their changelog (at Debian) suggests they do actually remove expired certificates*, but sometimes it does take them a while to do so. The package seems to be only getting updates once a year lately. The RHEL/CentOS package with the same name may handle things differently**.
*which makes sense considering that the package was never intended for anything non-TLS.
**Found the CentOS changelog here, looks similar to what Debian does.
