RHEL/CentOS 7 OpenSSL client compatibility after new chain

Seems this works for CentOS 7 and OpenSSL 1.0.2 at least. For CentOS 6 and OpenSSL 1.0.1 it doesn't seem to work though. Which seems to match @Nummer378 experience.

4 Likes

Yes, this is now a documented workaround for OpenSSL 1.0.2 only:

https://www.openssl.org/blog/blog/2021/09/13/LetsEncryptRootCertExpire/

(1.1+ is fixed anyway, doesn't work on 1.0.1)

5 Likes

I would hope RHEL 7 / CentOS 7 will push an updated ca-certificates package soon after Sep 30, with the expired DST Root CA X3 removed, fixing the problem without further manual intervention.

4 Likes

Does it ever happen that quickly?
If you want to remove it, you can always do so yourself; But I don't think they update it that often.
See:

-----BEGIN CERTIFICATE-----
MIIF0DCCBLigAwIBAgIEOrZQizANBgkqhkiG9w0BAQUFADB/MQswCQYDVQQGEwJC
TTEZMBcGA1UEChMQUXVvVmFkaXMgTGltaXRlZDElMCMGA1UECxMcUm9vdCBDZXJ0
aWZpY2F0aW9uIEF1dGhvcml0eTEuMCwGA1UEAxMlUXVvVmFkaXMgUm9vdCBDZXJ0
aWZpY2F0aW9uIEF1dGhvcml0eTAeFw0wMTAzMTkxODMzMzNaFw0yMTAzMTcxODMz
MzNaMH8xCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1pdGVkMSUw
IwYDVQQLExxSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MS4wLAYDVQQDEyVR
dW9WYWRpcyBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MIIBIjANBgkqhkiG
9w0BAQEFAAOCAQ8AMIIBCgKCAQEAv2G1lVO6V/z68mcLOhrfEYBklbTRvM16z/Yp
li4kVEAkOPcahdxYTMukJ0KX0J+DisPkBgNbAKVRHnAEdOLB1Dqr1607BxgFjv2D
rOpm2RgbaIr1VxqYuvXtdj182d6UajtLF8HVj71lODqV0D1VNk7feVcxKh7YWWVJ
WCCYfqtffp/p1k3sg3Spx2zY7ilKhSoGFPlU5tPaZQeLYzcS19Dsw3sgQUSj7cug
F+FxZc4dZjH3dgEZyH0DWLaVSR2mEiboxgx24ONmy+pdpibu5cxfvWenAScOospU
xbF6lR1xHkopigPcakXBpBlebzbNw6Kwt/5cOOJSvPhEQ+aQuwIDAQABo4ICUjCC
Ak4wPQYIKwYBBQUHAQEEMTAvMC0GCCsGAQUFBzABhiFodHRwczovL29jc3AucXVv
dmFkaXNvZmZzaG9yZS5jb20wDwYDVR0TAQH/BAUwAwEB/zCCARoGA1UdIASCAREw
ggENMIIBCQYJKwYBBAG+WAABMIH7MIHUBggrBgEFBQcCAjCBxxqBxFJlbGlhbmNl
IG9uIHRoZSBRdW9WYWRpcyBSb290IENlcnRpZmljYXRlIGJ5IGFueSBwYXJ0eSBh
c3N1bWVzIGFjY2VwdGFuY2Ugb2YgdGhlIHRoZW4gYXBwbGljYWJsZSBzdGFuZGFy
ZCB0ZXJtcyBhbmQgY29uZGl0aW9ucyBvZiB1c2UsIGNlcnRpZmljYXRpb24gcHJh
Y3RpY2VzLCBhbmQgdGhlIFF1b1ZhZGlzIENlcnRpZmljYXRlIFBvbGljeS4wIgYI
KwYBBQUHAgEWFmh0dHA6Ly93d3cucXVvdmFkaXMuYm0wHQYDVR0OBBYEFItLbe3T
KbkGGew5Oanwl4Rqy+/fMIGuBgNVHSMEgaYwgaOAFItLbe3TKbkGGew5Oanwl4Rq
y+/foYGEpIGBMH8xCzAJBgNVBAYTAkJNMRkwFwYDVQQKExBRdW9WYWRpcyBMaW1p
dGVkMSUwIwYDVQQLExxSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5MS4wLAYD
VQQDEyVRdW9WYWRpcyBSb290IENlcnRpZmljYXRpb24gQXV0aG9yaXR5ggQ6tlCL
MA4GA1UdDwEB/wQEAwIBBjANBgkqhkiG9w0BAQUFAAOCAQEAitQUtf70mpKnGdSk
fnIYj9lofFIk3WdvOXrEql494liwTXCYhGHoG+NpGA7O+0dQoE7/8CQfvbLO9Sf8
7C9TqnN7Az10buYWnuulLsS/VidQK2K6vkscPFVcQR0kvoIgR13VRH56FmjffU1R
cHhXHTMe/QKZnAzNCgVPx7uOpHX6Sm2xgI4JVrmcGmD+XcHXetwReNDWXcG31a0y
mQM6isxUJTkxgXsTIlG6Rmyhu576BGxJJnSP0nPrzDCi5upZIof4l/UO/erMkqQW
xFIY6iHOsfHmhIHluqmGKPJDWl0Snawe2ajlCmqnf6CHKc/yiU3U7MXi5nrQNiOK
SnQ2+Q==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIDIDCCAgigAwIBAgIBHTANBgkqhkiG9w0BAQUFADA5MQswCQYDVQQGEwJGSTEP
MA0GA1UEChMGU29uZXJhMRkwFwYDVQQDExBTb25lcmEgQ2xhc3MyIENBMB4XDTAx
MDQwNjA3Mjk0MFoXDTIxMDQwNjA3Mjk0MFowOTELMAkGA1UEBhMCRkkxDzANBgNV
BAoTBlNvbmVyYTEZMBcGA1UEAxMQU29uZXJhIENsYXNzMiBDQTCCASIwDQYJKoZI
hvcNAQEBBQADggEPADCCAQoCggEBAJAXSjWdyvANlsdE+hY3/Ei9vX+ALTU74W+o
Z6m/AxxNjG8yR9VBaKQTBME1DJqEQ/xcHf+Js+gXGM2RX/uJ4+q/Tl18GybTdXnt
5oTjV+WtKcT0OijnpXuENmmz/V52vaMtmdOQTiMofRhj8VQ7Jp12W5dCsv+u8E7s
3TmVToMGf+dJQMjFAbJUWmYdPfz56TwKnoG4cPABi+QjVHzIrviQHgCWctRUz2Ej
vOr7nQKV0ba5cTppCD8PtOFCx4j1P5iop7oc4HFx71hXgVB6XGt0Rg6DA5jDjqhu
8nYybieDwnPz3BjotJPqdURrBGAgcVeHnfO+oJAjPYok4doh28MCAwEAAaMzMDEw
DwYDVR0TAQH/BAUwAwEB/zARBgNVHQ4ECgQISqCqWITTXjwwCwYDVR0PBAQDAgEG
MA0GCSqGSIb3DQEBBQUAA4IBAQBazof5FnIVV0sd2ZvnoiYw7JNn39Yt0jSv9zil
zqsWuasvfDXLrNAPtEwr/IDva4yRXzZ299uzGxnq9LIR/WFxRL8oszodv7ND6J+/
3DEIcbCdjdY0RzKQxmUk96BKfARzjzlvF4xytb1LyHr4e4PDKE6cCepnP7JnBBvD
FNr450kkkdAdavphOe9r5yF1BgfYErQhIHBCcYHaPJo2vqZbDWpsmh+Re/n570K6
Tk6ezAyNlNzZRZxe7EJQY670XcSxEtzKO6gunRRaBXW37Ndj4ro1tgQIkejanZz2
ZrUYrAqmVCY0M9IbwdR/GjqOC6oybtv8TyWf2TLHllpwrN9M
-----END CERTIFICATE-----

Some examples of expired certs that remain in the ca-certificates.crt file.

3 Likes

Retaining expired certificates in trust stores is sometimes required, in order to verify (timestamped) signatures from the past, when the certificate was not yet expired. This is especially important with certificates used for code signing, as those signatures do not always have expiry dates (but the roots they're signed with do). All of this depends on what the trust store is supposed to do, and how it integrates with path validators.

Regarding the ca-certificates package, their changelog (at Debian) suggests they do actually remove expired certificates*, but sometimes it does take them a while to do so. The package seems to be only getting updates once a year lately. The RHEL/CentOS package with the same name may handle things differently**.

*which makes sense considering that the package was never intended for anything non-TLS.

**Found the CentOS changelog here, looks similar to what Debian does.

4 Likes

Indeed, openssl.org is what I followed :slight_smile:

Yeah one can only hope!

Nice find!

4 Likes

Looks like CentOS 7 YUM update is available for up to date ca-certificates RPM which updates the system CA Trust store and removes the soon to expire CA cert.

yum list updates -q
Updated Packages ca-certificates.noarch 2021.2.50-72.el7_9 updates
rpm -qa --changelog ca-certificates | head -n5
* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate. 
- Removing:
- # Certificate "DST Root CA X3"
8 Likes

That is excellent news and prevents the need for a manual workaround.

4 Likes

Sure does. Bring on September 30th!

2 Likes

I just want to mention that system applications should be fine but if you deployed a custom Python application which uses requests in a virtualenv your ca-bundle will be provided by certifi and they still provide the expired certificate.

So if your Python application makes outgoing requests and tries to verify the TLS certificate you should pass the path to the system-wide CA bundle.

5 Likes

OK, I see ca-certificates.noarch 2021.2.50-72.el7_9

But Not the sep 14 fix-

rpm -qa --changelog ca-certificates | head -n5

  • Update to CKBI 2.22 from NSS 3.35
  • Update to CKBI 2.20 from NSS 3.34.1
1 Like

You are querying your currently installed package. Have you already installed the update?

2 Likes

@Niamh Seems like you did not install updates since 2018... :sleepy:

The changelog should look like this:

* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate.
-    Removing:
-     # Certificate "DST Root CA X3"

* Wed Jun 16 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-71
- Update to CKBI 2.50 from NSS 3.67
   - version number update only

* Fri Jun 11 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.48-71
- Update to CKBI 2.48 from NSS 3.66
-    Removing:
...

You can verify the installed version like this: $ rpm -q ca-certificates

2 Likes

I ran into this yesterday. I had a fresh centos 7 minimal install with the latest updates. My ansible playbook that installs the postgresql repo from https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm started failing with an SSL verification error. wget reported the following:

ERROR: cannot verify download.postgresql.org's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
  Issued certificate has expired.

However, I have the updated package

rpm -q ca-certificates
ca-certificates-2021.2.50-72.el7_9.noarch
rpm -qa --changelog ca-certificates | head -n5
 Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
 Fix expired certificate.
   Removing:
     # Certificate "DST Root CA X3"

And sure enough this returns no entries:

trust list | grep -C3 'DST Root CA X3'

On a whim I thought that the package didn't update the extract when it was installed so I ran this manually:

update-ca-trust extract

And now everything is working as expected. So now I need to write some ansible code to somehow detect that the extract needs to be run.

3 Likes

Great that this resolved the issue for you. However, as one should expect, haven't seen anywhere on new or existing CentOS 7 boxes that this was needed after updating ca-certificates.

If you look at the output of rpm -q --scripts ca-certificates you can see that running update-ca-trust is already managed by the package itself.

Just wanting to clarify in this thread that it is ordinarily not required to run update-ca-trust extract after updating.

2 Likes

Good info, @FelixSchwarz! One easy way to do this should be to set the REQUESTS_CA_BUNDLE environment variable.

2 Likes

Guess spoke too soon, though eventually got my CentOS 7 setups for my Centmin Mod LEMP stack systems sorted out including the option to just switch away from Letsencrypt to ZeroSSL based SSL certificates https://blog.centminmod.com/2021/10/02/2425/centmin-mod-managing-letsencrypt-dst-root-ca-x3-certificate-expiration-on-centos-7/ :slight_smile:

1 Like

Oh, I see they also provide a way to switch back from ZeroSSL to LetsEncrypt by changing the ACME_DEFAULT_CA='zerossl' back to ACME_DEFAULT_CA='letsencrypt'.
Thank you for posting that link, @eva2000! :smiley:

5 Likes

Yes, if you use acme.sh directly see ZeroSSL.com CA · acmesh-official/acme.sh Wiki · GitHub and Change default CA to ZeroSSL · acmesh-official/acme.sh Wiki · GitHub. The same method allows you to switch acme.sh to Buypass Go SSL ACME free certs too.

2 Likes