RHEL/CentOS 7 OpenSSL client compatibility after new chain

Looks like CentOS 7 YUM update is available for up to date ca-certificates RPM which updates the system CA Trust store and removes the soon to expire CA cert.

yum list updates -q
Updated Packages ca-certificates.noarch 2021.2.50-72.el7_9 updates
rpm -qa --changelog ca-certificates | head -n5
* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate. 
- Removing:
- # Certificate "DST Root CA X3"
8 Likes

That is excellent news and prevents the need for a manual workaround.

4 Likes

Sure does. Bring on September 30th!

2 Likes

I just want to mention that system applications should be fine but if you deployed a custom Python application which uses requests in a virtualenv your ca-bundle will be provided by certifi and they still provide the expired certificate.

So if your Python application makes outgoing requests and tries to verify the TLS certificate you should pass the path to the system-wide CA bundle.

5 Likes

OK, I see ca-certificates.noarch 2021.2.50-72.el7_9

But Not the sep 14 fix-

rpm -qa --changelog ca-certificates | head -n5

  • Update to CKBI 2.22 from NSS 3.35
  • Update to CKBI 2.20 from NSS 3.34.1
1 Like

You are querying your currently installed package. Have you already installed the update?

2 Likes

@Niamh Seems like you did not install updates since 2018... :sleepy:

The changelog should look like this:

* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate.
-    Removing:
-     # Certificate "DST Root CA X3"

* Wed Jun 16 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-71
- Update to CKBI 2.50 from NSS 3.67
   - version number update only

* Fri Jun 11 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.48-71
- Update to CKBI 2.48 from NSS 3.66
-    Removing:
...

You can verify the installed version like this: $ rpm -q ca-certificates

2 Likes

I ran into this yesterday. I had a fresh centos 7 minimal install with the latest updates. My ansible playbook that installs the postgresql repo from https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm started failing with an SSL verification error. wget reported the following:

ERROR: cannot verify download.postgresql.org's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
  Issued certificate has expired.

However, I have the updated package

rpm -q ca-certificates
ca-certificates-2021.2.50-72.el7_9.noarch
rpm -qa --changelog ca-certificates | head -n5
 Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
 Fix expired certificate.
   Removing:
     # Certificate "DST Root CA X3"

And sure enough this returns no entries:

trust list | grep -C3 'DST Root CA X3'

On a whim I thought that the package didn't update the extract when it was installed so I ran this manually:

update-ca-trust extract

And now everything is working as expected. So now I need to write some ansible code to somehow detect that the extract needs to be run.

3 Likes

Great that this resolved the issue for you. However, as one should expect, haven't seen anywhere on new or existing CentOS 7 boxes that this was needed after updating ca-certificates.

If you look at the output of rpm -q --scripts ca-certificates you can see that running update-ca-trust is already managed by the package itself.

Just wanting to clarify in this thread that it is ordinarily not required to run update-ca-trust extract after updating.

2 Likes

Good info, @FelixSchwarz! One easy way to do this should be to set the REQUESTS_CA_BUNDLE environment variable.

2 Likes

Guess spoke too soon, though eventually got my CentOS 7 setups for my Centmin Mod LEMP stack systems sorted out including the option to just switch away from Letsencrypt to ZeroSSL based SSL certificates https://blog.centminmod.com/2021/10/02/2425/centmin-mod-managing-letsencrypt-dst-root-ca-x3-certificate-expiration-on-centos-7/ :slight_smile:

1 Like

Oh, I see they also provide a way to switch back from ZeroSSL to LetsEncrypt by changing the ACME_DEFAULT_CA='zerossl' back to ACME_DEFAULT_CA='letsencrypt'.
Thank you for posting that link, @eva2000! :smiley:

5 Likes

Yes, if you use acme.sh directly see ZeroSSL.com CA · acmesh-official/acme.sh Wiki · GitHub and Change default CA to ZeroSSL · acmesh-official/acme.sh Wiki · GitHub. The same method allows you to switch acme.sh to Buypass Go SSL ACME free certs too.

2 Likes

FYI, with acme.sh v3.0.1+ you can also now set the preferred chain system wide by default Preferred Chain · acmesh-official/acme.sh Wiki · GitHub

i.e. setting shorter ISRG preferred chain system wide by default with letsencrypt and then renewing all certificates

acme.sh --upgrade
acme.sh --set-default-chain --preferred-chain "ISRG" --server  letsencrypt
acme.sh --renewAll --force
2 Likes

CentOS 7 users with certbot from EPEL unfortunately cannot use --prefered-chain because the latest package in EPEL is only 1.11.0. @FelixSchwarz since you have been working on these packages, any idea what we can do to update certbot in EPEL7?

1 Like

Hey Pim,

the problem is that certbot 1.12 requires Python 3. EPEL ships all certbot plugins this creates a pretty impressive dependency tree. I had two fix/tweak/adapt roughly two dozen packages to get all the required plugins. Unfortunately the project stalled about 4 months ago as I did not have enough free time to finish the transition but most pieces are actually in place already.

If you only need certbot (or certbot-nginx/a few select plugins) and you could try my COPR repo.

However that COPR was only meant for my packaging experiments so no actual deployment testing. The stuff that was built should work but this is really not tested like the usual certbot updates you get via EPEL. Also certbot-apache is not yet available in my copr (missing Python 3 version for augeas).

I have to admit that I missed the problem that certbot 1.11 does not support "--preferred-chain". If you need a fix within the next weeks the best way forward is likely to use a custom virtualenv and install certbot there until EPEL 7 packages are ready. If you only use packages which I already built successfully and the machine is not really critical you could try my COPR and check if everything is working.

I'll try to get the transition in the next weeks but there are a lot of packages involved so I can't promise a quick solution.

3 Likes

Ouch! I’ll find my way using a virtualenv for now.

Thanks a lot for your detailed explanation. And hopefully you will be able to get all dependencies ready without headaches.

1 Like

How is that a problem, in a time where Python 2.7. should not be in use any longer?

1 Like

Redhat/CentOS 7.x all still default to Python 2.7 by default How is Python 2 supported in RHEL after 2020? - Red Hat Customer Portal & https://blog.centminmod.com/2021/01/05/2189/centos-7-python-2-7-upgrade-deprecation-end-of-life-january-1st-2020/

To save headaches, I don't use certbot due to python usage. Using acme.sh for past 5+ yrs for Letsencrypt SSL certs fine GitHub - acmesh-official/acme.sh: A pure Unix shell script implementing ACME client protocol :slight_smile:

1 Like

Well, unfortunately my Gentoo also requires the presence of Python 2.7 (so I can compile Firefox or Chromium I believe..),, but that's no excuse not to be able to install Python 3.x IMO.

2 Likes