RHEL/CentOS 7 OpenSSL client compatibility after new chain

You are querying your currently installed package. Have you already installed the update?


@Niamh Seems like you did not install updates since 2018... :sleepy:

The changelog should look like this:

* Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
- Fix expired certificate.
-    Removing:
-     # Certificate "DST Root CA X3"

* Wed Jun 16 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-71
- Update to CKBI 2.50 from NSS 3.67
   - version number update only

* Fri Jun 11 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.48-71
- Update to CKBI 2.48 from NSS 3.66
-    Removing:

You can verify the installed version like this: $ rpm -q ca-certificates


I ran into this yesterday. I had a fresh centos 7 minimal install with the latest updates. My ansible playbook that installs the postgresql repo from https://download.postgresql.org/pub/repos/yum/reporpms/EL-7-x86_64/pgdg-redhat-repo-latest.noarch.rpm started failing with an SSL verification error. wget reported the following:

ERROR: cannot verify download.postgresql.org's certificate, issued by '/C=US/O=Let\'s Encrypt/CN=R3':
  Issued certificate has expired.

However, I have the updated package

rpm -q ca-certificates
rpm -qa --changelog ca-certificates | head -n5
 Tue Sep 14 2021 Bob Relyea <rrelyea@redhat.com> - 2021.2.50-72
 Fix expired certificate.
     # Certificate "DST Root CA X3"

And sure enough this returns no entries:

trust list | grep -C3 'DST Root CA X3'

On a whim I thought that the package didn't update the extract when it was installed so I ran this manually:

update-ca-trust extract

And now everything is working as expected. So now I need to write some ansible code to somehow detect that the extract needs to be run.


Great that this resolved the issue for you. However, as one should expect, haven't seen anywhere on new or existing CentOS 7 boxes that this was needed after updating ca-certificates.

If you look at the output of rpm -q --scripts ca-certificates you can see that running update-ca-trust is already managed by the package itself.

Just wanting to clarify in this thread that it is ordinarily not required to run update-ca-trust extract after updating.


Good info, @FelixSchwarz! One easy way to do this should be to set the REQUESTS_CA_BUNDLE environment variable.


Guess spoke too soon, though eventually got my CentOS 7 setups for my Centmin Mod LEMP stack systems sorted out including the option to just switch away from Letsencrypt to ZeroSSL based SSL certificates https://blog.centminmod.com/2021/10/02/2425/centmin-mod-managing-letsencrypt-dst-root-ca-x3-certificate-expiration-on-centos-7/ :slight_smile:

1 Like

Oh, I see they also provide a way to switch back from ZeroSSL to LetsEncrypt by changing the ACME_DEFAULT_CA='zerossl' back to ACME_DEFAULT_CA='letsencrypt'.
Thank you for posting that link, @eva2000! :smiley:


Yes, if you use acme.sh directly see ZeroSSL.com CA · acmesh-official/acme.sh Wiki · GitHub and Change default CA to ZeroSSL · acmesh-official/acme.sh Wiki · GitHub. The same method allows you to switch acme.sh to Buypass Go SSL ACME free certs too.


FYI, with acme.sh v3.0.1+ you can also now set the preferred chain system wide by default Preferred Chain · acmesh-official/acme.sh Wiki · GitHub

i.e. setting shorter ISRG preferred chain system wide by default with letsencrypt and then renewing all certificates

acme.sh --upgrade
acme.sh --set-default-chain --preferred-chain "ISRG" --server  letsencrypt
acme.sh --renewAll --force

CentOS 7 users with certbot from EPEL unfortunately cannot use --prefered-chain because the latest package in EPEL is only 1.11.0. @FelixSchwarz since you have been working on these packages, any idea what we can do to update certbot in EPEL7?

1 Like

Hey Pim,

the problem is that certbot 1.12 requires Python 3. EPEL ships all certbot plugins this creates a pretty impressive dependency tree. I had two fix/tweak/adapt roughly two dozen packages to get all the required plugins. Unfortunately the project stalled about 4 months ago as I did not have enough free time to finish the transition but most pieces are actually in place already.

If you only need certbot (or certbot-nginx/a few select plugins) and you could try my COPR repo.

However that COPR was only meant for my packaging experiments so no actual deployment testing. The stuff that was built should work but this is really not tested like the usual certbot updates you get via EPEL. Also certbot-apache is not yet available in my copr (missing Python 3 version for augeas).

I have to admit that I missed the problem that certbot 1.11 does not support "--preferred-chain". If you need a fix within the next weeks the best way forward is likely to use a custom virtualenv and install certbot there until EPEL 7 packages are ready. If you only use packages which I already built successfully and the machine is not really critical you could try my COPR and check if everything is working.

I'll try to get the transition in the next weeks but there are a lot of packages involved so I can't promise a quick solution.


Ouch! I’ll find my way using a virtualenv for now.

Thanks a lot for your detailed explanation. And hopefully you will be able to get all dependencies ready without headaches.

1 Like

How is that a problem, in a time where Python 2.7. should not be in use any longer?

1 Like

Redhat/CentOS 7.x all still default to Python 2.7 by default How is Python 2 supported in RHEL after 2020? - Red Hat Customer Portal & https://blog.centminmod.com/2021/01/05/2189/centos-7-python-2-7-upgrade-deprecation-end-of-life-january-1st-2020/

To save headaches, I don't use certbot due to python usage. Using acme.sh for past 5+ yrs for Letsencrypt SSL certs fine GitHub - acmesh-official/acme.sh: A pure Unix shell script implementing ACME client protocol :slight_smile:

1 Like

Well, unfortunately my Gentoo also requires the presence of Python 2.7 (so I can compile Firefox or Chromium I believe..),, but that's no excuse not to be able to install Python 3.x IMO.


CentOS/Redhat 7 users can just install Python 3.x side by side :slight_smile:

1 Like

Soooo, the following:

..actually isn't a problem then? :thinking:

1 Like

No, Python 3 is not necessarily the issue here. The issue is that a lot of dependencies have to be packaged due to EPEL shipping all certbot plugins as well. An update simply means a huge amount of work for @FelixSchwarz and co.


When CentOS 7 was released (June 2014) it did not provide a Python 3 package. Fedora EPEL 7 did include Python 3.4 much later, then upgraded to Python 3.6 and at some point also Red Hat included that Python 3.6 in their base distribution.

So you can install the Python 3.6 interpreter with the standard library even on CentOS 7. The problem is that Fedora EPEL ships all official certbot plugins which have a significant dependency tree. In EPEL 7 we started to ship certbot 0.6 in 2016! By that time the Fedora maintainers relied on Python 2 as the Python 3 stack was too incomplete on CentOS 7.

Fast forward a few years: Since certbot 1.12 Python 3 is required. Still there are only 1-2 persons in Fedora who did some work on the certbot stack and mind you - both are volunteers with day jobs, family and a lot of other Python packages. For example my pagure profile says I have access to 521 packages - though most of these came through the Python maintainers group. Even if I can leave most of them to fellow maintainers there are roughly 50 packages where I'm the primary committer. Most notable parts are the certbot stack (~ 30 packages), the WeasyPrint stack(~ 10 packages) and BorgBackup.

Even after the Python 3 stack in CentOS 7 was mostly complete I had to adapt/tweak ~2 dozen packages until this point (details: bugzilla, click "Expand All").

Sometimes this means other packagers made mistakes previously which went unnoticed and I have to figure out a contingency strategy to fix these as well (to be clear: I'm also making mistakes, no blaming here!). It can involve writing lengthy emails with planning, prodding inactive maintainers, getting test suites to run in software I'm completely unfamiliar with and sometimes also asking certbot upstream to relax their dependency requirements. Also there are roughly monthly certbot updates to handle for Fedora 33, 34 and rawhide plus CentOS 8.

So this is a lot of work involved and none it is "sexy" or "fun" work. Still it is gratifying for me if I get such a complex transition to work but the time I can dedicate to that is limited.

If someone is really interested in this work and wants to help out (though it might not be the best start as a Fedora packager as this is not the easiest project) you welcome to check out:

  • github repo with work-in-progress patches for Python 3 support
  • COPR repo with work-in-progress builds

Still I'm sorry that I missed the importance of certbot 1.12+ for our CentOS certbot users. I'm sorry about that and feeling the need for sure motivates me to get recent certbot versions to work but I can't fix this quickly.


I pushed an update to certbot in EPEL 7 so you can also use --preferred-chain as in certbot 1.12+: Certbot update for CentOS 7: use --prefered-chain to select a shorter chain