Did OpenSSL 3.0 break Certbot?

Client dev
1

Certbot 2.7.3
FreeBSD 13.2
OpenSSL 3.0.12
Python 3.9.18
py39-openssl 23.2.0

I've been using Certbot since 2016 when it was still called letsencrypt. It's been working perfectly for years. But then I broke everything.

I upgraded to OpenSSL 3 a couple of weeks ago, and ever since then Certbot hasn't worked. I can't even understand the error I'm getting, but I think it's something to do with FIPS support in OpenSSL. But I have no idea what FIPS is! The error ended with:

/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: Undefined symbol "EVP_default_properties_is_fips_enabled"

I tried recompiling OpenSSL without FIPS support but it didn't make any difference. There's no ability to compile Certbot without FIPS, and I couldn't find a reference to FIPS in the Certbot manpage. The error also references Python and Rust, but they have no FIPS options either.

I've recompiled half my server and I still can't get Certbot to run. Does anybody have any ideas or suggestions?

The full error is:

Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==2.7.3', 'console_scripts', 'certbot')())
  File "/usr/local/bin/certbot", line 25, in importlib_load_entry_point
    return next(matches).load()
  File "/usr/local/lib/python3.9/importlib/metadata.py", line 86, in load
    module = import_module(match.group('module'))
  File "/usr/local/lib/python3.9/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1030, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1007, in _find_and_load
  File "<frozen importlib._bootstrap>", line 986, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 680, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 850, in exec_module
  File "<frozen importlib._bootstrap>", line 228, in _call_with_frames_removed
  File "/usr/local/lib/python3.9/site-packages/certbot/main.py", line 6, in <module>
    from certbot._internal import main as internal_main
  File "/usr/local/lib/python3.9/site-packages/certbot/_internal/main.py", line 21, in <module>
    import josepy as jose
  File "/usr/local/lib/python3.9/site-packages/josepy/__init__.py", line 40, in <module>
    from josepy.json_util import (
  File "/usr/local/lib/python3.9/site-packages/josepy/json_util.py", line 14, in <module>
    from OpenSSL import crypto
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import SSL, crypto
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/SSL.py", line 9, in <module>
    from OpenSSL._util import (
  File "/usr/local/lib/python3.9/site-packages/OpenSSL/_util.py", line 6, in <module>
    from cryptography.hazmat.bindings.openssl.binding import Binding
  File "/usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/openssl/binding.py", line 15, in <module>
    from cryptography.exceptions import InternalError
  File "/usr/local/lib/python3.9/site-packages/cryptography/exceptions.py", line 9, in <module>
    from cryptography.hazmat.bindings._rust import exceptions as rust_exceptions
ImportError: /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so: Undefined symbol "EVP_default_properties_is_fips_enabled"
2

My first guess would be that the python cryptography package you have installed is too old to support OpenSSL 3. I would look into updating that first.

4 Likes
3

I forgot to include Python versions, so I updated my initial post.

I'm using Python 3.9.18 and py39-openssl 23.2.0. I just recompiled both and I'm getting the same error.

Are there any other specific packages I should recompile? Everything is up to date, but sometimes I need to recompile . Python 3.9 is new enough, right? It's the default in FreeBSD, but I can update to a newer version (but then I'd have to recompile everything that relies on Python!)

4

Cryptography uses Rust to compile this library. It's not just Python code. It's probably linked to OpenSSL on compile time (i.e.: installation time of cryptography).

Usually, distribution package managers would urge or force you to reinstall (i.e.: recompile parts of) cryptography when OpenSSL is changed significantly, such as when upgrading from 1.1.1 to 3.0.0.

You probably can get cryptography working again (and with that Certbot) by reinstalling it. It should take some time while compiling the Rust bindings and with that linking to the new OpenSSL library.

Looking at the /usr/local/lib/ prefix you might have used pip to install the packages globally on your system, which is often not recommended. pip doesn't have a very strong package management so to say. A venv is usually recommended.

2 Likes
5

Sorry for the delay in replying, I was recompiling at lot.

I recompiled py39-cryptography, along with several other python ports, nettle, libgcrypt, and a few other crypto related ports (just in case). Still no luck.

It's up to me to upgrade properly, but I think I missed a step. After upgrading to OpenSSL 3, I had to recompile several packages to get Nextcloud up and running again, but I simply forgot about Certbot until I saw the error a couple of weeks later.

Nah, on FreeBSD, all user installed ports and packages go into /usr/local. Only the operating system stuff goes into /bin or /lib or /etc. I've never used pip. The BSDs aren't distros, they're complete OSes.

I might do some more reading, see what other dependencies I missed. How frustrating! I hate it when I absent-mindedly break my stuff!

3 Likes
6

It seems like there should be a way to run certbot via docker so that incidental dependencies are irrelevant, perhaps by mapping the required volumes to a docker container instance? Juggling dependencies on your actual operating system seems difficult when it's a balance between certbot vs. your actual applications.

1 Like
7

What's the output of:

ldd /usr/local/lib/python3.9/site-packages/cryptography/hazmat/bindings/_rust.abi3.so

By the way, the abi3 might indicate it was already compiled for OpenSSL 3.x.x :roll_eyes:

You're not the first one by the way, see 273770 – security/py-cryptography: Run-time error with OpenSSL 3.0 for more information about this issue. The error is the same as another user with FreeBSD 13.2. From what I can gather there, it is a linking issue with FreeBSD. Even with abi3 it's linking to incorrect files it seems.

Also, in the end it's not really an issue with Certbot itself, but with the underlying library used. Which is a library very commonly used and a very important one at that. I'm afraid the solution needs to come from the FreeBSD maintainers.

3 Likes
8

While that would work too, I would expect that running certbot via pip would be good enough for most cases. Maybe not this case, though.

4 Likes
9

Yeah sure it would, it's just someone else was complaining about certbot dependencies on another thread and it just got me thinking that kind of thing is already (more or less) a solved problem.

1 Like
10

Yes, but the feature set and use cases are so different it's not really advisable as a drop-in replacement for the certbot executable.

5 Likes
11

If not required, Docker would only introduce unnecessary complexity IMO.

3 Likes
12

Sure, but there's a point where upgrading all the dependencies on a production server just to get certbot to run is perhaps not always the right solution. Maybe at that point it's best to get your certs on another machine and just deploy them to the target server.

1 Like
13

Totally agreed!
[especially now with virtualization]
It has never been cheaper/simpler to spin up a "dedicated/single task system".

3 Likes
14

I have the same problem on many servers. All packages are regularly updated.

15

Other Python packages could/are also (be) affected, as cryptography is a common dependency.

2 Likes
16

Hi @dlsumy, and welcome to the LE community forum :slight_smile:

It would be best to open a separate topic to address your specific problem.
[even if it is identical to this one]

3 Likes
17

And even if it would be the exact same issue (which it commonly isn't), then in this specific case it's up to the cryptography package manager(s) of FreeBSD to fix it. Nothing we or Certbot can do about it unfortunately.

2 Likes
18

Absolutly identifical

19

@dlsumy, so you got this same error?:

2 Likes
20

Yes. In FreeBSD 13.2.
In FreeBSD 12.4 certbot works fine.
The application software versions on both servers are the same and were updated from ports 2 hours ago.