Cryptography 39 breaks certbot

[Schamschula]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

I'm the port maintainer of py-acme and certbot* for MacPorts.
See original report: #66939 (certbot @2.3.0_0+python310: Runtime error after update of py310-cryptography @39.0.1) – MacPorts

My domain is:
not relevant (various)

I ran this command:
sudo certbot

It produced this output:
Traceback (most recent call last):
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 617, in _build_master
ws.require(requires)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 956, in require
needed = self.resolve(parse_requirements(requirements))
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 815, in resolve
dist = self._resolve_dist(
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 861, in _resolve_dist
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (cryptography 39.0.1 (/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'PyOpenSSL'})

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/opt/local/bin/certbot", line 33, in
sys.exit(load_entry_point('certbot==2.3.0', 'console_scripts', 'certbot')())
File "/opt/local/bin/certbot", line 25, in importlib_load_entry_point
return next(matches).load()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/metadata/init.py", line 171, in load
module = import_module(match.group('module'))
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/importlib/init.py", line 126, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1050, in _gcd_import
File "", line 1027, in _find_and_load
File "", line 1006, in _find_and_load_unlocked
File "", line 688, in _load_unlocked
File "", line 883, in exec_module
File "", line 241, in _call_with_frames_removed
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certbot/main.py", line 6, in
from certbot._internal import main as internal_main
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certbot/_internal/main.py", line 27, in
from certbot import configuration
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certbot/configuration.py", line 11, in
from certbot import util
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certbot/util.py", line 26, in
from certbot._internal import constants
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/certbot/_internal/constants.py", line 6, in
import pkg_resources
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 3249, in
def _initialize_master_working_set():
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 3223, in _call_aside
f(*args, **kwargs)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 3261, in _initialize_master_working_set
working_set = WorkingSet._build_master()
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 619, in _build_master
return cls._build_from_requirements(requires)
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 632, in _build_from_requirements
dists = ws.resolve(reqs, Environment())
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 815, in resolve
dist = self._resolve_dist(
File "/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/pkg_resources/init.py", line 861, in _resolve_dist
raise VersionConflict(dist, req).with_context(dependent_req)
pkg_resources.ContextualVersionConflict: (cryptography 39.0.1 (/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'PyOpenSSL'})

My web server is (include version):
not relevant (apache2/nginx)

The operating system my web server runs on is (include version):
macOS 12.6.3 on build machine

My hosting provider, if applicable, is:
N/A

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.3.0

[Schamschula]

P.S.: I also rebuilt py310-acme.

[Osiris]

@Schamschula

pkg_resources.ContextualVersionConflict: (cryptography 39.0.1 (/opt/local/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages), Requirement.parse('cryptography<39,>=38.0.0'), {'PyOpenSSL'})

Are you sure it's Certbot breaking things? It might be that Certbot is broken, but looking at the error message I've quoted above, it's actually PyOpenSSL requiring cryptography 38.x.x. Looking at 23.0.0 version bump by alex · Pull Request #1175 · pyca/pyopenssl · GitHub it seems PyOpenSSL 23.0.0 has cryptography 39.x.x support.

Certbot 2.3.0 itself doesn't really care if it's cryptography 38 or 39, see https://github.com/certbot/certbot/blob/v2.3.0/certbot/setup.py#L53.

Also, regarding the post on the MacPorts bugtracker:

2Changed 4 hours ago by stromnov (Andrey Stromnov)

It looks like a dependency inconsistency in the certbot source tree:

​https://github.com/certbot/certbot/blob/v2.3.0/tools/requirements.txt#L35

That requirements.txt also tells pip to depend on PyOpenSSL 23.0.0, which would have the correct cryptography dependency as stated above. So we can conclude the contents of requirements.txt are not the issue in your current problem.

Does MacPorts even use version based dependencies?! Looking at https://github.com/macports/macports-ports/blob/master/python/py-openssl/Portfile it just states a version of the application and a list of packages without any version number as dependencies.. Which is kinda weird if you ask me..

My conclusions are:

• MacPorts seems to be lacking in the dependency calculation department;
• The error is due to an incompatibility between py-cryptography-39 (current version in MacPorts) and py-openssl-22.1.0 (current version in MacPorts);
• Certbot is not responsible for this issue.

[Schamschula]

Nothing weird here!

py-cryptography was updated to version 39.0.1 last night. However, the maintainer of py-openssl wasn't aware of the update to fix the resulting breakage (two different maintainers). It just happened that a user of certbot first ran into this issue. With over 1150 ports in the python category this sort of thing is bound to happen from time to time.

Thank you for pointing out the cause.