ContextualVersionConflict: protobuf 3.20.1

[valkyr.dev]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

https://valkyr.dev

I ran this command:

sudo certbot renew

It produced this output:

An unexpected error occurred:
pkg_resources.ContextualVersionConflict: (protobuf 3.20.1 (/usr/lib/python3.10/site-packages),
Requirement.parse('protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4
.21.5,<5.0.0dev,>=3.19.5'), {'google-api-core'})

My web server is (include version):

nginx 1.23.3

The operating system my web server runs on is (include version):

Arch Linux (kernel: 6.1.14-hardened1-1-hardened)

My hosting provider, if applicable, is:

N/A

I can login to a root shell on my machine (yes or no, or I don't know):

Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

those commands don't work, but pacman shows that I have version 2.3.0-1 installed

I think it's worth noting that my installation has been working for years hands free. Recently I got an email that my certs are about to expire when I expected that they should have renewed automatically. It was when I attempted to do it manually that I discovered this problem. My issue is almost exactly the same as the one found here, but the "solution" listed doesn't work for me.

Additionally, running "pip list | grep proto" shows:

googleapis-common-protos 1.58.0
protobuf 4.21.12

[_az]

What Certbot plugins do you have installed?

[valkyr.dev]

certbot-dns-google and certbot-nginx

[_az]

I don't have any experience with Arch but things installed and worked cleanly for me with:

pacman -S certbot
pacman -S certbot-dns-google

Here is the output of my pip freeze:

# pip freeze
acme==2.3.0
attrs==22.2.0
autocommand==2.2.2
cachetools==5.3.0
certbot==2.3.0
certbot-dns-google==2.3.0
certifi==2022.12.7
cffi==1.15.1
chardet==5.1.0
ConfigArgParse==1.5.3
configobj==5.1.0.dev0
cryptography==39.0.2
Cython==0.29.33
distro==1.8.0
fastjsonschema==2.16.3
future==0.18.2
google-api-core==2.11.0
google-api-python-client==2.79.0
google-auth==2.16.0
google-auth-httplib2==0.1.0
googleapis-common-protos==1.58.0
httplib2==0.21.0
idna==3.4
inflect==6.0.2
jaraco.context==4.3.0
jaraco.functools==3.6.0
jaraco.text==3.11.1
josepy==1.13.0
more-itertools==9.0.0
oauth2client==4.1.3
ordered-set==4.1.0
packaging==23.0
parsedatetime==2.6
platformdirs==3.0.0
ply==3.11
protobuf==4.21.12
pyasn1==0.4.8
pyasn1-modules==0.2.8
pycparser==2.21
pydantic==1.10.5
pyOpenSSL==23.0.0
pyparsing==3.0.9
pyRFC3339==1.1
pytz==2022.7
requests==2.28.2
rsa==4.9
six==1.16.0
tomli==2.0.1
trove-classifiers==2023.3.1
typing_extensions==4.5.0
uritemplate==4.1.1
urllib3==1.26.12
validate==5.1.0.dev0
validate-pyproject==0.12.1

[valkyr.dev]

Unfortunately, I have already tried re-installing everything that seems related to the issue (certbot, certbot-dns-google, certbot-nginx, python-googleapis-common-protos, python-protobuf, python-google-api-core), rebooting, etc. and it's still giving me the same error. I have also pip uninstalled all the above packages and then re-installed them with pacman to ensure there weren't conflicts like that, though somehow

$ pacman -Q protobuf
protobuf 21.12-1

and

$ pip list | grep protobuf
protobuf 4.21.12

seem to conflict. Do you have any ideas for how to proceed troubleshooting from here?

[_az]

@valkyr.dev

$ pacman -Q protobuf
protobuf 21.12-1

and

$ pip list | grep protobuf
protobuf 4.21.12

seem to conflict

Those versions work fine together on my Arch instance.

The most suspicious thing for me is:

@valkyr.dev

pkg_resources.ContextualVersionConflict: (protobuf 3.20.1

I think this probably means that you have a Python package installed which wants this version of the Python protobuf module installed.

What does this show:

pip show protobuf | grep ^Required

[valkyr.dev]

$ pip show protobuf | grep ^Required
Required-by: google-api-core, googleapis-common-protos

is the response.

[_az]

Yeah, me too.

I'm not sure where Python is getting that 3.20.1 to be honest. It's going to be somewhere in your global Python installation, but if your google-api-core and googleapis-common-protos are up to date, I'm not sure how best to find it.

We do recommend using the Certbot snap or the pip-based virtual environment because they are resilient against weird stuff happening in the OS Python packages. I get that Arch users generally don't want to use snaps, but you can try the latter option, it should work out of the box for you as a replacement, without having to do anything.

[valkyr.dev]

I appreciate the help. I'll marinate on the issue for another night or two, and then if I still have no luck I'll probably go ahead with the pip-based virtual environment solution you suggested. I'll post an update however it turns out.

[Osiris]

If Arch is a little bit like Gentoo, it might be possible to install multiple versions of the same library.

Perhaps you could search your file system for the files libprotobuf.so.3.20* with e.g. find or mlocate (sudo updatedb and locate libprotobuf.so.3.20).

[valkyr.dev]

My solution was to uninstall everything using pacman, and then re-install it using the pip-based virtual environment installation instructions as @_az stated in this reply. Thanks for the suggestions everyone.