Get Undefined symbol: __letf2 when running any certbot command

[DonB312]

Whenever I run any certbot command I receive the following error (see below).

I am running Ubuntu 18.04. I have tried completely uninstalling and reinstalling Certbot via snap. I have also tried reinstalling Python3 and the python packages mentioned in the error (python3_josepy, python3_openssl, python3_cryptography. My gcc version is 7.5.0.

I had no issues when I installed Certbot a few months ago so I am confused as to what caused this current issue. Any help or insight would be greatly appreciated.

$ sudo certbot certificates
Traceback (most recent call last):
File "/snap/certbot/2611/bin/certbot", line 5, in
from certbot.main import main
File "/snap/certbot/2611/lib/python3.8/site-packages/certbot/main.py", line 6, in
from certbot._internal import main as internal_main
File "/snap/certbot/2611/lib/python3.8/site-packages/certbot/_internal/main.py", line 19, in
import josepy as jose
File "/snap/certbot/2611/lib/python3.8/site-packages/josepy/init.py", line 40, in
from josepy.json_util import (
File "/snap/certbot/2611/lib/python3.8/site-packages/josepy/json_util.py", line 14, in
from OpenSSL import crypto
File "/snap/certbot/2611/lib/python3.8/site-packages/OpenSSL/init.py", line 8, in
from OpenSSL import SSL, crypto
File "/snap/certbot/2611/lib/python3.8/site-packages/OpenSSL/SSL.py", line 19, in
from OpenSSL.crypto import (
File "/snap/certbot/2611/lib/python3.8/site-packages/OpenSSL/crypto.py", line 21, in
from cryptography import utils, x509
File "/snap/certbot/2611/lib/python3.8/site-packages/cryptography/x509/init.py", line 6, in
from cryptography.x509 import certificate_transparency
File "/snap/certbot/2611/lib/python3.8/site-packages/cryptography/x509/certificate_transparency.py", line 10, in
from cryptography.hazmat.bindings._rust import x509 as rust_x509
ImportError: /snap/core20/current/lib/x86_64-linux-gnu/libgcc_s.so.1: undefined symbol: __letf2, version GCC_4.3.0

[rg305]

Hi @DonB312, and welcome to the LE community forum

Is snap up-to-date?

[DonB312]

I think snap is up to date but I am not really sure how to check. "snap list" shows the following:

$ sudo snap list
Name Version Rev Tracking Publisher Notes
certbot 2.1.1 2611 latest/beta certbot-eff✓ classic
certbot-dns-linode 2.1.1 2200 latest/beta certbot-eff✓ -
core 16-2.57.6 14399 latest/stable canonical✓ core
core20 20221212 1778 latest/stable canonical✓ base

[rg305]

This should tell us:
apt update
apt install snap

[DonB312]

Yep, that says "snap is already the newest version"

[DonB312]

Actually I may be having an issue with apt. sudo apt update gives the following error:

Hit:1 Index of /ubuntu bionic InRelease
Hit:2 Index of /ubuntu bionic-updates InRelease
Hit:3 Index of /ubuntu bionic-backports InRelease
Hit:4 Index of /232905/apt/ubuntu bionic InRelease
Hit:5 http://apt-longview.linode.com bionic InRelease
Hit:6 Index of /certbot/certbot/ubuntu bionic InRelease
Get:7 Index of /ubuntu bionic-security InRelease [88.7 kB]
Hit:8 Index of /ubuntu bionic InRelease
Hit:9 Index of /ondrej/php/ubuntu bionic InRelease
Fetched 88.7 kB in 1s (81.4 kB/s)
Traceback (most recent call last):
File "/usr/lib/cnf-update-db", line 8, in
from CommandNotFound.db.creator import DbCreator
File "/usr/lib/python3/dist-packages/CommandNotFound/db/creator.py", line 11, in
import apt_pkg
ModuleNotFoundError: No module named 'apt_pkg'
Reading package lists... Done
E: Problem executing scripts APT::Update::Post-Invoke-Success 'if /usr/bin/test -w /var/lib/command-not-found/ -a -e /usr/lib/cnf-update-db; then /usr/lib/cnf-update-db > /dev/null; fi'
E: Sub-process returned an error code

[_az]

What type of server is this?

sudo systemd-detect-virt
cat /proc/cpuinfo

[DonB312]

This is a virtual server running on Linode

s$ sudo systemd-detect-virt
kvm

cat /proc/cpuinfo returns information for 4 cpus. Here is one of them:
processor : 0
vendor_id : AuthenticAMD
cpu family : 23
model : 1
model name : AMD EPYC 7501 32-Core Processor
stepping : 2
microcode : 0x1000065
cpu MHz : 1999.999
cache size : 512 KB
physical id : 0
siblings : 1
core id : 0
cpu cores : 1
apicid : 0
initial apicid : 0
fpu : yes
fpu_exception : yes
cpuid level : 13
wp : yes
flags : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm rep_good nopl cpuid extd_apicid tsc_known_freq pni pclmulqdq ssse3 fma cx16 sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm cmp_legacy cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw perfctr_core ssbd ibpb vmmcall fsgsbase tsc_adjust bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves virt_ssbd arat
bugs : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed
bogomips : 4001.66
TLB size : 1024 4K pages
clflush size : 64
cache_alignment : 64
address sizes : 40 bits physical, 48 bits virtual
power management:

[DonB312]

I resolved the sudo apt update error.
Then I re-ran

apt update
apt install snap

and it still shows that snap is the latest version.

[_az]

Well, I tried installing Certbot on Linode on a similar, fully updated Ubuntu 18.04 server:

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 23
model           : 1
model name      : AMD EPYC 7601 32-Core Processor

and it works OK for me.

Unfortunately I'm not sure what's going on. I wonder if some core libraries on your server have been overrwritten. Or maybe you have LD_LIBRARY_PATH (or a similar mechanism) set and it's doing something weird.

One thing you could try is to reinstall these packages on your host system, but it's a long shot:

sudo apt install --reinstall libgcc1 libc6
sudo LD_LIBRARY_PATH= certbot certificates

If not, and unless you want to reinstall your Ubuntu server, you might need to uninstall the Certbot snap and install Certbot from the pip instructions. Or use another ACME client.

[DonB312]

Thank you so much for your efforts. I tried the two commands you suggested but it didn't resolve the issue. I suspect that there is some deeper issue with the server and a reinstall of Ubuntu may be the only fix.

At some point I will rebuild the problem server with the latest Ubuntu version and I am sure that will resolve the original issue. In the meantime Certbot works fine on my Ubuntu 20 server so I can use it to create the certificates and then manually copy them to the problem server.

Thanks again to all who have responded.

[DonB312]

Just a followup on the outcome of this situation.

I use the Linode backup service and I found that I had a backup from the day before this error started showing up. I restored my Linode from that backup and now everything is working fine. I did a certbot renew --dry-run and it was successful.

Thanks again to all who helped here! Your input led me to the conclusion that more drastic action was needed and that is what made me think of restoring from a backup.