Certbot suddenly failing

watusimoto · January 2, 2021, 11:20pm

I got an email today that my certificate was about to expire, which is surprising because I have certbot running in a cronjob on my server. Sure enough, the process is kicking back errors. Config file is at the bottom.

Running as root (yes, I know!):

# /root/bin/certbot-auto renew --no-self-upgrade --no-bootstrap
Your system is not supported by certbot-auto anymore.
Certbot will no longer receive updates.
Please visit https://certbot.eff.org/ to check for other alternatives.
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/sensorbot.org.conf
-------------------------------------------------------------------------------
Attempting to parse the version 0.31.0 renewal configuration file found at /etc/letsencrypt/renewal/sensorbot.org.conf with version 0.21.1 of Certbot. This might not work.
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator apache, Installer apache
Renewing an existing certificate
Attempting to renew cert (sensorbot.org) from /etc/letsencrypt/renewal/sensorbot.org.conf produced an unexpected error: 'Directory field not found': new_authz. Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/sensorbot.org/fullchain.pem (failure)

-------------------------------------------------------------------------------

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/sensorbot.org/fullchain.pem (failure)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)
#

I am on a VPS with Ubuntu 16.04.7 LTS. I don't have a lot of control over which distro I'm using in this context.

Nothing has changed in the server config for a very long time, so I'm not sure why the process started failing.

My config is the same as on another machine that is working.

Any advice would be welcome.

Thanks!

=====

# cat /etc/letsencrypt/renewal/sensorbot.org.conf
# renew_before_expiry = 30 days
version = 0.31.0
archive_dir = /etc/letsencrypt/archive/sensorbot.org
cert = /etc/letsencrypt/live/sensorbot.org/cert.pem
privkey = /etc/letsencrypt/live/sensorbot.org/privkey.pem
chain = /etc/letsencrypt/live/sensorbot.org/chain.pem
fullchain = /etc/letsencrypt/live/sensorbot.org/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = XXXXX[redacted]
authenticator = apache
installer = apache
server = https://acme-v02.api.letsencrypt.org/directory
#

Litbelb · January 2, 2021, 11:27pm

Have you updated your VPS using
sudo apt update && sudo apt upgrade -y

rg305 · January 2, 2021, 11:31pm

Please show:
certbot certificates
ls -lR /etc/letsencrypt/archive/

rg305 · January 2, 2021, 11:38pm

That might do more than needed (or expected).
Try just:
sudo apt update
sudo apt install ca-certificates
sudo apt install certbot

watusimoto · January 3, 2021, 12:12am

I've tried the update/upgrade.

Interestingly, certbot is throwing up a Python error, stacktrace as follows. Before posting here (and before noticing the warning that my OS was no longer supported for upgrades), I tried to force certbot to upgrade. I may have broken it, or it may have already been broken. Output from the ls follows that.

# certbot certificates
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 561, in load_entry_point
    return get_distribution(dist).load_entry_point(group, name)
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2631, in load_entry_point
    return ep.load()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2291, in load
    return self.resolve()
  File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line 2297, in resolve
    module = __import__(self.module_name, fromlist=['__name__'], level=0)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 10, in <module>
    import josepy as jose
  File "/usr/lib/python3/dist-packages/josepy/__init__.py", line 44, in <module>
    from josepy.interfaces import JSONDeSerializable
  File "/usr/lib/python3/dist-packages/josepy/interfaces.py", line 8, in <module>
    from josepy import errors, util
  File "/usr/lib/python3/dist-packages/josepy/util.py", line 4, in <module>
    import OpenSSL
  File "/usr/lib/python3/dist-packages/OpenSSL/__init__.py", line 8, in <module>
    from OpenSSL import crypto, SSL
  File "/usr/lib/python3/dist-packages/OpenSSL/crypto.py", line 12, in <module>
    from cryptography import x509
  File "/usr/lib/python3/dist-packages/cryptography/x509/__init__.py", line 9, in <module>
    from cryptography.x509.base import (
  File "/usr/lib/python3/dist-packages/cryptography/x509/base.py", line 16, in <module>
    from cryptography.x509.extensions import Extension, ExtensionType
  File "/usr/lib/python3/dist-packages/cryptography/x509/extensions.py", line 18, in <module>
    from cryptography.hazmat.primitives import constant_time, serialization
  File "/usr/lib/python3/dist-packages/cryptography/hazmat/primitives/constant_time.py", line 9, in <module>
    from cryptography.hazmat.bindings._constant_time import lib
ModuleNotFoundError: No module named '_cffi_backend'
#

# ls -lR /etc/letsencrypt/archive/
/etc/letsencrypt/archive/:
total 4
drwxr-xr-x 2 root root 4096 Oct 24 08:39 sensorbot.org

/etc/letsencrypt/archive/sensorbot.org:
total 272
-rw-r--r-- 1 root root 1903 Aug 28  2019 cert10.pem
-rw-r--r-- 1 root root 1907 Oct 28  2019 cert11.pem
-rw-r--r-- 1 root root 1907 Dec 27  2019 cert12.pem
-rw-r--r-- 1 root root 1907 Feb 25  2020 cert13.pem
-rw-r--r-- 1 root root 1903 Apr 26  2020 cert14.pem
-rw-r--r-- 1 root root 1903 Jun 25  2020 cert15.pem
-rw-r--r-- 1 root root 1903 Aug 24 16:40 cert16.pem
-rw-r--r-- 1 root root 1903 Oct 24 08:39 cert17.pem
-rw-r--r-- 1 root root 1789 Mar  2  2018 cert1.pem
-rw-r--r-- 1 root root 2147 May  1  2018 cert2.pem
-rw-r--r-- 1 root root 2147 Jun 30  2018 cert3.pem
-rw-r--r-- 1 root root 2151 Aug 29  2018 cert4.pem
-rw-r--r-- 1 root root 2147 Oct 28  2018 cert5.pem
-rw-r--r-- 1 root root 1903 Jan  2  2019 cert6.pem
-rw-r--r-- 1 root root 1907 Mar  1  2019 cert7.pem
-rw-r--r-- 1 root root 1907 Apr 30  2019 cert8.pem
-rw-r--r-- 1 root root 1903 Jun 29  2019 cert9.pem
-rw-r--r-- 1 root root 1647 Aug 28  2019 chain10.pem
-rw-r--r-- 1 root root 1647 Oct 28  2019 chain11.pem
-rw-r--r-- 1 root root 1647 Dec 27  2019 chain12.pem
-rw-r--r-- 1 root root 1647 Feb 25  2020 chain13.pem
-rw-r--r-- 1 root root 1647 Apr 26  2020 chain14.pem
-rw-r--r-- 1 root root 1647 Jun 25  2020 chain15.pem
-rw-r--r-- 1 root root 1647 Aug 24 16:40 chain16.pem
-rw-r--r-- 1 root root 1647 Oct 24 08:39 chain17.pem
-rw-r--r-- 1 root root 1647 Mar  2  2018 chain1.pem
-rw-r--r-- 1 root root 1647 May  1  2018 chain2.pem
-rw-r--r-- 1 root root 1647 Jun 30  2018 chain3.pem
-rw-r--r-- 1 root root 1647 Aug 29  2018 chain4.pem
-rw-r--r-- 1 root root 1647 Oct 28  2018 chain5.pem
-rw-r--r-- 1 root root 1647 Jan  2  2019 chain6.pem
-rw-r--r-- 1 root root 1647 Mar  1  2019 chain7.pem
-rw-r--r-- 1 root root 1647 Apr 30  2019 chain8.pem
-rw-r--r-- 1 root root 1647 Jun 29  2019 chain9.pem
-rw-r--r-- 1 root root 3550 Aug 28  2019 fullchain10.pem
-rw-r--r-- 1 root root 3554 Oct 28  2019 fullchain11.pem
-rw-r--r-- 1 root root 3554 Dec 27  2019 fullchain12.pem
-rw-r--r-- 1 root root 3554 Feb 25  2020 fullchain13.pem
-rw-r--r-- 1 root root 3550 Apr 26  2020 fullchain14.pem
-rw-r--r-- 1 root root 3550 Jun 25  2020 fullchain15.pem
-rw-r--r-- 1 root root 3550 Aug 24 16:40 fullchain16.pem
-rw-r--r-- 1 root root 3550 Oct 24 08:39 fullchain17.pem
-rw-r--r-- 1 root root 3436 Mar  2  2018 fullchain1.pem
-rw-r--r-- 1 root root 3794 May  1  2018 fullchain2.pem
-rw-r--r-- 1 root root 3794 Jun 30  2018 fullchain3.pem
-rw-r--r-- 1 root root 3798 Aug 29  2018 fullchain4.pem
-rw-r--r-- 1 root root 3794 Oct 28  2018 fullchain5.pem
-rw-r--r-- 1 root root 3550 Jan  2  2019 fullchain6.pem
-rw-r--r-- 1 root root 3554 Mar  1  2019 fullchain7.pem
-rw-r--r-- 1 root root 3554 Apr 30  2019 fullchain8.pem
-rw-r--r-- 1 root root 3550 Jun 29  2019 fullchain9.pem
-rw-r--r-- 1 root root 1704 Aug 28  2019 privkey10.pem
-rw-r--r-- 1 root root 1704 Oct 28  2019 privkey11.pem
-rw-r--r-- 1 root root 1704 Dec 27  2019 privkey12.pem
-rw-r--r-- 1 root root 1704 Feb 25  2020 privkey13.pem
-rw-r--r-- 1 root root 1704 Apr 26  2020 privkey14.pem
-rw-r--r-- 1 root root 1704 Jun 25  2020 privkey15.pem
-rw-r--r-- 1 root root 1704 Aug 24 16:40 privkey16.pem
-rw-r--r-- 1 root root 1708 Oct 24 08:39 privkey17.pem
-rw-r--r-- 1 root root 1704 Mar  2  2018 privkey1.pem
-rw-r--r-- 1 root root 1704 May  1  2018 privkey2.pem
-rw-r--r-- 1 root root 1704 Jun 30  2018 privkey3.pem
-rw-r--r-- 1 root root 1704 Aug 29  2018 privkey4.pem
-rw-r--r-- 1 root root 1704 Oct 28  2018 privkey5.pem
-rw-r--r-- 1 root root 1708 Jan  2  2019 privkey6.pem
-rw-r--r-- 1 root root 1704 Mar  1  2019 privkey7.pem
-rw-r--r-- 1 root root 1704 Apr 30  2019 privkey8.pem
-rw-r--r-- 1 root root 1708 Jun 29  2019 privkey9.pem
#

watusimoto · January 3, 2021, 12:15am

I also tried removing and reinstalling certbot using apt, and still get the same stacktrace.

Litbelb · January 3, 2021, 12:44am

Are you using certbot snap or apt?

watusimoto · January 3, 2021, 12:47am

I'm using apt; I used the established install process to set things up, and haven't tinkered with them since (until today).

Litbelb · January 3, 2021, 12:48am

Another question that might seem odd: do you have python 3 installed?

watusimoto · January 3, 2021, 12:59am

I do, but "python" gives me python 2.7. My python3 does appear to be a frail and decrepit version (3.10) but it's there.

rg305 · January 3, 2021, 1:52am

Yeah, the upgrade broke it.
It's probably for the best.
I would remove it from apt and install it from snap.

watusimoto · January 3, 2021, 1:54am

Can I use snap on a headless machine?

rg305 · January 3, 2021, 1:54am

Yes.

watusimoto · January 3, 2021, 1:55am

Actually, if it is possible to install from apt, I'd prefer that, as I'll never remember that when this particular item needs attention, it is the one thing that doesn't use apt.

Is there a reason that wouldn't work?

rg305 · January 3, 2021, 1:58am

Certbot dependencies are borked.
I don't know if you use python, or not, elsewhere.
The "fix" would require removing things that may affect other things.
Besides... the APT version is no longer being updated.
You are better off with the SNAP version.

rg305 · January 3, 2021, 1:59am

Ubuntu 16 comes with SNAP already installed - LOL

watusimoto · January 3, 2021, 2:06am

Well, that's a bummer. Snapd is not installed on my system (or it wasn't... it is now). But...

# sudo snap install core; sudo snap refresh core
error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
       /tmp/sanity-mountpoint-824181801: mount failed: Unknown error -1
error: system does not fully support snapd: cannot mount squashfs image using "squashfs": mount:
       /tmp/sanity-mountpoint-824181801: mount failed: Unknown error -1
#

So it looks like this might not be an option.

rg305 · January 3, 2021, 2:17am

Please show output of:
sudo apt-get update
sudo apt-cache policy snapd

Mine shows:

snapd:
  Installed: 2.48
  Candidate: 2.48
  Version table:
 *** 2.48 500
        500 http://us.archive.ubuntu.com/ubuntu xenial-updates/main amd64 Packages
        100 /var/lib/dpkg/status
     2.45.1ubuntu0.2 500
        500 http://security.ubuntu.com/ubuntu xenial-security/main amd64 Packages
     2.0.2 500
        500 http://us.archive.ubuntu.com/ubuntu xenial/main amd64 Packages

Litbelb · January 3, 2021, 2:23am

After looking at certbot’s GitHub repository, I’m a bit confused about that error. It seems that it’s an error with python, not certbot itself. I would, like @rg305 said, install the snaps version of certbot.

rg305 · January 3, 2021, 2:26am

Two more questions/requests:

Is your system virtualized?
Please show the output of:
uname -r
Mine shows: 4.4.0-197-generic

Topic		Replies	Views
Certbot's renewal error Issuance Tech	11	4090	January 5, 2017
Cronjob to renew cert via certbot-auto is failing Server	3	1795	April 2, 2017
Auto renew fails for Apache + Ubuntu + certbot 0.31.0 Help	6	1911	November 30, 2021
Auto-renew via Certbot not working - Receives HTTP 200 Help	15	1040	September 9, 2023
Cert Renewal Fails Help	6	2865	April 18, 2019

Certbot suddenly failing

Related topics