Certbot-auto (renew) failed on 6 servers (Creating virtual environment)

I’ve been running certbot-auto for almost 2 years now daily through cron for renewals. It has never skipped a heartbeat not even once in all this time.

Lo and behold this morning, renewal emails from all 6 servers showed certbot-auto completely failed after doing it’s bootstrap dependencies.

Rather troubling to say the least, what has changed so drastically in the last 24 hours to cause such a catastrophic mess?

Every server gave me this output in my email log I email to myself.

Bootstrapping dependencies for Debian-based OSes... (you can skip this with --no-bootstrap)
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://ppa.launchpad.net/nginx/development/ubuntu bionic InRelease
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:4 http://dl.google.com/linux/chrome/deb stable Release
Hit:5 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:6 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:7 http://ppa.launchpad.net/ondrej/nginx/ubuntu bionic InRelease
Hit:8 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:9 http://espejito.fder.edu.uy/mariadb/repo/10.4/ubuntu xenial InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
augeas-lenses is already the newest version (1.10.1-2).
ca-certificates is already the newest version (20180409).
libaugeas0 is already the newest version (1.10.1-2).
libffi-dev is already the newest version (3.2.1-8).
python is already the newest version (2.7.15~rc1-1).
python-dev is already the newest version (2.7.15~rc1-1).
python-virtualenv is already the newest version (15.1.0+ds-1.1).
virtualenv is already the newest version (15.1.0+ds-1.1).
gcc is already the newest version (4:7.4.0-1ubuntu2.3).
libssl-dev is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
openssl is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Creating virtual environment... Bootstrapping dependencies for Debian-based OSes... (you can skip this with --no-bootstrap)
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://dl.google.com/linux/chrome/deb stable Release
Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:6 http://ppa.launchpad.net/nginx/development/ubuntu bionic InRelease
Hit:7 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:8 http://ppa.launchpad.net/ondrej/nginx/ubuntu bionic InRelease
Hit:9 http://espejito.fder.edu.uy/mariadb/repo/10.4/ubuntu xenial InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
augeas-lenses is already the newest version (1.10.1-2).
ca-certificates is already the newest version (20180409).
libaugeas0 is already the newest version (1.10.1-2).
libffi-dev is already the newest version (3.2.1-8).
python is already the newest version (2.7.15~rc1-1).
python-dev is already the newest version (2.7.15~rc1-1).
python-virtualenv is already the newest version (15.1.0+ds-1.1).
virtualenv is already the newest version (15.1.0+ds-1.1).
gcc is already the newest version (4:7.4.0-1ubuntu2.3).
libssl-dev is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
openssl is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Creating virtual environment...

As you can see it got stuck at Creating virtual environment.

To fix this I had to SSH into every server this morning, move to my /opt/certbot directory and run any certbot-auto command to get it to fix itself

In my case I ran the following on every server

cd /opt/certbot
sudo ./certbot-auto plugins
Creating virtual environment...
Installing Python packages...
Installation succeeded.

Thereafter my usual cron script which does nothing more than run ./certbot-auto renew works again but I now have doubts about the stability of this.

Can anyone explain what has changed to cause this after almost 2 years trouble free?

This happened on following distro’s

Ubuntu 16.04.2
Linux smtp 4.4.0-161-generic #189-Ubuntu SMP Tue Aug 27 08:10:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

and

Ubuntu 18.04.2
Linux ub2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Was the process still running when you logged into the server, or was it gone?

If it was gone, I would be checking dmesg log to see whether the kernel killed it for some reason.

If it was still running, it would be (have been) helpful to attach strace to it to see what it is doing.

1 Like

To be honest I never checked at all, rather spent my time fixing the issue to avoid problems tomorrow. I never get any hung processes of any sort on my boxes so I do not bother looking for any rather just solving the issue at hand. Could any of these recent Ubuntu updates have caused this??

Start-Date: 2019-09-04  06:27:50
Commandline: /usr/bin/unattended-upgrade -d
Upgrade: linux-generic-lts-utopic:amd64 (4.4.0.159.167, 4.4.0.161.169)
End-Date: 2019-09-04  06:27:50

Start-Date: 2019-09-04  06:27:53
Commandline: /usr/bin/unattended-upgrade -d
Upgrade: linux-libc-dev:amd64 (4.4.0-159.187, 4.4.0-161.189)
End-Date: 2019-09-04  06:27:53

Start-Date: 2019-09-04  06:27:56
Commandline: /usr/bin/unattended-upgrade -d
Upgrade: linux-headers-generic-lts-utopic:amd64 (4.4.0.159.167, 4.4.0.161.169)
End-Date: 2019-09-04  06:27:56

Start-Date: 2019-09-04  06:27:59
Commandline: /usr/bin/unattended-upgrade -d
Install: linux-headers-4.4.0-161-generic:amd64 (4.4.0-161.189, automatic), linux-headers-4.4.0-161:amd64 (4.4.0-161.189, automatic), linux-image-4.4.0-161-generic:amd64 (4.4.0-161.189, automatic), linux-modules-extra-4.4.0-161-generic:amd64 (4.4.0-161.189, automatic), linux-modules-4.4.0-161-generic:amd64 (4.4.0-161.189, automatic)
Upgrade: linux-headers-generic:amd64 (4.4.0.159.167, 4.4.0.161.169), linux-image-generic:amd64 (4.4.0.159.167, 4.4.0.161.169), linux-generic:amd64 (4.4.0.159.167, 4.4.0.161.169)
End-Date: 2019-09-04  06:28:46

Start-Date: 2019-09-05  06:59:49
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-headers-4.4.0-154-generic:amd64 (4.4.0-154.181)
End-Date: 2019-09-05  07:00:08

Start-Date: 2019-09-05  07:00:12
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-modules-extra-4.4.0-154-generic:amd64 (4.4.0-154.181), linux-modules-4.4.0-154-generic:amd64 (4.4.0-154.181), linux-image-4.4.0-154-generic:amd64 (4.4.0-154.181)
End-Date: 2019-09-05  07:00:35

Start-Date: 2019-09-05  07:00:38
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-headers-4.4.0-154:amd64 (4.4.0-154.181)
End-Date: 2019-09-05  07:00:55

Start-Date: 2019-09-05  07:00:57
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-image-4.4.0-157-generic:amd64 (4.4.0-157.185), linux-modules-extra-4.4.0-157-generic:amd64 (4.4.0-157.185)
End-Date: 2019-09-05  07:01:18

Start-Date: 2019-09-05  07:01:21
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-headers-4.4.0-157-generic:amd64 (4.4.0-157.185)
End-Date: 2019-09-05  07:01:31

Start-Date: 2019-09-05  07:01:34
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-headers-4.4.0-157:amd64 (4.4.0-157.185)
End-Date: 2019-09-05  07:01:47

Start-Date: 2019-09-05  07:01:50
Commandline: /usr/bin/unattended-upgrade -d
Remove: linux-modules-4.4.0-157-generic:amd64 (4.4.0-157.185)
End-Date: 2019-09-05  07:01:51

Start-Date: 2019-09-05  15:50:28
Commandline: apt-get -y install bind9 bind9-doc bind9-host bind9utils bsdutils dnsutils google-chrome-stable libbind9-140 libblkid1 libdns-export162 libdns162 libfdisk1 libirs141 libisc-export160 libisc160 libisccc140 libisccfg140 libldap-2.4-2 liblwres141 libmount1 libsmartcols1 libuuid1 mount psmisc util-linux uuid-runtime
Upgrade: libdns-export162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), libisccfg140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), uuid-runtime:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), libfdisk1:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), libldap-2.4-2:amd64 (2.4.42+dfsg-2ubuntu3.6, 2.4.42+dfsg-2ubuntu3.7), libmount1:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), libirs141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), bind9-host:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), dnsutils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), google-chrome-stable:amd64 (76.0.3809.100-1, 76.0.3809.132-1), util-linux:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), libisc160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), bind9utils:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), mount:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), libblkid1:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), libisc-export160:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), psmisc:amd64 (22.21-2.1build1, 22.21-2.1ubuntu0.1), libuuid1:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), liblwres141:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), libsmartcols1:amd64 (2.27.1-6ubuntu3.7, 2.27.1-6ubuntu3.8), bsdutils:amd64 (1:2.27.1-6ubuntu3.7, 1:2.27.1-6ubuntu3.8), bind9:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), libdns162:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), libisccc140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), libbind9-140:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15), bind9-doc:amd64 (1:9.10.3.dfsg.P4-8ubuntu1.14, 1:9.10.3.dfsg.P4-8ubuntu1.15)
End-Date: 2019-09-05  15:51:03

I created a much better and more solid solution for taking care of Renewals using a mini(Conda) environment - Guide here for anyone else

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.