I’ve been running certbot-auto for almost 2 years now daily through cron for renewals. It has never skipped a heartbeat not even once in all this time.
Lo and behold this morning, renewal emails from all 6 servers showed certbot-auto completely failed after doing it’s bootstrap dependencies.
Rather troubling to say the least, what has changed so drastically in the last 24 hours to cause such a catastrophic mess?
Every server gave me this output in my email log I email to myself.
Bootstrapping dependencies for Debian-based OSes... (you can skip this with --no-bootstrap)
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://ppa.launchpad.net/nginx/development/ubuntu bionic InRelease
Hit:3 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:4 http://dl.google.com/linux/chrome/deb stable Release
Hit:5 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:6 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:7 http://ppa.launchpad.net/ondrej/nginx/ubuntu bionic InRelease
Hit:8 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:9 http://espejito.fder.edu.uy/mariadb/repo/10.4/ubuntu xenial InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
augeas-lenses is already the newest version (1.10.1-2).
ca-certificates is already the newest version (20180409).
libaugeas0 is already the newest version (1.10.1-2).
libffi-dev is already the newest version (3.2.1-8).
python is already the newest version (2.7.15~rc1-1).
python-dev is already the newest version (2.7.15~rc1-1).
python-virtualenv is already the newest version (15.1.0+ds-1.1).
virtualenv is already the newest version (15.1.0+ds-1.1).
gcc is already the newest version (4:7.4.0-1ubuntu2.3).
libssl-dev is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
openssl is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Creating virtual environment... Bootstrapping dependencies for Debian-based OSes... (you can skip this with --no-bootstrap)
Ign:1 http://dl.google.com/linux/chrome/deb stable InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease
Hit:4 http://dl.google.com/linux/chrome/deb stable Release
Hit:5 http://security.ubuntu.com/ubuntu bionic-security InRelease
Hit:6 http://ppa.launchpad.net/nginx/development/ubuntu bionic InRelease
Hit:7 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease
Hit:8 http://ppa.launchpad.net/ondrej/nginx/ubuntu bionic InRelease
Hit:9 http://espejito.fder.edu.uy/mariadb/repo/10.4/ubuntu xenial InRelease
Reading package lists...
Reading package lists...
Building dependency tree...
Reading state information...
augeas-lenses is already the newest version (1.10.1-2).
ca-certificates is already the newest version (20180409).
libaugeas0 is already the newest version (1.10.1-2).
libffi-dev is already the newest version (3.2.1-8).
python is already the newest version (2.7.15~rc1-1).
python-dev is already the newest version (2.7.15~rc1-1).
python-virtualenv is already the newest version (15.1.0+ds-1.1).
virtualenv is already the newest version (15.1.0+ds-1.1).
gcc is already the newest version (4:7.4.0-1ubuntu2.3).
libssl-dev is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
openssl is already the newest version (1.1.1c-1+ubuntu18.04.1+deb.sury.org+1).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.
Creating virtual environment...
As you can see it got stuck at Creating virtual environment.
To fix this I had to SSH into every server this morning, move to my /opt/certbot directory and run any certbot-auto command to get it to fix itself
In my case I ran the following on every server
cd /opt/certbot
sudo ./certbot-auto plugins
Creating virtual environment...
Installing Python packages...
Installation succeeded.
Thereafter my usual cron script which does nothing more than run ./certbot-auto renew works again but I now have doubts about the stability of this.
Can anyone explain what has changed to cause this after almost 2 years trouble free?
This happened on following distro’s
Ubuntu 16.04.2
Linux smtp 4.4.0-161-generic #189-Ubuntu SMP Tue Aug 27 08:10:16 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux
and
Ubuntu 18.04.2
Linux ub2 4.15.0-60-generic #67-Ubuntu SMP Thu Aug 22 16:55:30 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux