Certbot for Apache on Debian 8 Jessie Stopped Working

Server
1

I’ve been using Certbot on my Raspberry Pi running emonCMS for over a year without issue. A few days ago I noticed my certificates had failed to renew (usually done with a cron job). I logged into my server and noticed that I can’t run any certbot commands at all. Here’s a typical example:

pi@emonpi(rw):~$ certbot --help
Traceback (most recent call last):
  File "/usr/bin/certbot", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 48, in <module>
    from pkg_resources.extern import six
ImportError: No module named extern

Here’s another:

pi@emonpi(rw):~$ sudo certbot renew --authenticator standalone --pre-hook "apachectl -k stop" --post-hook "apachectl -k start"
Traceback (most recent call last):
  File "/usr/bin/certbot", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 48, in <module>
    from pkg_resources.extern import six
ImportError: No module named extern

I tried following the latest instructions for Apache on Jessie, which suggest switching to certbot-auto. Unfortunately I got stopped at this step :
sudo /path/to/certbot-auto --apache

… with this final error message:

Certbot has problem setting up the virtual environment.

We were not be able to guess the right solution from your pip
output.

Consult https://certbot.eff.org/docs/install.html#problems-with-python-virtual-environment
for possible solutions.
You may also find some support resources at https://certbot.eff.org/support/ .

So I reinstalled Certbot the way I had previously done:

sudo apt-get install python-certbot-apache -t jessie-backports

But I get the same result I started with – the seeming dependency errors.

pi@emonpi(rw):~$ certbot --help
Traceback (most recent call last):
  File "/usr/bin/certbot", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 48, in <module>
    from pkg_resources.extern import six
ImportError: No module named extern
2

If /usr/lib/python2.7/dist-packages/pkg_resources/__init__.py can't find /usr/lib/python2.7/dist-packages/pkg_resources/extern/__init__.py, that doesn't sound good.

Maybe pkg_resources got partially downgraded with pip or something?

Are you sure your Pi hasn't lost files due to disk corruption?

What does "ls -lA /usr/lib/python2.7/dist-packages/pkg_resources/" look like?

You could use "dpkg -S /usr/lib/python2.7/dist-packages/pkg_resources/__init__.py" to find out what package pkg_resources belongs too and try to reinstall it, e.g. "sudo apt-get install --reinstall python-pkg-resources".

Edit: Just for fun, can you paste the contents of /usr/bin/certbot?

2 Likes
3

I don't think so. I ran history | grep pip just to check and see if I had been doing anything with it. I haven't.

I suppose that is possible, but everything else is functioning normally. An emonPi has a pretty good diagnostic admin page, and everything seems to be in order there. Also, my emonPi runs in a low-write mode. The idea was to make SD corruption less likely. So it is certainly possible, but ...

pi@emonpi(rw):~$ ls -lA /usr/lib/python2.7/dist-packages/pkg_resources/
total 228
drwxr-xr-x 2 root root   4096 Jul 12  2017 extern
-rw-r--r-- 1 root root 103308 Jan 16  2017 __init__.py
-rw-r--r-- 1 root root 115568 Jul 12  2017 __init__.pyc
drwxr-xr-x 3 root root   4096 Jul 12  2017 _vendor

pi@emonpi(rw):~$ cat /usr/bin/certbot
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'certbot==0.10.2','console_scripts','certbot'
__requires__ = 'certbot==0.10.2'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
    sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
    sys.exit(
        load_entry_point('certbot==0.10.2', 'console_scripts', 'certbot')()
    )

Here is the result of the dpkg command. I could use some help with what to do next. I'm not clear on the apt-get -reinstall command, in terms of what that last argument should be.

pi@emonpi(rw):~$ dpkg -S /usr/lib/python2.7/dist-packages/pkg_resources/__init__.py
python-pkg-resources: /usr/lib/python2.7/dist-packages/pkg_resources/__init__.py
4

That looks good.

Can you try "ls -lAR /usr/lib/python2.7/dist-packages/pkg_resources/" or "ls -lA ls -lA /usr/lib/python2.7/dist-packages/pkg_resources/extern/"?

I should have said that before, but I was kind of expecting it wouldn't exist at all...

python-pkg-resources, like I guessed it might be. (I had checked an Ubuntu system, but not a Debian Jessie system.)

5

Here is the result before running “ sudo apt-get install --reinstall python-pkg-resources ”:

pi@emonpi(rw):~$ ls -lAR /usr/lib/python2.7/dist-packages/pkg_resources/
/usr/lib/python2.7/dist-packages/pkg_resources/:
total 228
drwxr-xr-x 2 root root   4096 Jul 12  2017 extern
-rw-r--r-- 1 root root 103308 Jan 16  2017 __init__.py
-rw-r--r-- 1 root root 115568 Jul 12  2017 __init__.pyc
drwxr-xr-x 3 root root   4096 Jul 12  2017 _vendor

/usr/lib/python2.7/dist-packages/pkg_resources/extern:
total 0

/usr/lib/python2.7/dist-packages/pkg_resources/_vendor:
total 580
-rw-r--r-- 1 root root  22374 Jan 16  2017 appdirs.py
-rw-r--r-- 1 root root  20940 Jul 12  2017 appdirs.pyc
-rw-r--r-- 1 root root      0 Jan 16  2017 __init__.py
-rw-r--r-- 1 root root    153 Jul 12  2017 __init__.pyc
drwxr-xr-x 2 root root   4096 Jul 12  2017 packaging
-rw-r--r-- 1 root root 229867 Jan 16  2017 pyparsing.py
-rw-r--r-- 1 root root 234449 Jul 12  2017 pyparsing.pyc
-rw-r--r-- 1 root root  30098 Jan 16  2017 six.py
-rw-r--r-- 1 root root  31854 Jul 12  2017 six.pyc

/usr/lib/python2.7/dist-packages/pkg_resources/_vendor/packaging:
total 168
-rw-r--r-- 1 root root   720 Jan 16  2017 __about__.py
-rw-r--r-- 1 root root   792 Jul 12  2017 __about__.pyc
-rw-r--r-- 1 root root   860 Jan 16  2017 _compat.py
-rw-r--r-- 1 root root  1256 Jul 12  2017 _compat.pyc
-rw-r--r-- 1 root root   513 Jan 16  2017 __init__.py
-rw-r--r-- 1 root root   614 Jul 12  2017 __init__.pyc
-rw-r--r-- 1 root root  8248 Jan 16  2017 markers.py
-rw-r--r-- 1 root root 12069 Jul 12  2017 markers.pyc
-rw-r--r-- 1 root root  4355 Jan 16  2017 requirements.py
-rw-r--r-- 1 root root  5197 Jul 12  2017 requirements.pyc
-rw-r--r-- 1 root root 28025 Jan 16  2017 specifiers.py
-rw-r--r-- 1 root root 25540 Jul 12  2017 specifiers.pyc
-rw-r--r-- 1 root root  1416 Jan 16  2017 _structures.py
-rw-r--r-- 1 root root  4236 Jul 12  2017 _structures.pyc
-rw-r--r-- 1 root root   421 Jan 16  2017 utils.py
-rw-r--r-- 1 root root   600 Jul 12  2017 utils.pyc
-rw-r--r-- 1 root root 11556 Jan 16  2017 version.py
-rw-r--r-- 1 root root 14592 Jul 12  2017 version.pyc

Here is the error message I get trying to run certbot --help after running “ sudo apt-get install --reinstall python-pkg-resources ”:

pi@emonpi(rw):~$ certbot --help
Traceback (most recent call last):
  File "/usr/bin/certbot", line 6, in <module>
    from pkg_resources import load_entry_point
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3019, in <module>
    @_call_aside
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3003, in _call_aside
    f(*args, **kwargs)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 3032, in _initialize_master_working_set
    working_set = WorkingSet._build_master()
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 655, in _build_master
    ws.require(__requires__)
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 963, in require
    needed = self.resolve(parse_requirements(requirements))
  File "/usr/lib/python2.7/dist-packages/pkg_resources/__init__.py", line 849, in resolve
    raise DistributionNotFound(req, requirers)
pkg_resources.DistributionNotFound: The 'ipaddress' distribution was not found and is required by cryptography
6

That's definitely not right. We probably have different versions installed, but it should be something like:

$ ls -lA /usr/lib/python2.7/dist-packages/pkg_resources/extern/
total 8
-rw-r--r-- 1 root root 2487 Jan 16  2017 __init__.py
-rw-r--r-- 1 root root 2941 May  2  2017 __init__.pyc

This time can you post "dpkg -l python-ipaddress", "dpkg -L python-ipaddress" and "ls -l /usr/lib/python2.7/dist-packages/ipaddress.py"?

1 Like
7 
pi@emonpi(rw):~$ dpkg -l python-ipaddress
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name           Version      Architecture Description
+++-==============-============-============-=================================
ii  python-ipaddre 1.0.16-1~bpo all          Backport of Python 3 ipaddress mo

pi@emonpi(rw):~$ dpkg -L python-ipaddress
/.
/usr
/usr/lib
/usr/lib/python2.7
/usr/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages/ipaddress.py
/usr/lib/python2.7/dist-packages/ipaddress-1.0.16.egg-info
/usr/lib/python2.7/dist-packages/ipaddress-1.0.16.egg-info/PKG-INFO
/usr/lib/python2.7/dist-packages/ipaddress-1.0.16.egg-info/top_level.txt
/usr/lib/python2.7/dist-packages/ipaddress-1.0.16.egg-info/dependency_links.txt
/usr/share
/usr/share/doc
/usr/share/doc/python-ipaddress
/usr/share/doc/python-ipaddress/README.Debian
/usr/share/doc/python-ipaddress/copyright
/usr/share/doc/python-ipaddress/changelog.Debian.gz

pi@emonpi(rw):~$ ls -l /usr/lib/python2.7/dist-packages/ipaddress.py
-rw-r--r-- 1 root root 79904 Dec 28  2015 /usr/lib/python2.7/dist-packages/ipaddress.py
8

That doesn’t make any sense. The file is right there! :confounded:

1 Like
9

Maybe pkg_resources checks for the package metadata instead of the module itself? If so, this might be interesting:

ls -lAR /usr/lib/python2.7/dist-packages/ipaddress*

I dunno though. :confused:

10

Well, here’s the result of that last request.

pi@emonpi(rw):~$ ls -lAR /usr/lib/python2.7/dist-packages/ipaddress*
-rw-r--r-- 1 root root 79904 Dec 28  2015 /usr/lib/python2.7/dist-packages/ipaddress.py
-rw-r--r-- 1 root root 75533 Jul 12  2017 /usr/lib/python2.7/dist-packages/ipaddress.pyc

/usr/lib/python2.7/dist-packages/ipaddress-1.0.16.egg-info:
total 0

I’m at a loss. Maybe the next thing I’ll do is try building a new emonPi on another Raspberry Pi and see if it isn’t something to do with the SD card being corrupt?

11

As a comparison, here’s what that outputs on one of my Ubuntu systems:

$ ls -lAR /usr/lib/python2.7/dist-packages/ipaddress*
-rw-r--r-- 1 root root 80176 Sep  9  2016 /usr/lib/python2.7/dist-packages/ipaddress.py
-rw-r--r-- 1 root root 75820 Apr 28  2017 /usr/lib/python2.7/dist-packages/ipaddress.pyc

/usr/lib/python2.7/dist-packages/ipaddress-1.0.17.egg-info:
total 12
-rw-r--r-- 1 root root   1 Nov  2  2016 dependency_links.txt
-rw-r--r-- 1 root root 783 Nov  2  2016 PKG-INFO
-rw-r--r-- 1 root root  10 Nov  2  2016 top_level.txt

It’s possible the .egg-info directory being empty is a Debian thing or a packaging thing, but it looks like a second case of a directory on your system being mysteriously empty. :confused:

I think either pip mangled some of your packages somehow, or there was disk or filesystem damage.

12

Hey! I just solved it. I ran this:

apt-get install -t jessie-backports python-cryptography

certbot --help starting working again. And I could actually run my renewal command, but when I did, one thing still failed:

unexpected error: No module named idna. Skipping.

so I ran these commands to reinstall:
sudo apt-get remove -t jessie-backports python-idna
sudo apt-get install -t jessie-backports python-idna
sudo apt-get install python-certbot-apache -t jessie-backports

Then everything worked again.

Thanks so much for looking at this with me. I am still not sure what changed and caused the issue. But anyway, it is now resolved.

2 Likes
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.