Python error after Upgrade from Deb8 to Deb9


#1

Hello,

after Upgrading from Debian 8 to Debian 9, I get the following error when running certbot:

Traceback (most recent call last):
File "/usr/bin/certbot", line 11, in <module>
  load_entry_point('certbot==0.23.0', 'console_scripts', 'certbot')()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line  561, in load_entry_point
  return get_distribution(dist).load_entry_point(group, name)
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line  2631, in load_entry_point
  return ep.load()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line  2291, in load
  return self.resolve()
File "/usr/lib/python3/dist-packages/pkg_resources/__init__.py", line  2297, in resolve
  module = __import__(self.module_name, fromlist=['__name__'],    level=0)
File "/usr/lib/python3/dist-packages/certbot/main.py", line 9, in  <module>
  import configobj
File "/usr/lib/python3/dist-packages/configobj.py", line 22, in  <module>
  import six
ImportError: bad magic number in 'six': b'\x03\xf3\r\n'

I can’t find anything helpful on the net regarding this special error in “six”. I can find references that if the magic number is not recognized, the file may be defective and can be deleted. It will be recompiled by Python afterwards.

rm /usr/lib/python3/dist-packages/__pycache__/six.cpython-35.pyc /usr/lib/python3/dist-packages/pkg_resources/_vendor/__pycache__/six.cpython-35.pyc; apt-get install --reinstall python3-six

=> Error persists.

Deinstalled (purged) Certbot and any python packages no longer being held as as dependency. Deleted anything regarding python from /usr/local. Reinstalled certbot: apt-get install python-certbot-apache -t stretch-backports

=> Error persists.

Perhaps Bytecode will be rewritten?

dpkg-reconfigure python python3 python3.5 python3.5-minimal python2.7 python2.7-minimal libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.5 libpython3.5-minimal libpython3.5-stdlib

=> Error persists.

Perhaps defective packages?

apt-get clean; apt-get install --reinstall python3.5 python3.5-minimal python2.7 python2.7-minimal libpython-stdlib libpython2.7 libpython2.7-minimal libpython2.7-stdlib libpython3-stdlib libpython3.5 libpython3.5-minimal libpython3.5-stdlib python3-configobj python3-six

=> Error persists.

I’m out of knowledge, since I’m a Python-Noob. :slight_smile: May I ask kindly for help and forward-apology if this is not a Certbot problem?

Thank you!

:wq! PoC


#2

Hi @PoC, “certbot problem” or not you’re in the right place! :grinning:

Please paste the output of:

python3 -vc 'import six'

That should tell us if Python is finding a version of six in some other place besides Debian’s dist-packages, and maybe a little bit more about why it is failing.


#3

Hi Patches,

thank you for offering to help!

Output follows:
import _frozen_importlib # frozen
import _imp # builtin
import sys # builtin
import ‘_warnings’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘_thread’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘_weakref’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘_frozen_importlib_external’ # <class ‘_frozen_importlib.FrozenImporter’>
import ‘_io’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘marshal’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘posix’ # <class ‘_frozen_importlib.BuiltinImporter’>
import _thread # previously loaded (’_thread’)
import ‘_thread’ # <class ‘_frozen_importlib.BuiltinImporter’>
import _weakref # previously loaded (’_weakref’)
import ‘_weakref’ # <class ‘_frozen_importlib.BuiltinImporter’>
# installing zipimport hook
import ‘zipimport’ # <class ‘_frozen_importlib.BuiltinImporter’>
# installed zipimport hook
# /usr/lib/python3.5/encodings/pycache/init.cpython-35.pyc matches /usr/lib/python3.5/encodings/init.py
# code object from ‘/usr/lib/python3.5/encodings/pycache/init.cpython-35.pyc’
# /usr/lib/python3.5/pycache/codecs.cpython-35.pyc matches /usr/lib/python3.5/codecs.py
# code object from ‘/usr/lib/python3.5/pycache/codecs.cpython-35.pyc’
import ‘_codecs’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘codecs’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d7309fcf8>
# /usr/lib/python3.5/encodings/pycache/aliases.cpython-35.pyc matches /usr/lib/python3.5/encodings/aliases.py
# code object from ‘/usr/lib/python3.5/encodings/pycache/aliases.cpython-35.pyc’
import ‘encodings.aliases’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d730377f0>
import ‘encodings’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d7309f908>
# /usr/lib/python3.5/encodings/pycache/utf_8.cpython-35.pyc matches /usr/lib/python3.5/encodings/utf_8.py
# code object from ‘/usr/lib/python3.5/encodings/pycache/utf_8.cpython-35.pyc’
import ‘encodings.utf_8’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d730425c0>
import ‘_signal’ # <class ‘_frozen_importlib.BuiltinImporter’>
# /usr/lib/python3.5/encodings/pycache/latin_1.cpython-35.pyc matches /usr/lib/python3.5/encodings/latin_1.py
# code object from ‘/usr/lib/python3.5/encodings/pycache/latin_1.cpython-35.pyc’
import ‘encodings.latin_1’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d730430f0>
# /usr/lib/python3.5/pycache/io.cpython-35.pyc matches /usr/lib/python3.5/io.py
# code object from ‘/usr/lib/python3.5/pycache/io.cpython-35.pyc’
# /usr/lib/python3.5/pycache/abc.cpython-35.pyc matches /usr/lib/python3.5/abc.py
# code object from ‘/usr/lib/python3.5/pycache/abc.cpython-35.pyc’
# /usr/lib/python3.5/pycache/_weakrefset.cpython-35.pyc matches /usr/lib/python3.5/_weakrefset.py
# code object from ‘/usr/lib/python3.5/pycache/_weakrefset.cpython-35.pyc’
import ‘_weakrefset’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d73043d68>
import ‘abc’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d73043588>
import ‘io’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d73043320>
# /usr/lib/python3.5/pycache/site.cpython-35.pyc matches /usr/lib/python3.5/site.py
# code object from ‘/usr/lib/python3.5/pycache/site.cpython-35.pyc’
# /usr/lib/python3.5/pycache/os.cpython-35.pyc matches /usr/lib/python3.5/os.py
# code object from ‘/usr/lib/python3.5/pycache/os.cpython-35.pyc’
import ‘errno’ # <class ‘_frozen_importlib.BuiltinImporter’>
# /usr/lib/python3.5/pycache/stat.cpython-35.pyc matches /usr/lib/python3.5/stat.py
# code object from ‘/usr/lib/python3.5/pycache/stat.cpython-35.pyc’
import ‘_stat’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘stat’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fee400>
# /usr/lib/python3.5/pycache/posixpath.cpython-35.pyc matches /usr/lib/python3.5/posixpath.py
# code object from ‘/usr/lib/python3.5/pycache/posixpath.cpython-35.pyc’
# /usr/lib/python3.5/pycache/genericpath.cpython-35.pyc matches /usr/lib/python3.5/genericpath.py
# code object from ‘/usr/lib/python3.5/pycache/genericpath.cpython-35.pyc’
import ‘genericpath’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72ff1c88>
import ‘posixpath’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fee748>
# /usr/lib/python3.5/pycache/_collections_abc.cpython-35.pyc matches /usr/lib/python3.5/_collections_abc.py
# code object from ‘/usr/lib/python3.5/pycache/_collections_abc.cpython-35.pyc’
import ‘_collections_abc’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72ff9320>
import ‘os’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d73058ba8>
# /usr/lib/python3.5/pycache/_sitebuiltins.cpython-35.pyc matches /usr/lib/python3.5/_sitebuiltins.py
# code object from ‘/usr/lib/python3.5/pycache/_sitebuiltins.cpython-35.pyc’
import ‘_sitebuiltins’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d73058e80>
# /usr/lib/python3.5/pycache/sysconfig.cpython-35.pyc matches /usr/lib/python3.5/sysconfig.py
# code object from ‘/usr/lib/python3.5/pycache/sysconfig.cpython-35.pyc’
import ‘sysconfig’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d73025da0>
# /usr/lib/python3.5/pycache/_sysconfigdata.cpython-35.pyc matches /usr/lib/python3.5/_sysconfigdata.py
# code object from ‘/usr/lib/python3.5/pycache/_sysconfigdata.cpython-35.pyc’
# /usr/lib/python3.5/plat-x86_64-linux-gnu/pycache/_sysconfigdata_m.cpython-35.pyc matches /usr/lib/python3.5/plat-x86_64-linux-gnu/_sysconfigdata_m.py
# code object from ‘/usr/lib/python3.5/plat-x86_64-linux-gnu/pycache/_sysconfigdata_m.cpython-35.pyc’
import ‘_sysconfigdata_m’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fb6828>
import ‘_sysconfigdata’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fb6630>
# /usr/lib/python3.5/pycache/_bootlocale.cpython-35.pyc matches /usr/lib/python3.5/_bootlocale.py
# code object from ‘/usr/lib/python3.5/pycache/_bootlocale.cpython-35.pyc’
import ‘_locale’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘_bootlocale’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fc38d0>
# /usr/lib/python3.5/pycache/types.cpython-35.pyc matches /usr/lib/python3.5/types.py
# code object from ‘/usr/lib/python3.5/pycache/types.cpython-35.pyc’
# /usr/lib/python3.5/pycache/functools.cpython-35.pyc matches /usr/lib/python3.5/functools.py
# code object from ‘/usr/lib/python3.5/pycache/functools.cpython-35.pyc’
import ‘_functools’ # <class ‘_frozen_importlib.BuiltinImporter’>
# /usr/lib/python3.5/collections/pycache/init.cpython-35.pyc matches /usr/lib/python3.5/collections/init.py
# code object from ‘/usr/lib/python3.5/collections/pycache/init.cpython-35.pyc’
# /usr/lib/python3.5/pycache/operator.cpython-35.pyc matches /usr/lib/python3.5/operator.py
# code object from ‘/usr/lib/python3.5/pycache/operator.cpython-35.pyc’
import ‘_operator’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘operator’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d719a0cc0>
# /usr/lib/python3.5/pycache/keyword.cpython-35.pyc matches /usr/lib/python3.5/keyword.py
# code object from ‘/usr/lib/python3.5/pycache/keyword.cpython-35.pyc’
import ‘keyword’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d719abe48>
# /usr/lib/python3.5/pycache/heapq.cpython-35.pyc matches /usr/lib/python3.5/heapq.py
# code object from ‘/usr/lib/python3.5/pycache/heapq.cpython-35.pyc’
import ‘_heapq’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘heapq’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d719ad710>
import ‘itertools’ # <class ‘_frozen_importlib.BuiltinImporter’>
# /usr/lib/python3.5/pycache/reprlib.cpython-35.pyc matches /usr/lib/python3.5/reprlib.py
# code object from ‘/usr/lib/python3.5/pycache/reprlib.cpython-35.pyc’
import ‘reprlib’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d719adba8>
import ‘_collections’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘collections’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fda7f0>
# /usr/lib/python3.5/pycache/weakref.cpython-35.pyc matches /usr/lib/python3.5/weakref.py
# code object from ‘/usr/lib/python3.5/pycache/weakref.cpython-35.pyc’
import ‘weakref’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d719b9908>
import ‘functools’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fcf518>
# /usr/lib/python3.5/collections/pycache/abc.cpython-35.pyc matches /usr/lib/python3.5/collections/abc.py
# code object from ‘/usr/lib/python3.5/collections/pycache/abc.cpython-35.pyc’
import ‘collections.abc’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fd44e0>
import ‘types’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fcb898>
# /usr/lib/python3.5/pycache/sitecustomize.cpython-35.pyc matches /usr/lib/python3.5/sitecustomize.py
# code object from ‘/usr/lib/python3.5/pycache/sitecustomize.cpython-35.pyc’
import ‘sitecustomize’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d72fcb710>
import ‘site’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d730536a0>
Python 3.5.3 (default, Jan 19 2017, 14:11:04)
[GCC 6.3.0 20170118] on linux
Type “help”, “copyright”, “credits” or “license” for more information.
# /usr/lib/python3/dist-packages/pycache/six.cpython-35.pyc matches /usr/lib/python3/dist-packages/six.py
# code object from ‘/usr/lib/python3/dist-packages/pycache/six.cpython-35.pyc’
# /usr/lib/python3.5/pycache/future.cpython-35.pyc matches /usr/lib/python3.5/future.py
# code object from ‘/usr/lib/python3.5/pycache/future.cpython-35.pyc’
import ‘future’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d719614a8>
# /usr/lib/python3.5/pycache/struct.cpython-35.pyc matches /usr/lib/python3.5/struct.py
# code object from ‘/usr/lib/python3.5/pycache/struct.cpython-35.pyc’
import ‘_struct’ # <class ‘_frozen_importlib.BuiltinImporter’>
import ‘struct’ # <_frozen_importlib_external.SourceFileLoader object at 0x7f8d7196ec88>
import ‘six’ # <frozen_importlib_external.SourceFileLoader object at 0x7f8d72fcb7f0>
# clear builtins.

# clear sys.path
# clear sys.argv
# clear sys.ps1
# clear sys.ps2
# clear sys.last_type
# clear sys.last_value
# clear sys.last_traceback
# clear sys.path_hooks
# clear sys.path_importer_cache
# clear sys.meta_path
# clear sys.interactivehook
# clear sys.flags
# clear sys.float_info
# restore sys.stdin
# restore sys.stdout
# restore sys.stderr
# cleanup[2] removing _imp
# cleanup[2] removing collections.abc
# cleanup[2] removing weakref
# destroy weakref
# cleanup[2] removing _operator
# cleanup[2] removing posix
# cleanup[2] removing types
# cleanup[2] removing itertools
# cleanup[2] removing sys
# cleanup[2] removing functools
# cleanup[2] removing keyword
# destroy keyword
# cleanup[2] removing builtins
# cleanup[2] removing io
# cleanup[2] removing _sitebuiltins
# cleanup[2] removing _frozen_importlib
# cleanup[2] removing _thread
# cleanup[2] removing _frozen_importlib_external
# cleanup[2] removing _stat
# cleanup[2] removing _io
# cleanup[2] removing marshal
# cleanup[2] removing _codecs
# cleanup[2] removing zipimport
# cleanup[2] removing operator
# cleanup[2] removing main
# destroy main
# cleanup[2] removing _sysconfigdata
# destroy _sysconfigdata
# cleanup[2] removing _warnings
# cleanup[2] removing codecs
# cleanup[2] removing _collections_abc
# cleanup[2] removing future
# destroy future
# cleanup[2] removing collections
# cleanup[2] removing errno
# cleanup[2] removing heapq
# cleanup[2] removing _signal
# cleanup[2] removing sitecustomize
# destroy sitecustomize
# cleanup[2] removing reprlib
# destroy reprlib
# cleanup[2] removing _weakrefset
# destroy _weakrefset
# cleanup[2] removing abc
# cleanup[2] removing _struct
# cleanup[2] removing encodings.utf_8
# cleanup[2] removing _locale
# cleanup[2] removing _heapq
# cleanup[2] removing encodings.aliases
# cleanup[2] removing os
# cleanup[2] removing _collections
# cleanup[2] removing six
# destroy six
# cleanup[2] removing genericpath
# cleanup[2] removing stat
# cleanup[2] removing _bootlocale
# destroy _bootlocale
# cleanup[2] removing _sysconfigdata_m
# destroy _sysconfigdata_m
# cleanup[2] removing sysconfig
# destroy sysconfig
# cleanup[2] removing encodings
# destroy encodings
# cleanup[2] removing site
# destroy site
# cleanup[2] removing encodings.latin_1
# cleanup[2] removing struct
# destroy struct
# cleanup[2] removing os.path
# cleanup[2] removing _functools
# cleanup[2] removing _weakref
# cleanup[2] removing posixpath
# destroy zipimport
# destroy _signal
# destroy abc
# destroy _sitebuiltins
# destroy posixpath
# destroy errno
# destroy _stat
# destroy genericpath
# destroy stat
# destroy os
# destroy _locale
# destroy _functools
# destroy _collections_abc
# destroy collections.abc
# destroy heapq
# destroy _struct
# destroy _heapq
# destroy _collections
# destroy collections
# destroy _operator
# destroy operator
# destroy types
# destroy six.moves.urllib
# destroy functools
# destroy itertools
# destroy six.moves
# destroy io
# destroy six.moves.urllib.error
# destroy six.moves.urllib.robotparser
# destroy six.moves.urllib.request
# destroy six.moves.urllib_parse
# destroy six.moves.urllib.response
# cleanup[3] wiping _imp
# cleanup[3] wiping posix
# cleanup[3] wiping _frozen_importlib
# destroy _frozen_importlib_external
# cleanup[3] wiping _thread
# cleanup[3] wiping _io
# cleanup[3] wiping marshal
# cleanup[3] wiping _codecs
# cleanup[3] wiping _warnings
# cleanup[3] wiping codecs
# cleanup[3] wiping encodings.utf_8
# cleanup[3] wiping encodings.aliases
# cleanup[3] wiping encodings.latin_1
# cleanup[3] wiping _weakref
# cleanup[3] wiping sys
# cleanup[3] wiping builtins

/usr/lib/python3/dist-packages/__pycache__/six.cpython-35.pyc seems to be the only location being referenced.

I renamed the file, tried again, same error with certbot. But interestingly, no new pyc-file has been created.

Any hints how to proceed?

Thanks for taking time!

:wq! PoC


#4

Strangely, import six does not fail with the bad magic error when called by itself, though it does seem to when certbot calls it.

So let’s run certbot under python -v and see if that is more helpful:

 python3 -v /usr/bin/certbot

The output from this may be too long to directly paste here, so you may need to use a service like pastebin.com.


#5

Thanks! See https://pastebin.com/si5DGkuq.


#6

So, based on the first command you ran, I thought you might have to use six to get your error. But the output from the second command shows that it really happens right when you import it, before you use it, and that when certbot does import six something different happens than when you do it from a fresh Python interpreter.

The most likely explanation for that is certbot is somehow importing a different version of six than a plain Python interpreter does, but unfortunately python3 -v only told us the paths of the successful imports, so we’ll have to retrieve the path of the failing one from the exception.

Execute this in your terminal:

python3 <<EOF
try: exec(open('/usr/bin/certbot').read())
except ImportError as e: print(e.path)
EOF

This should hopefully output the failing path (but if we’re unlucky it might just say None). If this six isn’t in the normal Debian dist-packages directory you’ve already reinstalled, go ahead and backup and delete it from there, and then try running certbot again.


#7

Funny things happen! The error is not thrown when exec’ing your recommendation!

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache

Which names would you like to activate HTTPS for?
[…]
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): An unexpected error occurred:
EOFError
Please see the logfiles in /var/log/letsencrypt for more details.

Now I’m even more confused! Just starting /usr/bin/certbot immediately throws the error. The first line in certbot references /usr/bin/python3, which is a symlink to python3.5. There’s no further python3 or python3.5 executable on the system. Do you have more ideas how to proceed?

I have vmware-vcli installed, though, and I had to reinstall after Deb9-Upgrade, because the perl-version changed and some modules seem to be build on the fly while installing. There’s also a /usr/lib/vmware-vcli/bin/esxcli/lib64/python3.5-directory but it’s not in $PATH and not in $LD_LIBRARY_PATH. It does not contain any .py or .pyc-files, only shared objects. /usr/lib/vmware-vcli/bin/esxcli/lib64 is not mentioned in /etc/ld.so.conf*, so I don’t think that this could be the issue. I do mention this for completeness and if I’m possibly in error with my thoughts.


#8

Next Idea I had failed. I found two locations of python-six-modules:
/usr/lib/python3/dist-packages/pycache/six.cpython-35.pyc
/usr/lib/python3/dist-packages/pkg_resources/_vendor/pycache/six.cpython-35.pyc

Renaming these and reinstalling python3-six-package doesn’t change the behavior of certbot: ImportError: bad magic number in 'six': b'\x03\xf3\r\n'

The modules have been recreated, though.


#9

Interesting, maybe exec is bypassing bytecode interpretation somehow.

While checking if there was a way to do that on the command line as a quick workaround (apparently not, you can only stop Python from writing it) so you can go ahead and take care of your certificates, I noticed that Python should tell me what I needed before if you pass it two 'v's, so please pastebin:

 python3 -vv /usr/bin/certbot

P.S. is your certificate really close to expiration? We might be able to use what we’ve found to renew it real quick if necessary if it is an emergency. If you still have some time we might as well focus on fixing it permanently though. :wink:


#10

For output of -vv see https://pastebin.com/0ZGYzD8w.

No need to hurry, next cert expiration happens at May 31.

Thanks again!


#11
ls -l /usr/bin/six*

Is there actually some sort of six module in /usr/bin?

Edit: By the way, \x03\xf3 (62211) is Python 2.7.


#12

Bingo!

-r--r--r-- 1 root root 31988 Apr 29 13:29 /usr/bin/six.pyc

/usr/bin/six.pyc: python 2.7 byte-compiled

Removed it, certbot runs perfectly.

Jackpot questions:

  • How did this file appear?
  • Why does python3 try to load this file?

Thanks to the two of you! Problem solved!


#13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.