Certbot not running in nginx on ubuntu 18.04

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
api.homadorma.com
I ran this command:
sudo certbot --nginx
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Plugins selected: Authenticator nginx, Installer nginx

An unexpected error occurred:

TypeError: init() got an unexpected keyword argument ‘cert_file’

Please see the logfiles in /var/log/letsencrypt for more details.

Error in logfile:

2020-05-06 18:23:44,348:DEBUG:certbot.main:certbot version: 0.31.0
2020-05-06 18:23:44,349:DEBUG:certbot.main:Arguments: [’–nginx’]
2020-05-06 18:23:44,349:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#certbot-route53:auth,PluginEntryPoint#dns-route53,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-05-06 18:23:44,359:DEBUG:certbot.log:Root logging level set at 20
2020-05-06 18:23:44,360:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-05-06 18:23:44,362:DEBUG:certbot.plugins.selection:Requested authenticator nginx and installer nginx
2020-05-06 18:23:44,455:DEBUG:certbot.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator
Initialized: <certbot_nginx.configurator.NginxConfigurator object at 0x7f5b0b4d5400>
Prep: True
2020-05-06 18:23:44,456:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_nginx.configurator.NginxConfigurator object at 0x7f5b0b4d5400> and installer <certbot_nginx.configurator.NginxConfigurator object at 0x7f5b0b4d5400>
2020-05-06 18:23:44,456:INFO:certbot.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2020-05-06 18:23:44,461:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri=‘https://acme-v02.api.letsencrypt.org/acme/acct/83299035’, new_authzr_uri=None, terms_of_service=None), e60609b79128dc421de8e5c64136a142, Meta(creation_dt=datetime.datetime(2020, 4, 13, 14, 3, 27, tzinfo=), creation_host=‘ip-172-31-26-211.us-east-2.compute.internal’))>
2020-05-06 18:23:44,463:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2020-05-06 18:23:44,465:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org:443
2020-05-06 18:23:44,465:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.31.0’, ‘console_scripts’, ‘certbot’)()
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1365, in main
return config.func(config, plugins)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 1111, in run
le_client = _init_le_client(config, authenticator, installer)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 612, in _init_le_client
return client.Client(config, acc, authenticator, installer, acme=acme)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 266, in init
acme = acme_from_config_key(config, self.account.key, self.account.regr)
File “/usr/lib/python3/dist-packages/certbot/client.py”, line 51, in acme_from_config_key
return acme_client.BackwardsCompatibleClientV2(net, key, config.server)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 833, in init
directory = messages.Directory.from_json(net.get(server).json())
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1171, in get
self._send_request(‘GET’, url, **kwargs), content_type=content_type)
File “/usr/lib/python3/dist-packages/acme/client.py”, line 1120, in _send_request
response = self.session.request(method, url, *args, **kwargs)
File “/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py”, line 530, in request
resp = self.send(prep, **send_kwargs)
File “/home/ubuntu/.local/lib/python3.6/site-packages/requests/sessions.py”, line 643, in send
r = adapter.send(request, **kwargs)
File “/home/ubuntu/.local/lib/python3.6/site-packages/requests/adapters.py”, line 449, in send
timeout=timeout
File “/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 659, in urlopen
conn = self._get_conn(timeout=pool_timeout)
File “/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 279, in _get_conn
return conn or self._new_conn()
File “/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connectionpool.py”, line 963, in _new_conn
**self.conn_kw
File “/usr/lib/python3/dist-packages/botocore/awsrequest.py”, line 70, in init
HTTPConnection.init(self, *args, **kwargs)
File “/home/ubuntu/.local/lib/python3.6/site-packages/urllib3/connection.py”, line 115, in init
_HTTPConnection.init(self, *args, **kw)
TypeError: init() got an unexpected keyword argument ‘cert_file’

My web server is (include version):
nginx
The operating system my web server runs on is (include version):
ubuntu 18.04
My hosting provider, if applicable, is:
AWS
I can login to a root shell on my machine (yes or no, or I don’t know):
yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
certbot 0.31.0

Further note:
The ssl is currently set up manually using certbot-auto. The thing with certbot-auto is that the auto renewal does not work.

It seems that fixing the autorenewal might be easier than probably uninstalling and reinstalling certbot.
But that’s your call.

Towards an auto-renewal fix:

  • please show the current renewal conf file for the cert in question.

Towards fixing certbot:

  • How (exactly) was it installed ?
    [some of the directories shown don’t seem correct]
  • Run a complete “sanity” check:
    sudo apt-get install debsums
    sudo debsums | grep -v OK
    sudo debsums -s
    [these may take a few minutes]
  • reinstall certbot:
    sudo apt-get install python-certbot-nginx
  • compare your packages with my Ubuntu 18.04 certbot +nginx package list:
    sudo dpkg --list | grep cerbot
ii  certbot                               0.31.0-1+ubuntu18.04.1+certbot+1                all          automatically configure HTTPS using Let's Encrypt
ii  python-certbot-nginx                  0.31.0-1+ubuntu18.04.1+certbot+1                all          transitional dummy package
ii  python3-acme                          0.31.0-2+ubuntu18.04.3+certbot+2                all          ACME protocol library for Python 3
ii  python3-augeas                        0.5.0-1+ubuntu18.04.1+certbot+1                 all          Python3 bindings for Augeas
ii  python3-certbot                       0.31.0-1+ubuntu18.04.1+certbot+1                all          main library for certbot
ii  python3-certbot-nginx                 0.31.0-1+ubuntu18.04.1+certbot+1                all          Nginx plugin for Certbot
ii  python3-configobj                     5.0.6-2+ubuntu18.04.1+certbot+1                 all          simple but powerful config file reader and writer for Python 3
ii  python3-future                        0.15.2-4+ubuntu18.04.1+certbot+3                all          Clean single-source support for Python 3 and 2 - Python 3.x
ii  python3-josepy                        1.1.0-2+ubuntu18.04.1+certbot+1                 all          JOSE implementation for Python 3.x
ii  python3-parsedatetime                 2.4-3+ubuntu18.04.1+certbot+3                   all          Python 3 module to parse human-readable date/time expressions
ii  python3-requests-toolbelt             0.8.0-1+ubuntu18.04.1+certbot+1                 all          Utility belt for advanced users of python3-requests
ii  python3-zope.component                4.3.0-1+ubuntu18.04.1+certbot+3                 all          Zope Component Architecture
ii  python3-zope.hookable                 4.0.4-4+ubuntu18.04.1+certbot+1                 amd64        Hookable object support
ii  python3-zope.interface                4.3.2-1+ubuntu18.04.1+certbot+1                 amd64        Interfaces for Python3
  • compare your python packages with my Ubuntu 18.04 python package list:
    sudo dpkg --list | grep python
rc  libpython2.7-minimal:amd64            2.7.15-4ubuntu4~18.04.2                         amd64        Minimal subset of the Python language (version 2.7)
ii  libpython3-stdlib:amd64               3.6.7-1~18.04                                   amd64        interactive high-level object-oriented language (default python3 version)
ii  libpython3.6:amd64                    3.6.9-1~18.04ubuntu1                            amd64        Shared Python runtime library (version 3.6)
ii  libpython3.6-minimal:amd64            3.6.9-1~18.04ubuntu1                            amd64        Minimal subset of the Python language (version 3.6)
ii  libpython3.6-stdlib:amd64             3.6.9-1~18.04ubuntu1                            amd64        Interactive high-level object-oriented language (standard library, version 3.6)
rc  python                                2.7.15~rc1-1                                    amd64        interactive high-level object-oriented language (default version)
ii  python-apt-common                     1.6.5ubuntu0.2                                  all          Python interface to libapt-pkg (locales)
ii  python-certbot-nginx                  0.31.0-1+ubuntu18.04.1+certbot+1                all          transitional dummy package
rc  python2.7-minimal                     2.7.15-4ubuntu4~18.04.2                         amd64        Minimal subset of the Python language (version 2.7)
ii  python3                               3.6.7-1~18.04                                   amd64        interactive high-level object-oriented language (default python3 version)
ii  python3-acme                          0.31.0-2+ubuntu18.04.3+certbot+2                all          ACME protocol library for Python 3
ii  python3-apport                        2.20.9-0ubuntu7.14                              all          Python 3 library for Apport crash report handling
ii  python3-apt                           1.6.5ubuntu0.2                                  amd64        Python 3 interface to libapt-pkg
ii  python3-asn1crypto                    0.24.0-1                                        all          Fast ASN.1 parser and serializer (Python 3)
ii  python3-attr                          17.4.0-2                                        all          Attributes without boilerplate (Python 3)
ii  python3-augeas                        0.5.0-1+ubuntu18.04.1+certbot+1                 all          Python3 bindings for Augeas
ii  python3-automat                       0.6.0-1                                         all          Self-service finite-state machines for the programmer on the go
ii  python3-blinker                       1.4+dfsg1-0.1                                   all          fast, simple object-to-object and broadcast signaling library
ii  python3-certbot                       0.31.0-1+ubuntu18.04.1+certbot+1                all          main library for certbot
ii  python3-certbot-nginx                 0.31.0-1+ubuntu18.04.1+certbot+1                all          Nginx plugin for Certbot
ii  python3-certifi                       2018.1.18-2                                     all          root certificates for validating SSL certs and verifying TLS hosts (python3)
ii  python3-cffi-backend                  1.11.5-1                                        amd64        Foreign Function Interface for Python 3 calling C code - runtime
ii  python3-chardet                       3.0.4-1                                         all          universal character encoding detector for Python3
ii  python3-click                         6.7-3                                           all          Simple wrapper around optparse for powerful command line utilities - Python 3.x
ii  python3-colorama                      0.3.7-1                                         all          Cross-platform colored terminal text in Python - Python 3.x
ii  python3-commandnotfound               18.04.5                                         all          Python 3 bindings for command-not-found.
ii  python3-configargparse                0.11.0-1                                        all          replacement for argparse with config files and environment variables (Python 3)
ii  python3-configobj                     5.0.6-2+ubuntu18.04.1+certbot+1                 all          simple but powerful config file reader and writer for Python 3
ii  python3-constantly                    15.1.0-1                                        all          Symbolic constants in Python
ii  python3-cryptography                  2.1.4-1ubuntu1.3                                amd64        Python library exposing cryptographic recipes and primitives (Python 3)
ii  python3-dbus                          1.2.6-1                                         amd64        simple interprocess messaging system (Python 3 interface)
ii  python3-debconf                       1.5.66ubuntu1                                   all          interact with debconf from Python 3
ii  python3-debian                        0.1.32                                          all          Python 3 modules to work with Debian-related data formats
ii  python3-distro-info                   0.18ubuntu0.18.04.1                             all          information about distributions' releases (Python 3 module)
ii  python3-distupgrade                   1:18.04.37                                      all          manage release upgrades
ii  python3-future                        0.15.2-4+ubuntu18.04.1+certbot+3                all          Clean single-source support for Python 3 and 2 - Python 3.x
ii  python3-gdbm:amd64                    3.6.9-1~18.04                                   amd64        GNU dbm database support for Python 3.x
ii  python3-gi                            3.26.1-2ubuntu1                                 amd64        Python 3 bindings for gobject-introspection libraries
ii  python3-httplib2                      0.9.2+dfsg-1ubuntu0.1                           all          comprehensive HTTP client library written for Python3
ii  python3-hyperlink                     17.3.1-2                                        all          Immutable, Pythonic, correct URLs.
ii  python3-icu                           1.9.8-0ubuntu1                                  amd64        Python 3 extension wrapping the ICU C++ API
ii  python3-idna                          2.6-1                                           all          Python IDNA2008 (RFC 5891) handling (Python 3)
ii  python3-incremental                   16.10.1-3                                       all          Library for versioning Python projects.
ii  python3-jinja2                        2.10-1ubuntu0.18.04.1                           all          small but fast and easy to use stand-alone template engine
ii  python3-josepy                        1.1.0-2+ubuntu18.04.1+certbot+1                 all          JOSE implementation for Python 3.x
ii  python3-json-pointer                  1.10-1                                          all          resolve JSON pointers - Python 3.x
ii  python3-jsonpatch                     1.19+really1.16-1fakesync1                      all          library to apply JSON patches - Python 3.x
ii  python3-jsonschema                    2.6.0-2                                         all          An(other) implementation of JSON Schema (Draft 3 and 4) - Python 3.x
ii  python3-jwt                           1.5.3+ds1-1                                     all          Python 3 implementation of JSON Web Token
ii  python3-markupsafe                    1.0-1build1                                     amd64        HTML/XHTML/XML string library for Python 3
ii  python3-minimal                       3.6.7-1~18.04                                   amd64        minimal subset of the Python language (default python3 version)
ii  python3-mock                          2.0.0-3                                         all          Mocking and Testing Library (Python3 version)
ii  python3-ndg-httpsclient               0.4.4-1                                         all          enhanced HTTPS support for httplib and urllib2 using PyOpenSSL for Python3
ii  python3-netifaces                     0.10.4-0.1build4                                amd64        portable network interface information - Python 3.x
ii  python3-newt:amd64                    0.52.20-1ubuntu1                                amd64        NEWT module for Python3
ii  python3-oauthlib                      2.0.6-1                                         all          generic, spec-compliant implementation of OAuth for Python3
ii  python3-openssl                       17.5.0-1ubuntu1                                 all          Python 3 wrapper around the OpenSSL library
ii  python3-pam                           0.4.2-13.2ubuntu4                               amd64        Python interface to the PAM library
ii  python3-parsedatetime                 2.4-3+ubuntu18.04.1+certbot+3                   all          Python 3 module to parse human-readable date/time expressions
ii  python3-pbr                           3.1.1-3ubuntu3                                  all          inject useful and sensible default behaviors into setuptools - Python 3.x
ii  python3-pkg-resources                 39.0.1-2                                        all          Package Discovery and Resource Access using pkg_resources
ii  python3-problem-report                2.20.9-0ubuntu7.14                              all          Python 3 library to handle problem reports
ii  python3-pyasn1                        0.4.2-3                                         all          ASN.1 library for Python (Python 3 module)
ii  python3-pyasn1-modules                0.2.1-0.2                                       all          Collection of protocols modules written in ASN.1 language (Python 3)
ii  python3-pyparsing                     2.2.0+dfsg1-2                                   all          alternative to creating and executing simple grammars - Python 3.x
ii  python3-requests                      2.18.4-2ubuntu0.1                               all          elegant and simple HTTP library for Python3, built for human beings
ii  python3-requests-toolbelt             0.8.0-1+ubuntu18.04.1+certbot+1                 all          Utility belt for advanced users of python3-requests
ii  python3-requests-unixsocket           0.1.5-3                                         all          Use requests to talk HTTP via a UNIX domain socket - Python 3.x
ii  python3-rfc3339                       1.0-4                                           all          parser and generator of RFC 3339-compliant timestamps (Python 3)
ii  python3-serial                        3.4-2                                           all          pyserial - module encapsulating access for the serial port
ii  python3-service-identity              16.0.0-2                                        all          Service identity verification for pyOpenSSL (Python 3 module)
ii  python3-six                           1.11.0-2                                        all          Python 2 and 3 compatibility library (Python 3 interface)
ii  python3-software-properties           0.96.24.32.12                                   all          manage the repositories that you install software from
ii  python3-systemd                       234-1build1                                     amd64        Python 3 bindings for systemd
ii  python3-twisted                       17.9.0-2ubuntu0.1                               all          Event-based framework for internet applications
ii  python3-twisted-bin:amd64             17.9.0-2ubuntu0.1                               amd64        Event-based framework for internet applications
ii  python3-tz                            2018.3-2                                        all          Python3 version of the Olson timezone database
ii  python3-update-manager                1:18.04.11.12                                   all          python 3.x module for update-manager
ii  python3-urllib3                       1.22-1ubuntu0.18.04.1                           all          HTTP library with thread-safe connection pooling for Python3
ii  python3-yaml                          3.12-1build2                                    amd64        YAML parser and emitter for Python3
ii  python3-zope.component                4.3.0-1+ubuntu18.04.1+certbot+3                 all          Zope Component Architecture
ii  python3-zope.event                    4.2.0-1                                         all          Very basic event publishing system
ii  python3-zope.hookable                 4.0.4-4+ubuntu18.04.1+certbot+1                 amd64        Hookable object support
ii  python3-zope.interface                4.3.2-1+ubuntu18.04.1+certbot+1                 amd64        Interfaces for Python3
ii  python3.6                             3.6.9-1~18.04ubuntu1                            amd64        Interactive high-level object-oriented language (version 3.6)
ii  python3.6-minimal                     3.6.9-1~18.04ubuntu1                            amd64        Minimal subset of the Python language (version 3.6)

I haven't looked into the specifics, but...

I guess the urllib3 Python module installed in /home/ubuntu/ is incompatible with something else. Maybe it's too new, or too old, or broken somehow.

Do you know why it's installed, and what version it is?

If you run "sudo -H certbot --nginx", it may work around whatever's happening, since it will set HOME to /root and Python may be less likely to find modules in /home/ubuntu.

Edit: Googling the error, it's probably requests and urllib3 that are incompatible.

Thanks. sudo -H certbot --nginx works. No error reported this time.

I then tested renew. I ran sudo certbot renew --dry-run, same error.
I ran sudo -H certbot renew --dry-run, passed. So I believe I should change the way the renew command is run.

I ran crontab -e, found this line:
0 0 * * 0 sudo certbot renew

My understanding is that if I change it to 0 0 * * 0 sudo -H certbot renew, the renew should work.

Is my understanding correct?