Cannot update Let's Encrypt certificate for site

DFrank · October 26, 2018, 10:40am

Hi,

So the certificate has expired and I want to update it. Baring in mind that I haven’t installed this initially but rather falling in from the previous dude. I can see data in /etc/letsencrypt/live , /etc/letsencrypt/renewal, /etc/letsencrypt/archive. The latest log file (path = /var/log/letsencrypt) with info in it is 2 months back with the below error. I have checked all over where the certbot script might be located / installed but no luck. The python package does seem to be installed:

||/ Name Version Architecture Description
++±==============================================-============================-============================-==================================================================================================
ii python-certbot-nginx 0.25.0-2+ubuntu16.04.1+certb all transitional dummy package

2018-08-19 22:55:37,146:DEBUG:certbot.main:certbot version: 0.21.1
2018-08-19 22:55:37,148:DEBUG:certbot.main:Arguments: [’-q’]
2018-08-19 22:55:37,148:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2018-08-19 22:55:37,693:DEBUG:certbot.log:Root logging level set at 30
2018-08-19 22:55:37,694:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2018-08-19 22:55:37,726:DEBUG:certbot.plugins.selection:Requested authenticator <certbot.cli._Default object at 0x7f4f17e0d358> and installer <certbot.cli._Default object at 0x7f4f17e0d358>
2018-08-19 22:55:37,726:DEBUG:certbot.cli:Default Detector is Namespace(account=<certbot.cli._Default object at 0x7f4f17df3898>, agree_dev_preview=None, allow_subset_of_names=<certbot.cli._Default object at 0x7f4f17df36d8>, apache=<certbot.cli._Default object at 0x7f4f17e0d5f8>, authenticator=<certbot.cli._Default object at 0x7f4f17e0d358>, break_my_certs=<certbot.cli._Default object at 0x7f4f17dede10>, cert_path=<certbot.cli._Default object at 0x7f4f17e0bbe0>, certname=<certbot.cli._Default object at 0x7f4f17e62c88>, chain_path=<certbot.cli._Default object at 0x7f4f17e0be80>, checkpoints=<certbot.cli._Default object at 0x7f4f17e0b710>, config_dir=<certbot.cli._Default object at 0x7f4f17e0bf60>, config_file=None, configurator=<certbot.cli._Default object at 0x7f4f17e0d358>, csr=<certbot.cli._Default object at 0x7f4f17e0b358>, debug=<certbot.cli._Default object at 0x7f4f17e62518>, debug_challenges=<certbot.cli._Default object at 0x7f4f17e623c8>, delete_after_revoke=<certbot.cli._Default object at 0x7f4f17e0b550>, deploy_hook=<certbot.cli._Default object at 0x7f4f17df3c18>, dialog=None, directory_hooks=<certbot.cli._Default object at 0x7f4f17df3dd8>, dns_cloudflare=<certbot.cli._Default object at 0x7f4f17e0db38>, dns_cloudxns=<certbot.cli._Default object at 0x7f4f17e0dc18>, dns_digitalocean=<certbot.cli._Default object at 0x7f4f17e0dcf8>, dns_dnsimple=<certbot.cli._Default object at 0x7f4f17e0ddd8>, dns_dnsmadeeasy=<certbot.cli._Default object at 0x7f4f17e0deb8>, dns_google=<certbot.cli._Default object at 0x7f4f17e0df98>, dns_luadns=<certbot.cli._Default object at 0x7f4f17e100b8>, dns_nsone=<certbot.cli._Default object at 0x7f4f17e10198>, dns_rfc2136=<certbot.cli._Default object at 0x7f4f17e10278>, dns_route53=<certbot.cli._Default object at 0x7f4f17e10358>, domains=<certbot.cli._Default object at 0x7f4f17e62dd8>, dry_run=<certbot.cli._Default object at 0x7f4f17e62b38>, duplicate=<certbot.cli._Default object at 0x7f4f17df39b0>, eff_email=<certbot.cli._Default object at 0x7f4f17efc6a0>, email=<certbot.cli._Default object at 0x7f4f17e627f0>, expand=<certbot.cli._Default object at 0x7f4f17df3320>, force_interactive=<certbot.cli._Default object at 0x7f4f17e62f60>, fullchain_path=<certbot.cli._Default object at 0x7f4f17e0bda0>, func=<function renew at 0x7f4f1d598400>, hsts=<certbot.cli._Default object at 0x7f4f17df0710>, http01_address=<certbot.cli._Default object at 0x7f4f17dea630>, http01_port=<certbot.cli._Default object at 0x7f4f17dea390>, ifaces=<certbot.cli._Default object at 0x7f4f17e0ba20>, init=<certbot.cli._Default object at 0x7f4f17e0b7f0>, installer=<certbot.cli._Default object at 0x7f4f17e0d358>, key_path=<certbot.cli._Default object at 0x7f4f17e0bcc0>, logs_dir=<certbot.cli._Default object at 0x7f4f17e0d160>, manual=<certbot.cli._Default object at 0x7f4f17e0d908>, manual_auth_hook=<certbot.cli._Default object at 0x7f4f17e10470>, manual_cleanup_hook=<certbot.cli._Default object at 0x7f4f17e10588>, manual_public_ip_logging_ok=<certbot.cli._Default object at 0x7f4f17e10668>, max_log_backups=‘0’, must_staple=<certbot.cli._Default object at 0x7f4f17dedfd0>, nginx=<certbot.cli._Default object at 0x7f4f17e0d710>, nginx_ctl=<certbot.cli._Default object at 0x7f4f17e0dc88>, nginx_server_root=<certbot.cli._Default object at 0x7f4f17e0de48>, no_bootstrap=<certbot.cli._Default object at 0x7f4f17ec4438>, no_self_upgrade=<certbot.cli._Default object at 0x7f4f17e4df28>, no_verify_ssl=<certbot.cli._Default object at 0x7f4f17e62ac8>, noninteractive_mode=<certbot.cli._Default object at 0x7f4f17dea0f0>, num=<certbot.cli._Default object at 0x7f4f17e0b080>, os_packages_only=<certbot.cli._Default object at 0x7f4f17df3a90>, post_hook=<certbot.cli._Default object at 0x7f4f17df31d0>, pre_hook=<certbot.cli._Default object at 0x7f4f17df33c8>, pref_challs=<certbot.cli._Default object at 0x7f4f17df3588>, prepare=<certbot.cli._Default object at 0x7f4f17e0b908>, quiet=True, reason=<certbot.cli._Default object at 0x7f4f17e0b470>, redirect=<certbot.cli._Default object at 0x7f4f17df01d0>, register_unsafely_without_email=<certbot.cli._Default object at 0x7f4f17e629e8>, reinstall=<certbot.cli._Default object at 0x7f4f17df30f0>, renew_by_default=<certbot.cli._Default object at 0x7f4f17df3518>, renew_hook=<certbot.cli._Default object at 0x7f4f17df3b38>, renew_with_new_domains=<certbot.cli._Default object at 0x7f4f17df35f8>, rsa_key_size=<certbot.cli._Default object at 0x7f4f17dedcf8>, server=<certbot.cli._Default object at 0x7f4f17e0d240>, staging=<certbot.cli._Default object at 0x7f4f17ee7eb8>, standalone=<certbot.cli._Default object at 0x7f4f17e0d828>, standalone_supported_challenges=<certbot.cli._Default object at 0x7f4f17e0dac8>, staple=<certbot.cli._Default object at 0x7f4f17df0f60>, strict_permissions=<certbot.cli._Default object at 0x7f4f17df3748>, text_mode=<certbot.cli._Default object at 0x7f4f17dea400>, tls_sni_01_address=<certbot.cli._Default object at 0x7f4f17dea080>, tls_sni_01_port=<certbot.cli._Default object at 0x7f4f17e62d68>, tos=<certbot.cli._Default object at 0x7f4f17df37b8>, uir=<certbot.cli._Default object at 0x7f4f17df0b00>, update_registration=<certbot.cli._Default object at 0x7f4f17e62898>, user_agent=<certbot.cli._Default object at 0x7f4f17e0b198>, user_agent_comment=<certbot.cli._Default object at 0x7f4f17e0b1d0>, validate_hooks=<certbot.cli._Default object at 0x7f4f17df3cf8>, verb=‘renew’, verbose_count=<certbot.cli._Default object at 0x7f4f17dea550>, webroot=<certbot.cli._Default object at 0x7f4f17e0da20>, webroot_map=<certbot.cli._Default object at 0x7f4f17e0d6a0>, webroot_path=<certbot.cli._Default object at 0x7f4f17e0df60>, work_dir=<certbot.cli._Default object at 0x7f4f17e0d080>)
2018-08-19 22:55:38,252:INFO:certbot.renewal:Cert not yet due for renewal
2018-08-19 22:55:38,252:DEBUG:certbot.renewal:no renewal failures

My domain is:
Not needed.

I ran this command:
certbot renew
OR
certbot renew --dry-run
OR
certbot

It produced this output:
certbot: command not found

My web server is (include version):
Nginx 1.10.3

The operating system my web server runs on is (include version):
Ubuntu 16.04.2

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No

What am I missing? Thanks in advance for any assistance on this

mnordhoff · October 26, 2018, 11:40am

Is that the only Certbot package installed? If you run “dpkg -l '*certbot*'”, what does it output?

Does /usr/bin/certbot exist? If you "echo $PATH", is /usr/bin in your PATH?

DFrank · October 26, 2018, 11:53am

Hello,

Command “ dpkg -l '*certbot*' ” outputs:

||/ Name Version Architecture Description
++±==============================================-============================-============================-==================================================================================================
ii certbot 0.21.1-1+ubuntu16.04.1+certb all automatically configure HTTPS using Let’s Encrypt
un python-certbot-apache (no description available)
un python-certbot-doc (no description available)
ii python-certbot-nginx 0.25.0-2+ubuntu16.04.1+certb all transitional dummy package
un python-certbot-nginx-doc (no description available)
ii python3-certbot 0.21.1-1+ubuntu16.04.1+certb all main library for certbot
un python3-certbot-apache (no description available)
ii python3-certbot-nginx 0.21.1-1+ubuntu16.04.1+certb all Nginx plugin for Certbot

When checking if “/usr/bin/certbot” exists, it’s not there however “ls -la /usr/bin/letsencrypt” shows a symbolic link to certbot (/usr/bin/letsencrypt -> certbot).

“echo $PATH” shows that ‘/usr/bin’ is in my path.

mnordhoff · October 26, 2018, 11:57am

That’s weird.

For one thing, it’s weird that Certbot is two different old versions. The current version in the PPA is 0.26.1.

For another thing, it’s weird that /usr/bin/certbot is missing. It’s (in the current version, and probably in 0.21.1) part of the certbot package, which you have installed.

Do you have any idea what’s happened to your system?

What happens of you run sudo apt update and sudo apt full-upgrade?

DFrank · October 26, 2018, 12:11pm

I know weird.

From my knowledge, nothing strange happened to the system.

I’m not sure I want to ‘apt full-upgrade’…

The log files are not showing what exactly broke or where it went wrong (at least in my observation).

It’s looking more and more like I’ll have perhaps better luck reinstalling certbot / letsencrypt?

Clueless at the moment.

mnordhoff · October 26, 2018, 12:13pm

It's necessary for one Certbot upgrade. (Some of the packages got renamed.) I think that's why it's stuck on 0.21.1.

Reinstalling it might fix it -- though it would be better to know how it broke in the first place -- but you also should upgrade it.

DFrank · October 29, 2018, 8:03am

Hey there,

So I did upgrade of Certbot (apt-get upgrade certbot). When I run ‘certbot renew --dry-run’ in root user it outputs the following:

Traceback (most recent call last):
File “/usr/bin/certbot”, line 11, in
load_entry_point(‘certbot==0.26.1’, ‘console_scripts’, ‘certbot’)()
File “/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py”, line 484, in load_entry_point
return get_distribution(dist).load_entry_point(group, name)
File “/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py”, line 2707, in load_entry_point
return ep.load()
File “/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py”, line 2325, in load
return self.resolve()
File “/usr/local/lib/python3.5/dist-packages/pkg_resources/init.py”, line 2331, in resolve
module = import(self.module_name, fromlist=[‘name’], level=0)
File “/usr/lib/python3/dist-packages/certbot/main.py”, line 9, in
import configobj
ImportError: No module named ‘configobj’

I somethings up with the current version of Python and Certbot newer version since upgrade.

Python version = Python 2.7.12

Extract from checking Certbot version:

||/ Name Version Architecture Description
++±==============================================-============================-============================-==================================================================================================
ii certbot 0.26.1-1+ubuntu16.04.1+certb all automatically configure HTTPS using Let’s Encrypt

ii python-certbot-nginx 0.25.0-2+ubuntu16.04.1+certb all transitional dummy package

ii python3-certbot 0.26.1-1+ubuntu16.04.1+certb all main library for certbot

ii python3-certbot-nginx 0.25.0-2+ubuntu16.04.1+certb all Nginx plugin for Certbot

mnordhoff · October 29, 2018, 8:09am

The newer packages are actually using Python 3. (3.5, it seems.)

Can you post "dpkg -l python3-configobj"?

DFrank · October 29, 2018, 8:14am

||/ Name Version Architecture Description
++±==============================================-============================-============================-==================================================================================================
ii python3-configobj 5.0.6-2+ubuntu16.04.1+certbo all simple but powerful config file reader and writer for Python 3

DFrank · October 29, 2018, 8:15am

Might’ve been a messy copy / paste.

python3-configobj / 5.0.6-2+ubuntu16.04.1+certbo / all

mnordhoff · October 29, 2018, 8:22am

So it’s installed but Certbot can’t find it.

How about:

ls -l /usr/lib/python3*/dist-packages/configobj.py

python3 -c "import sys; print(sys.path)"

Edit: And also, just in case:

python3.5 -c "import sys; print('\n'.join(sys.path))"

which python3

which python3.5

DFrank · October 29, 2018, 8:29am

ls -l /usr/lib/python3*/dist-packages/configobj.py

ls: cannot access ‘/usr/lib/python3*/dist-packages/configobj.py’: No such file or directory

python3 -c “import sys; print(sys.path)”

[’’, ‘/usr/lib/python35.zip’, ‘/usr/lib/python3.5’, ‘/usr/lib/python3.5/plat-x86_64-linux-gnu’, ‘/usr/lib/python3.5/lib-dynload’, ‘/usr/local/lib/python3.5/dist-packages’, ‘/opt/src/solrclient’, ‘/opt/message-broker’, ‘/usr/lib/python3/dist-packages’]

python3.5 -c “import sys; print(’\n’.join(sys.path))”

/usr/lib/python35.zip
/usr/lib/python3.5
/usr/lib/python3.5/plat-x86_64-linux-gnu
/usr/lib/python3.5/lib-dynload
/usr/local/lib/python3.5/dist-packages
/opt/src/solrclient
/opt/message-broker
/usr/lib/python3/dist-packages

which python3

/usr/bin/python3

which python3.5

/usr/bin/python3.5

mnordhoff · October 29, 2018, 8:31am

If you don’t mind triple checking something else:

find /usr -name configobj.py -ls

dpkg -L python3-configobj

This makes no sense. You’re missing files from packages that are installed. First Certbot, and now the Python library ConfigObj.

DFrank · October 29, 2018, 8:36am

No worries,

Running ‘find /usr -name configobj.py -ls’ produced nothing - no results.

dpkg -L python3-configobj

/.
/usr
/usr/lib
/usr/lib/python3
/usr/lib/python3/dist-packages
/usr/lib/python3/dist-packages/_version.py
/usr/lib/python3/dist-packages/validate.py
/usr/lib/python3/dist-packages/configobj.py
/usr/lib/python3/dist-packages/configobj-5.0.6.egg-info
/usr/lib/python3/dist-packages/configobj-5.0.6.egg-info/requires.txt
/usr/lib/python3/dist-packages/configobj-5.0.6.egg-info/top_level.txt
/usr/lib/python3/dist-packages/configobj-5.0.6.egg-info/PKG-INFO
/usr/lib/python3/dist-packages/configobj-5.0.6.egg-info/dependency_links.txt
/usr/share
/usr/share/doc
/usr/share/doc/python3-configobj
/usr/share/doc/python3-configobj/changelog.Debian.gz
/usr/share/doc/python3-configobj/copyright

mnordhoff · October 29, 2018, 8:41am

It's right there!

But it's not!

Reinstalling python3-configobj might fix this, but it's extremely weird. And there could be other missing files as well.

DFrank · October 29, 2018, 8:48am

I’m just as gobsmacked, it’s installed but not entirely there. Same for Certbot.

Thank you will try reinstalling ‘python3-configobj’ and hope for the best.

Keep in touch.

DFrank · October 29, 2018, 10:55am

Hi @mnordhoff

I have managed to get this working again. However I cannot still explain where it went all wrong.

The steps I took (for interest sake) was: Removing ‘python3-configobj’ and ‘python-certbot-nginx’ then reinstalling ‘python-certbot-nginx’ and ‘python3 -m pip install pyOpenSSL --upgrade’ .

Thanks for your assistance, much appreciated!

Good day further.

Cheers.

rg305 · October 29, 2018, 11:45am

Sounds like incomplete malware|hacker|exploit attempt.
I would shore up the system and check the access logs.

mnordhoff · October 29, 2018, 11:48am

There are other – possibly worse – options.

Using apt and pip to add and remove the same packages could possibly result in them partially deleting each other’s files.

Or disk failure.

DFrank · October 29, 2018, 1:25pm

Noted, worth checking out indeed. Thanks.