Wildcard "Auto Renewal" Fails, But Renews Manually!

[zeropeek]

Hi,
This question is not related to any one domain name.

Environment: Five Ubuntu 16.04 VPS with Certbot Cloudflare DNS plugin installed. Have active WILDCARD certificates on all 5 VPS (*.example.com). Everything works!

From CLI, when “certbot renew” command is issued, it works. Wildcard certificate is renewed successfully.

However, letsencrypt log shows that AUTOMATIC renewal returns the following error:
2019-10-07 10:16:31,468:WARNING:certbot.renewal:Attempting to renew cert (example.com) from /etc/letsencrypt/renewal/example.com.conf produced an unexpected error: RSA is not supported by the backend. Skipping.

This error is the same on each of the 5 VPS!

To summarize, wildcard renewal works manually on all 5 VPS; but auto renewal returns aforementioned error on all 5.

Any ideas about the RSA error?
Thanks

[mnordhoff]

Does letsencrypt.log include a complete traceback? Can you post it?

Are you using any control panels? How was Certbot installed? Are there multiple Certbot installations? What version(s) of Certbot are you using? What does “which -a certbot” show? What timer or cron job is being used for automatic renewal? What command is it executing?

[zeropeek]

Hi Matt,
NO - not using any control panels. Seeing two different certbot versions! Please see upload.
which -a certbot
/usr/local/bin/certbot
/usr/bin/certbot

certbot.txt (6.9 KB).
Thank you.

[mnordhoff]

It seems like there are remnants of at least 3 Certbot installations:

Certbot 0.28.0.dev0 using Python 2 in /usr/local:

where /usr/local/bin/certbot:

#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'certbot==0.28.0.dev0','console_scripts','certbot'

Certbot 0.31.0 using Python 3 in /usr:

and where /usr/bin/certbot: 
#!/usr/bin/python3
# EASY-INSTALL-ENTRY-SCRIPT: 'certbot==0.31.0','console_scripts','certbot'

And Certbot 0.23.0 using Python 2 in /usr:

2019-10-07 10:16:31,373:DEBUG:certbot.main:certbot version: 0.23.0

  File "/usr/bin/certbot", line 9, in <module>
    load_entry_point('certbot==0.23.0', 'console_scripts', 'certbot')()

Some of that makes sense: the two /usr installations might be from the PPA – an older (Python 2) python-certbot package, and the current python3-certbot package.

The installation in /usr/local might have been made with pip.

But overall, I’m not sure what’s going on. It seems like there might be multiple Python 2 installations of the Python Cryptography module, which might be conflicting or broken.

I might suggest trying to get rid of both Python 2 versions of Certbot. That would bypass any Python 2 brokenness for Certbot, but any broken dependencies left behind could still cause problems with other software.

[zeropeek]

Matt, here is some additional info:
~$ sudo certbot --version
certbot 0.28.0.dev0

sudo /usr/bin/certbot --version
certbot 0.31.0

This tells me that the manual renewal is being performed by 0.28.0

sudo systemctl status certbot.timer
● certbot.timer - Run certbot twice daily
Loaded: loaded (/lib/systemd/system/certbot.timer; enabled; vendor preset: enabled)
Active: active (waiting) since Mon 2019-10-07 13:55:23 UTC; 4h 9min ago
Given that certbot.service is pointing to certbot 0.31.0 in /usr/bin/, and “sudo /usr/bin/certbot certificates” does NOT return any certificates, this tells me that auto renewal is running 0.31.0

To summarize, manual renewal using 0.28.0 and auto renewal 0.31.0. Assuming you concur with my conclusion, question becomes what’s the best way to merge 0.28.0 with 0.31.0?
Thanks