I'm trying to use the 'dns-hetzner' plugin on Rocky Linux 9.2, certbot 2.5.0, python 3.9.16, but after installation it is not listed in the output of 'certbot plugins'
As root I did:
dnf install certbot
pip3 install certbot-dns-hetzner
certbot plugins
Tried rebooting but plugin is still not listed.
The plugin was installed (by pip) in /usr/local/lib/python3.9/site-packages/certbot_dns_hetzner. Is there a way to "tell" certbot to "look" there for plugin(s)?
Hi @repacks, and welcome to the LE community forum 
I have zero experience with Hetzner...
But I do see that acme.sh supports it:
dnsapi · acmesh-official/acme.sh Wiki · GitHub
[also, certbot latest version is now 2.6.0]
Certbot should be able to see plugins installed with pip.
Is there just a single Python version installed or are there multiple by any chance?
Yeah that's what I expected. This was the behavior in Rocky 8.x, certbot just recognized the plugin no problem. I don't believe there is another version of Python installed. I tried:
ls -l /usr/bin | grep -i python and only see one version.
Although I assume there is a more thorough way to find installed Python versions.
Does the letsencrypt.log file mention anything funny?
Nothing that seems useful to me. I did try to retrieve a cert (certbot -v certonly ...) just in case in case 'certbot plugins' is/was not reliable. Log file shows:
2023-05-19 09:26:25,092:DEBUG:certbot._internal.main:certbot version: 2.5.0
2023-05-19 09:26:25,093:DEBUG:certbot._internal.main:Location of certbot entry point: /usr/bin/certbot
2023-05-19 09:26:25,093:DEBUG:certbot._internal.main:Arguments: ['-v', '--authenticator', 'dns-hetzner']
2023-05-19 09:26:25,093:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2023-05-19 09:26:25,102:DEBUG:certbot._internal.log:Root logging level set at 20
2023-05-19 09:26:25,102:DEBUG:certbot._internal.plugins.selection:Requested authenticator dns-hetzner and installer None
2023-05-19 09:26:25,102:DEBUG:certbot._internal.plugins.selection:No candidate plugin
2023-05-19 09:26:25,102:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
File "/usr/bin/certbot", line 8, in <module>
sys.exit(main())
File "/usr/lib/python3.9/site-packages/certbot/main.py", line 19, in main
return internal_main.main(cli_args)
File "/usr/lib/python3.9/site-packages/certbot/_internal/main.py", line 1864, in main
return config.func(config, plugins)
File "/usr/lib/python3.9/site-packages/certbot/_internal/main.py", line 1578, in certonly
installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
File "/usr/lib/python3.9/site-packages/certbot/_internal/plugins/selection.py", line 255, in choose_configurator_plugins
diagnose_configurator_problem("authenticator", req_auth, plugins)
File "/usr/lib/python3.9/site-packages/certbot/_internal/plugins/selection.py", line 373, in diagnose_configurator_problem
raise errors.PluginSelectionError(msg)
certbot.errors.PluginSelectionError: The requested dns-hetzner plugin does not appear to be installed
2023-05-19 09:26:25,104:ERROR:certbot._internal.log:The requested dns-hetzner plugin does not appear to be installed
So I found a workaround. After much poking around here are the steps that got me to a mostly working solution. Not sure how to handle automated renewals just yet. I'm posting hopefully enough detail to help others whose searches may end up here and/or someone who might be able to create a more permanent solution.
python -m trace --listfuncs /usr/bin/certbot plugins
This produced LOTS of output, but the line that stood out to me was:
filename: /usr/lib/python3.9/site-packages/certbot/_internal/plugins/disco.py, modulename: disco, funcname: find_all
In fact in the file .../disco.py in the function find_all I found the line:
plugin_paths_string = os.getenv('CERTBOT_PLUGIN_PATH')
Hmmm, does 'CERTBOT_PLUGIN_PATH' exist?
echo $CERTBOT_PLUGIN_PATH
Nope
So:
export CERTBOT_PLUGIN_PATH=/usr/local/lib/python3.9/site-packages/
certbot plugins
* dns-hetzner
Description: Obtain certificates using a DNS TXT record (if you are using
Hetzner for DNS).
Interfaces: Authenticator, Plugin
Entry point: dns-hetzner = certbot_dns_hetzner.dns_hetzner:Authenticator
And
certbot -v certonly --authenticator dns-hetzner --dns-hetzner-credentials /etc/hetzner.ini -d '...'
Now works!
Weird.
Certbot indeed can use that environment variable, but looking at the code that variable just extends the sys.path property, which should contain the paths where pip has put the plugin, right?
You might want to check your sys.path by just running a Python "shell" (e.g. python) and run:
import sys
print(sys.path)
And see what it comes up with.
Perhaps putting something like this in the cli.ini file might do the trick:
CERTBOT-PLUGIN-PATH=/usr/local/lib/python3.9/site-packages/
[wild guess]
Yeah I did try that:
python3
import sys
print(sys.path)
['', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/usr/local/lib/python3.9/site-packages', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']
Hm, interesting.
Looks like disco.py does do some stuff with the plugin_paths and not directly something with sys.path, I'm not sure how this actually works:
I'd assume sys.path is also used somehow? But where and when, I dunno.
Out of curiosity, I added this after line 182:
print(sys.path)
Then in another terminal that does not have CERTBOT_PLUGIN_PATH set:
certbot plugins
['/usr/bin', '/usr/lib64/python39.zip', '/usr/lib64/python3.9', '/usr/lib64/python3.9/lib-dynload', '/usr/lib64/python3.9/site-packages', '/usr/lib/python3.9/site-packages']
For some reason that path is different from what I see in the shell. Hmmm I also dunno what's going on.
That's weird, right? 
Indeed! It really does "smell" like there's another version of python installed somewhere. Or some python'ism that I don't understand. Anyway thank you for taking the time!
Ok - one last tidbit of info on this topic. The first line of /usr/bin/certbot is:
#! /usr/bin/python3 -s
From man python:
COMMAND LINE OPTIONS
-s Don't add user site directory to sys.path.
The certbot version in Rocky 8.x does not have the -s switch in /usr/bin/certbot. I'm assuming the certbot devs had a reason to add -s. At least I'll sleep better tonight having a better idea of what's happening. 
That's probably put there by the package maintainers. In the snapcraft file from Certbot I see they also use the -s option for the Certbot command for snap it looks like. But in the pip installed Certbot there's no such option. So it seems to differ.
Hmm, looking at site — Site-specific configuration hook — Python 3.11.3 documentation though it seems "user site" does not actually mean /usr/local/lib/python3.9 et cetera?
The output of python -m site might also be interesting it seems from some Googleing. And than also with the -s option to compare.
Nice job finding the cause.
It looks like this bug has already previously been filed against EPEL: https://bugzilla.redhat.com/show_bug.cgi?id=2145166, so adding your votes and input there would help.
As far as I can tell, this issue is new in EPEL9. The -s flag is not present in the EPEL8 package.
In general, mixing OS Python packages and pip ones is a bad idea. Using the official pip instructions is recommended here, or picking a plugin which has an installation method that matches your Certbot installation method.
Thanks for that, I definitely will!
Yeah am very much aware of the pitfalls of mixing OS and pip installs, as a rule I NEVER do it for production servers. Unfortunately we have many servers NOT hosted on Hetzner for which EPEL dns plugins do exist. So this is a bit of an exploration with the goal of maintaining commonality amongst servers to the extent possible.
Yes indeed.
python -s -m site
sys.path = [
'/root',
'/usr/lib64/python39.zip',
'/usr/lib64/python3.9',
'/usr/lib64/python3.9/lib-dynload',
'/usr/lib64/python3.9/site-packages',
'/usr/lib/python3.9/site-packages',
]
python -m site
sys.path = [
'/root',
'/usr/lib64/python39.zip',
'/usr/lib64/python3.9',
'/usr/lib64/python3.9/lib-dynload',
'/usr/local/lib/python3.9/site-packages',
'/usr/lib64/python3.9/site-packages',
'/usr/lib/python3.9/site-packages',
]