Manual renew works - systemd doesn't. dns-route-53 plugin

22-7ths · June 7, 2019, 8:00pm

System: Ubuntu 16
UID: root
Auth: dns-route53 plugin installed under pip/global.

I can renew just fine using the plugin if I become root and do certbot renew.

If I let certbot.service try and do it, I get this in logs:

2019-06-07 19:36:24,969:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2019-06-07 19:36:24,969:DEBUG:certbot.plugins.selection:Requested authenticator dns-route53 and installer None
2019-06-07 19:36:24,970:DEBUG:certbot.plugins.selection:No candidate plugin
2019-06-07 19:36:24,970:DEBUG:certbot.plugins.selection:Selected authenticator None and installer None
2019-06-07 19:36:24,970:INFO:certbot.main:Could not choose appropriate plugin: The requested dns-route53 plugin does not appear to be installed
2019-06-07 19:36:24,970:WARNING:certbot.renewal:Attempting to renew cert (wildcard.domain.com) from /etc/letsencrypt/renewal/wildcard.domain.com.conf produced an unexpected error: The requested dns-route53 plugin does not appear to be installed. Skipping.
2019-06-07 19:36:24,970:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 452, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1187, in renew_cert
    installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
  File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 237, in choose_configurator_plugins
    diagnose_configurator_problem("authenticator", req_auth, plugins)
  File "/usr/lib/python3/dist-packages/certbot/plugins/selection.py", line 341, in diagnose_configurator_problem
    raise errors.PluginSelectionError(msg)
certbot.errors.PluginSelectionError: The requested dns-route53 plugin does not appear to be installed

If do it from command line log:

2019-06-07 19:52:44,291:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2019-06-07 19:52:44,292:DEBUG:certbot.plugins.selection:Requested authenticator dns-route53 and installer None
2019-06-07 19:52:44,298:DEBUG:botocore.hooks:Changing event name from creating-client-class.iot-data to creating-client-class.iot-data-plane
2019-06-07 19:52:44,302:DEBUG:botocore.hooks:Changing event name from before-call.apigateway to before-call.api-gateway
2019-06-07 19:52:44,302:DEBUG:botocore.hooks:Changing event name from request-created.machinelearning.Predict to request-created.machine-learning.Predict
2019-06-07 19:52:44,304:DEBUG:botocore.hooks:Changing event name from before-parameter-build.autoscaling.CreateLaunchConfiguration to before-parameter-build.auto-scaling.CreateLaunchConfiguration
2019-06-07 19:52:44,304:DEBUG:botocore.hooks:Changing event name from before-parameter-build.route53 to before-parameter-build.route-53
2019-06-07 19:52:44,305:DEBUG:botocore.hooks:Changing event name from request-created.cloudsearchdomain.Search to request-created.cloudsearch-domain.Search
2019-06-07 19:52:44,305:DEBUG:botocore.hooks:Changing event name from docs.*.autoscaling.CreateLaunchConfiguration.complete-section to docs.*.auto-scaling.CreateLaunchConfiguration.complete-section
2019-06-07 19:52:44,308:DEBUG:botocore.hooks:Changing event name from before-parameter-build.cloudsearchdomain.Search to before-parameter-build.cloudsearch-domain.Search
2019-06-07 19:52:44,308:DEBUG:botocore.hooks:Changing event name from docs.*.cloudsearchdomain.Search.complete-section to docs.*.cloudsearch-domain.Search.complete-section
2019-06-07 19:52:44,308:DEBUG:botocore.hooks:Changing event name from before-parameter-build.logs.CreateExportTask to before-parameter-build.cloudwatch-logs.CreateExportTask
etc.... (it works)

# certbot plugins

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
* dns-route53
Description: Obtain certificates using a DNS TXT record (if you are using AWS
Route53 for DNS).
Interfaces: IAuthenticator, IPlugin
Entry point: dns-route53 = certbot_dns_route53.dns_route53:Authenticator

* standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator

* webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator

And it’s log:

2019-06-07 20:17:25,512:DEBUG:certbot.main:certbot version: 0.35.0
2019-06-07 20:17:25,512:DEBUG:certbot.main:Arguments: []
2019-06-07 20:17:25,512:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#certbot-route53:auth,PluginEntryPoint#dns-route53,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-06-07 20:17:25,520:DEBUG:certbot.log:Root logging level set at 20
2019-06-07 20:17:25,521:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-06-07 20:17:25,521:DEBUG:certbot.main:Expected interfaces: None
2019-06-07 20:17:25,522:DEBUG:certbot.main:Filtered plugins: PluginsRegistry(PluginEntryPoint#dns-route53,PluginEntryPoint#standalone,PluginEntryPoint#webroot)

A path/pip issue is all I can think of, but am at a loss on where the problem is.

Phil · June 7, 2019, 8:32pm

Hi @22-7ths,

Welcome to the community forum! Please take a look at this github issue that deals with Ubuntu 16.04 and the route53 plugin. Let me know if that solves the issue or not.

22-7ths · June 7, 2019, 8:58pm

Thanks for reply. Don’t think that’s it. I already saw that github thread earlier. It DOES renew as mentioned (from command line only - it finds plugin just fine).

BUT, if I let the systemd service try and do it I get a plugin not found.

Please review my logs above.

edit:
added pip3 per github and I still see route53 installed fine under list. And nope: Still fails from systemd service but succeeds from command line. Therefore, auto-renew is still failing.

systemctl start certbot.service
# time passes here - sample log is in last post - finally get:
Job for certbot.service failed because the control process exited with error code. See "systemctl status certbot.service" and "journalctl -xe" for details.

systemctl status certbot.service

● certbot.service - Certbot
   Loaded: loaded (/lib/systemd/system/certbot.service; static; vendor preset: enabled)
   Active: failed (Result: exit-code) since Fri 2019-06-07 20:45:04 UTC; 2min 17s ago
     Docs: file:///usr/share/doc/python-certbot-doc/html/index.html
           https://letsencrypt.readthedocs.io/en/latest/
  Process: 32641 ExecStart=/usr/bin/certbot -q renew (code=exited, status=1/FAILURE)
 Main PID: 32641 (code=exited, status=1/FAILURE)

Jun 07 20:45:04 ip-192-168-1-5 certbot[32641]: Attempting to renew cert (wildcard.domain.com) from /etc/letsencrypt/renewal/wildcard.domain.com.conf produced an unexpected error: The requested dns-route53 plugin does not appear to be installed. Skipping.
Jun 07 20:45:04 ip-192-168-1-5 certbot[32641]: All renewal attempts failed. The following certs could not be renewed:
Jun 07 20:45:04 ip-192-168-1-5 certbot[32641]:   /etc/letsencrypt/live/redirect1.domain.com/fullchain.pem (failure)
Jun 07 20:45:04 ip-192-168-1-5 certbot[32641]:   /etc/letsencrypt/live/redirect2.domain.com/fullchain.pem (failure)
Jun 07 20:45:04 ip-192-168-1-5 certbot[32641]:   /etc/letsencrypt/live/wildcard.domain.com/fullchain.pem (failure)
Jun 07 20:45:04 ip-192-168-1-5 certbot[32641]: 3 renew failure(s), 0 parse failure(s)
Jun 07 20:45:04 ip-192-168-1-5 systemd[1]: certbot.service: Main process exited, code=exited, status=1/FAILURE
Jun 07 20:45:04 ip-192-168-1-5 systemd[1]: Failed to start Certbot.
Jun 07 20:45:04 ip-192-168-1-5 systemd[1]: certbot.service: Unit entered failed state.
Jun 07 20:45:04 ip-192-168-1-5 systemd[1]: certbot.service: Failed with result 'exit-code'.

Command line:

$ certbot renew

...
Plugins selected: Authenticator dns-route53, Installer None
...
Congratulations, all renewals succeeded. The following certs have been renewed

22-7ths · June 7, 2019, 9:05pm

Think I figured it out.

Tried a which certbot from cli

It came up with /usr/local/bin/certbot (and my path hits that location first - from command line)
But of course there is a a repo version installed version in /usr/bin. Thinking this is a problem. lol

Ughh. sorry to waste your time.I’ll report back when it’s working and if this was only issue.

EDIT: Yes, I remember now how this went down. The Ubuntu repo version stopped working for me, so I installed the pip version. Which was working fine (as noted above in log) - except the system used the repo version on renew and of course was failing (again as noted in log).

Fixed:

Wiped out any local or global pip pkgs of certbot and related.
Wiped out (apt purged/autoremoved) the certbot and python3-certbot-dns-route53 packages. Was still getting a An unexpected error occurred: pkg_resources.VersionConflict: (certbot 0.31.0 (/usr/lib/python3/dist-packages), Requirement.parse('certbot>=0.34.0')) using the purged then reinstalled repo versions. (The certbot/ubuntu repo version is 0.31.0. Finally gave up on fixing it.)
Installed pip3 version of certbot and route53 related packages. (certbot 0.35.0)

Works great now.

Phil · June 10, 2019, 2:27pm

Thanks for reporting back! Best of luck out there.

gpatel-fr · June 10, 2019, 7:37pm

be aware that at next renewal chances are high that certbot will have updated itself and wiped out manually installed plugins in the process.

22-7ths · June 12, 2019, 2:54pm

Maybe I’m not understanding? How would this happen?
If I do not have anything Certbot related installed via apt from the Ubuntu-Certbot repo?
It’s strictly manually installed via pip now.

gpatel-fr · June 14, 2019, 12:14pm

I misunderstood. I was understanding that you installed certbot-auto and added plugins with pip. If you install certbot with pip, you won't get any auto-updates so this particular problem can't happen for you.

22-7ths · June 14, 2019, 1:47pm

No problem. Yeah, I had strictly installed repo version at first:
sudo apt install certbot python3-certbot-dns-route53
Nothing via pip/manually. Pip/Pip3 wasn’t even installed early on.

This ^ version worked well for awhile. I am assuming an apt update && apt upgrade broke it at some point. I forget the error (urllib?) but I wrote it up elsewhere.

This lead me to try the direct python versions via pip3. Somehow I managed to leave the apt/repo version of certbot installed as well. It still worked if run manually (path was okay to find newer certbot), but renewal would fail as the path was different under crond or systemd executions. It would execute older repo certbot version against newer route53 package I suspect. Version dependency would fail or plugin simply would not be found in current env’s PATH. (see earlier in thread: dns-route53 plugin does not appear to be installed and pkg_resources.VersionConflict)

Hope this helps anyone else.

system · July 14, 2019, 1:47pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.