Snap Certbot package in Cron

QROkes · February 28, 2022, 1:51am

I know that Certbot has an automated timer that runs twice a day for renewals. But as double-check, I have a cronjob running once a week checking for renewals 15 23 * * 6 certbot renew --post-hook "service nginx restart", also the MAILTO is declared, so I'm receiving an email once a week with the status of all my certs, which is very convenient.

The problem started when I started using the Snap package instead of the Official PPA (Ubuntu) which is now deprecated. Everything works fine until one cert is detected that needs to be renewed, as you can see below the script stops saying "Terminated". And if I run it directly from command line everything works fine and all the domains are listed as expected.

Now I know that when I receive an email with the "Terminated" at the end, and the domains listed are incomplete, it means that one cert needs to be renewed. But, it would be nice if it could work fine in Cron. It seems like some exit codes are processed differently in Snap.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/asia.example.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/curriculum.example.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not yet due for renewal

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/dl.example.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Terminated

It's worth mentioning that some domains cannot be automatically renewed because they are using a manual validation method. So, this double-check is very useful in this case.

9peppe · February 28, 2022, 3:05am

Cron is a pain in the a*s to debug, it runs commands in a different shell that could have a different environment and a different path.

You could try adding --verbose to your command and see if it tells you something more.

_az · February 28, 2022, 3:28am

There are a few places you can investigate the unexpected termination:

Previous log files in /var/log/letsencrypt/, if the termination was caused by Python
The journald logs for snap.certbot.renew.service, if the termination was caused by systemd
The journald logs for snap.certbot.renew.timer, if the termination was caused by systemd

I would advise against having multiple crons set up like this, because the pre-installed cron from the snap won't have the --post-hook attached to it.

Putting the post-hook into /etc/letsencrypt/cli.ini could be one option to make it affect the Certbot snap's cron task.

QROkes · February 28, 2022, 3:17pm

Looking for some clues in the server logs as you suggested, the only thing I noticed is that Cron fails when looking for the Manual plugin.

The same day running from that native Certbot timer we can see that it successfully detects that one cert need to be renewed, fails to run the manual plugin, and it successfully continues checking all the other domains. The same happens if I run the renew command directly from command line, it successfully display the error message (red), and all the domains are listed as expected.

2022-02-19 15:19:03,283:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/dl.example.com/cert15.pem is signed by the certificate's issuer.
2022-02-19 15:19:03,283:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/dl.example.com/cert15.pem is: OCSPCertStatus.GOOD
2022-02-19 15:19:03,284:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2022-03-12 14:44:40 UTC.
2022-02-19 15:19:03,285:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2022-02-19 15:19:03,285:INFO:certbot._internal.renewal:Non-interactive renewal: random delay of 27.70109920322203 seconds
2022-02-19 15:19:31,004:DEBUG:certbot._internal.plugins.selection:Requested authenticator manual and installer None
2022-02-19 15:19:31,008:DEBUG:certbot._internal.plugins.disco:Other error:(PluginEntryPoint#manual): An authentication script must be provided with --manual-auth-hook >
Traceback (most recent call last):
  File "/snap/certbot/1789/lib/python3.8/site-packages/certbot/_internal/plugins/disco.py", line 160, in prepare
    self._initialized.prepare()
  File "/snap/certbot/1789/lib/python3.8/site-packages/certbot/_internal/plugins/manual.py", line 115, in prepare
    raise errors.PluginError(
certbot.errors.PluginError: An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.
2022-02-19 15:19:31,012:DEBUG:certbot._internal.plugins.selection:No candidate plugin
2022-02-19 15:19:31,012:DEBUG:certbot._internal.plugins.selection:Selected authenticator None and installer None
2022-02-19 15:19:31,012:ERROR:certbot._internal.renewal:Failed to renew certificate dl.example.com with error: The manual plugin is not working; there may be problems w>
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')
2022-02-19 15:19:31,017:DEBUG:certbot._internal.renewal:Traceback was:
Traceback (most recent call last):
  File "/snap/certbot/1789/lib/python3.8/site-packages/certbot/_internal/renewal.py", line 485, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/snap/certbot/1789/lib/python3.8/site-packages/certbot/_internal/main.py", line 1485, in renew_cert
    installer, auth = plug_sel.choose_configurator_plugins(config, plugins, "certonly")
  File "/snap/certbot/1789/lib/python3.8/site-packages/certbot/_internal/plugins/selection.py", line 258, in choose_configurator_plugins
    diagnose_configurator_problem("authenticator", req_auth, plugins)
  File "/snap/certbot/1789/lib/python3.8/site-packages/certbot/_internal/plugins/selection.py", line 364, in diagnose_configurator_problem
    raise errors.PluginSelectionError(msg)
certbot.errors.PluginSelectionError: The manual plugin is not working; there may be problems with your existing configuration.
The error was: PluginError('An authentication script must be provided with --manual-auth-hook when using the manual plugin non-interactively.')

2022-02-19 15:19:31,017:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/example.com.conf
2022-02-19 15:19:31,041:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-02-19 15:19:31,136:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-02-19 15:19:31,137:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/example.com/cert24.pem is signed by the certificate's issuer.
2022-02-19 15:19:31,137:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/example.com/cert24.pem is: OCSPCertStatus.GOOD

If we look the same day when the renew command is running from Cron, we can see it fails immediately before the Manual plugin error and terminates without checking for all the domains.

2022-02-19 23:15:04,441:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2022-02-19 23:15:04,441:DEBUG:certbot._internal.display.obj:Notifying user: Processing /etc/letsencrypt/renewal/dl.example.com.conf
2022-02-19 23:15:04,462:DEBUG:urllib3.connectionpool:Starting new HTTP connection (1): r3.o.lencr.org:80
2022-02-19 23:15:04,545:DEBUG:urllib3.connectionpool:http://r3.o.lencr.org:80 "POST / HTTP/1.1" 200 503
2022-02-19 23:15:04,546:DEBUG:certbot.ocsp:OCSP response for certificate /etc/letsencrypt/archive/dl.example.com/cert15.pem is signed by the certificate's issuer.
2022-02-19 23:15:04,547:DEBUG:certbot.ocsp:OCSP certificate status for /etc/letsencrypt/archive/dl.example.com/cert15.pem is: OCSPCertStatus.GOOD
2022-02-19 23:15:04,548:DEBUG:certbot._internal.storage:Should renew, less than 30 days before certificate expiry 2022-03-12 14:44:40 UTC.
2022-02-19 23:15:04,548:INFO:certbot._internal.renewal:Certificate is due for renewal, auto-renewing...
2022-02-19 23:15:04,548:INFO:certbot._internal.renewal:Non-interactive renewal: random delay of 290.5710651417669 seconds

2022-02-20 04:54:02,185:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2022-02-20 04:54:02,942:DEBUG:certbot._internal.main:certbot version: 1.23.0
2022-02-20 04:54:02,943:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/1789/bin/certbot
2022-02-20 04:54:02,943:DEBUG:certbot._internal.main:Arguments: ['-q', '--preconfigured-renewal']

That's why I think is related to how snap handles the exit codes from Certbot app. Would be nice if it could be fixed!
Also, thanks for your post-hook suggestion!

_az · February 28, 2022, 9:01pm

Ah, I initially misread what you were saying. I thought it was the snap timer that wasn't working properly.

That this is being caught by your shell and included in your email:

Suggests that something external is sending SIGTERM to the certbot process during the random sleep delay. Certbot shouldn't/wouldn't have exited on its own, at that point.

I would check in dmesg and /var/log/cron logs as well.

I won't discount the possibility of this being some kind of weird snapd bug, like you suggest. I'd have to try reproduce this myself when I get the chance. Or if someone else can figure out the steps to create this reliably, that'd be much appreciated.

system · March 30, 2022, 9:02pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.