Renew with Crontab doesn't appear to be working

ccheath · May 16, 2020, 8:41pm

My domain is: server.eyethrees.net
My web server is: Apache v2.4.6
The OS my web server runs on is: centos-release-7-7.1908.0.el7.centos.x86_64
I can login to a root shell on my machine: yes
I’m using a control panel to manage my site: Webmin v1.94 1
The version of my client is: certbot version 1.0.0

so after installing my certs I used the following command to edit the crontab for renewal as recommended by the install page ( https://certbot.eff.org/lets-encrypt/centosrhel7-apache.html )
echo "0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew -q" | sudo tee -a /etc/crontab > /dev/null

the crontab seems to be running: tail of /var/log/cron shows
May 16 12:00:01 server CROND[34615]: (root) CMD (python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew -q)

but i’ve gotten two emails about expiration … 30days out and 20days out now…

i could run the renewal manually (but i did that last time)

$cat /etc/crontab
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew -q

sudo crontab -e
PATH=/sbin:/bin:/usr/sbin:/usr/bin
53 23 * * 4 /etc/webmin/package-updates/update.pl
@daily /etc/webmin/webalizer/webalizer.pl /etc/httpd/logs/access_log

_az · May 16, 2020, 8:48pm

Hmm. I don’t think those installation instructions should be suggesting setting up a separate renewal cron. I believe the certbot package from EPEL comes with renewal tasks already:

systemctl status certbot-renew.service
journalctl -u certbot-renew.service

Do those show that Certbot has been running on its own?

You might also want to consider going through some of the recent files in /var/log/letsencrypt/, find the logs from a couple of days ago, they should document why renewal failures have been occurring, if any.

mnordhoff · May 16, 2020, 8:48pm

What does “sudo certbot certificates” show?

ccheath · May 16, 2020, 8:49pm

$sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: server.eyethrees.net
    Domains: server.eyethrees.net alatrist.com blog.chrisheath.us chrisheath.us dominusbrand.com eyethrees.net hotttsun.com
    Expiry Date: 2020-06-21 23:08:10+00:00 (VALID: 36 days)
    Certificate Path: /etc/letsencrypt/live/server.eyethrees.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/server.eyethrees.net/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

ccheath · May 16, 2020, 8:51pm

$sudo systemctl status certbot-renew
● certbot-renew.service - This service automatically renews any certbot certificates found
   Loaded: loaded (/usr/lib/systemd/system/certbot-renew.service; static; vendor preset: disabled)
   Active: inactive (dead)

$sudo journalctl -u certbot-renew
-- No entries --

ccheath · May 16, 2020, 8:54pm

$sudo cat /var/log/letsencrypt/letsencrypt.log.1
2020-05-16 12:51:16,023:DEBUG:certbot._internal.main:certbot version: 1.3.0
2020-05-16 12:51:16,024:DEBUG:certbot._internal.main:Arguments: ['-q']
2020-05-16 12:51:16,024:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2020-05-16 12:51:16,068:DEBUG:certbot._internal.log:Root logging level set at 30
2020-05-16 12:51:16,068:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2020-05-16 12:51:16,087:DEBUG:certbot._internal.plugins.selection:Requested authenticator <certbot._internal.cli.cli_utils._Default object at 0x7f16663e8e90> and installer <certbot._internal.cli.cli_utils._Default object at 0x7f16663e8e90>
2020-05-16 12:51:16,149:DEBUG:certbot.ocsp:Querying OCSP for /etc/letsencrypt/archive/server.eyethrees.net/cert10.pem
2020-05-16 12:51:16,149:DEBUG:certbot.ocsp:openssl ocsp -no_nonce -issuer /etc/letsencrypt/archive/server.eyethrees.net/chain10.pem -cert /etc/letsencrypt/archive/server.eyethrees.net/cert10.pem -url http://ocsp.int-x3.letsencrypt.org -CAfile /etc/letsencrypt/archive/server.eyethrees.net/chain10.pem -verify_other /etc/letsencrypt/archive/server.eyethrees.net/chain10.pem -trust_other -header Host ocsp.int-x3.letsencrypt.org
2020-05-16 12:51:16,252:INFO:certbot._internal.renewal:Cert not yet due for renewal
2020-05-16 12:51:16,253:DEBUG:certbot._internal.plugins.selection:Requested authenticator webroot and installer None
2020-05-16 12:51:16,254:DEBUG:certbot._internal.renewal:no renewal failures

ccheath · May 16, 2020, 8:55pm

so is the log saying that it’s not due for renew yet mean that it is checking and it will renew just in time and i can disregard the emails?

ps: going for a walk… check back in 20-30

_az · May 16, 2020, 8:58pm

Yes, I don't think anything is wrong here. Your certificate will renew in a ~week.

You removed a bunch of domain names from your certificate a while ago, right? The www. subdomains specifically.

Let's Encrypt is emailing you because of the reason documented here: Expiration Emails - Let's Encrypt :

If you’ve issued a new certificate that adds or removes a name relative to your old certificate, you will get expiration email about your old certificate

Sorry for the false alarm with the systemd timer - TIL it comes disabled by default.

ccheath · May 16, 2020, 9:21pm

thanks

yes i was initially doing the www’s also but the last time I renewed i guess it didn’t do them

that’s fine, right?
i have a no-www thing going on in my httpd conf

ccheath · May 24, 2020, 2:14pm

just a follow up:

you are correct that the email expiration warning is for my old certificate
certbot certificates shows my current cert (without the www domains) valid for another 4 weeks
thanks for the help

ccheath · June 12, 2020, 12:35am

hey @_az … so my domains are 9 days from expiration for real now… shouldn’t they have renewed already?

$sudo certbot certificates
[sudo] password for cch: 
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Found the following certs:
  Certificate Name: server.eyethrees.net
    Serial Number: 360f6427aee493a15aead75dcd153fbf72c
    Domains: server.eyethrees.net alatrist.com blog.chrisheath.us chrisheath.us dominusbrand.com eyethrees.net hotttsun.com
    Expiry Date: 2020-06-21 23:08:10+00:00 (VALID: 9 days)
    Certificate Path: /etc/letsencrypt/live/server.eyethrees.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/server.eyethrees.net/privkey.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

and

$cat /etc/crontab 
SHELL=/bin/bash
PATH=/sbin:/bin:/usr/sbin:/usr/bin
MAILTO=root

# For details see man 4 crontabs

# Example of job definition:
# .---------------- minute (0 - 59)
# |  .------------- hour (0 - 23)
# |  |  .---------- day of month (1 - 31)
# |  |  |  .------- month (1 - 12) OR jan,feb,mar,apr ...
# |  |  |  |  .---- day of week (0 - 6) (Sunday=0 or 7) OR sun,mon,tue,wed,thu,fri,sat
# |  |  |  |  |
# *  *  *  *  * user-name  command to be executed

0 0,12 * * * root python -c 'import random; import time; time.sleep(random.random() * 3600)' && certbot renew -q

_az · June 12, 2020, 12:48am

Yes. Take a look in /var/log/letsencrypt/ through some of the recent logs.

Some of them (maybe letsencrypt.log or letsencrypt.log.1 or something) should have tried to renew your certificate. If something went wrong, the reason for the failure should be there.

ccheath · June 12, 2020, 2:01am

Lots of "invalid response from" exactly like

not sure their solution of a 'rouge www ServerAlias" applies to me, but maybe...

one thing that i think might help is that the log seems to create the .well-known validation challenges all under server.eyethrees.net but then the file not founds are all .well-known directories under each vhost's domain

my 'webroot path' in /etc/letsencrypt/renewal/server.eyethrees.net.conf is /var/www/html/server.eyethrees.net,
is the comma supposed to be there? is the webroot supposed to be just /var/www/html?

thanks for all your help, btw

_az · June 12, 2020, 2:05am

Could you post the entire .conf file?

It looks like your domains all have distinct webroots … so the renewal configuration would need to reflect that, in order to be able to renew your certificate.

ccheath · June 12, 2020, 2:16am

$cat /etc/letsencrypt/renewal/server.eyethrees.net.conf 
# renew_before_expiry = 30 days
version = 1.0.0
archive_dir = /etc/letsencrypt/archive/server.eyethrees.net
cert = /etc/letsencrypt/live/server.eyethrees.net/cert.pem
privkey = /etc/letsencrypt/live/server.eyethrees.net/privkey.pem
chain = /etc/letsencrypt/live/server.eyethrees.net/chain.pem
fullchain = /etc/letsencrypt/live/server.eyethrees.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
account = 89a1d03d755d103b12422eb24a811c26
server = https://acme-v02.api.letsencrypt.org/directory
manual_public_ip_logging_ok = True
rsa_key_size = 2048
webroot_path = /var/www/html/server.eyethrees.net,
[[webroot_map]]

… thanks

_az · June 12, 2020, 10:07am

Sorry for taking a while to respond, got busy.

Usually when using the webroot plugin is used with multiple distinct domains, you would specify each domain’s webroot separately. The resulting renewal config file would look something like:

webroot_path = /tmp/defaultwebroot, /tmp/www/html/example.org, /tmp/example.com/whatever, /tmp/blahblah
[[webroot_map]]
example.org = /tmp/defaultwebroot
example.com = /tmp/www/html/example.org
example.foo = /tmp/example.com/whatever

Otherwise, as in your case, it would try to authenticate each domain using the same /var/www/html/server.eyethrees.net webroot, which is probably not right. The result would be the unauthorized errors you saw.

Since you are using Apache anyway, perhaps consider trying using the Apache plugin as your authenticator. In general, it does things a lot more “automagically” than the webroot plugin.

You could test it out without any commitment with:

certbot renew --cert-name server.eyethrees.net -a apache --dry-run

Otherwise, I think you’d need to fix up your webroot config.

ccheath · June 13, 2020, 12:36am

@_az, thanks for the suggestion!

the dry run went fine… so running it without the --dry-run at the end will switch me over from the webroot config and nothing else needs to be done?

_az · June 13, 2020, 12:38am

Yes.

You could also modify the renewal config to authenticator = apache (leaving everything else the same), and wait for the next cron invocation.

ccheath · June 13, 2020, 12:51am

You could also modify the renewal config to authenticator = apache (leaving everything else the same), and wait for the next cron invocation.

i'll do that & report back later tonight or tomorrow

Thanks!

ccheath · June 13, 2020, 6:48pm

looks like I’m finally good to go… renewed last night

THANK YOU @_az