Crontab Certificate Renewal is missed

[4m3ndy]

Hello Everyone,
I'd like to discuss my problem with you and I have been facing this on multiple servers.

Certificate Renewal is being missed and all the configurations are correct, and when I run the dry run test, it runs successfully, however when I trace the let's encrypt log I get no renewal, and the certificate is due to renewal as it's 30 days before expiration date.

My Environment:

• OS: CentOS 7
• WebServer: nginx

This is how I obtained the certificate:

certbot certonly --webroot -w /var/www/html -d jobs.espace.com.eg -d jobs.espace-technologies.com

I do the certificate renewal through crontab

0 4 * * 0 /bin/certbot renew --post-hook "nginx -s reload" >> /var/log/letsencrypt-renew.log

and Here's the certificate renewal history of the certificate:
crt.sh | jobs.espace.com.eg

The second renewal done successfully which is crt.sh | 132764979, while the third renewal was not attempted crt.sh | 182708178, so I had to renew the certificate myself manually after the expiration of the certificate.

Here's my configurations

version = 0.8.1
cert = /etc/letsencrypt/live/jobs.espace-technologies.com/cert.pem
privkey = /etc/letsencrypt/live/jobs.espace-technologies.com/privkey.pem
chain = /etc/letsencrypt/live/jobs.espace-technologies.com/chain.pem
fullchain = /etc/letsencrypt/live/jobs.espace-technologies.com/fullchain.pem
[renewalparams]
authenticator = webroot
installer = None

and this is the output of the --dry-run test:

Processing /etc/letsencrypt/renewal/jobs.espace-technologies.com.conf
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/jobs.espace-technologies.com/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)

[jared.m]

Could you post the logs where it’s failing?

[4m3ndy]

No Failures happened, and this is the when list the files the /var/log/letsencrypt directory

There's no log file created within jun 16 and 2 august, which means that the cronjob did not run within this period

The Jun 16 log file is an attempt that was not due to renewal
The Aug 2 log file is my manual attempt to renew the certificate

drwx------. 2 root root 4.0K Aug 9 17:35 .
drwxr-xr-x. 11 root root 4.0K Aug 6 03:39 ..
-rw-r--r--. 1 root root 4.7K Aug 9 17:36 letsencrypt.log
-rw-r--r--. 1 root root 70K Aug 9 17:08 letsencrypt.log.1
-rw-r--r--. 1 root root 4.6K May 14 04:00 letsencrypt.log.10
-rw-r--r--. 1 root root 64K Aug 2 15:27 letsencrypt.log.2
-rw-r--r--. 1 root root 64K Aug 2 15:21 letsencrypt.log.3
-rw-r--r--. 1 root root 4.6K Jun 16 03:20 letsencrypt.log.4
-rw-r--r--. 1 root root 64K Jun 16 03:20 letsencrypt.log.5
-rw-r--r--. 1 root root 4.6K Jun 11 04:00 letsencrypt.log.6
-rw-r--r--. 1 root root 4.6K Jun 4 04:00 letsencrypt.log.7
-rw-r--r--. 1 root root 4.6K May 28 04:00 letsencrypt.log.8
-rw-r--r--. 1 root root 4.6K May 21 04:00 letsencrypt.log.9

[jared.m]

First note is that it’s recommended to run renewal twice daily, no weekly. Second, if it’s not actually running, you should probably try and figure out why your cron isn’t executing commands as expected. What’s the August 9th logs? Are those also only manual renewals? You should see log entries from certbot every time it runs, indicating that a renewal was or was not attempted and the outcome if it was attempted.

[rg305]

I would try adding webroot to the renewal process:
/bin/certbot renew --webroot -w /var/www/html --post-hook “nginx -s reload” >> /var/log/letsencrypt-renew.log

Also, webroot uses http and, unless you are bypassing the acme-challenges, one of the sites is re-re-re-directing the http (to https then http then https):
wget jobs.espace.com.eg
–2017-08-09 12:14:47-- http://jobs.espace.com.eg/
Resolving jobs.espace.com.eg (jobs.espace.com.eg)… 138.201.180.85
Connecting to jobs.espace.com.eg (jobs.espace.com.eg)|138.201.180.85|:80… connected.
HTTP request sent, awaiting response… 301 Moved Permanently
Location: https://jobs.espace.com.eg/ [following]
–2017-08-09 12:14:48-- https://jobs.espace.com.eg/
Connecting to jobs.espace.com.eg (jobs.espace.com.eg)|138.201.180.85|:443… connected.
HTTP request sent, awaiting response… 302 Moved Temporarily
Location: http://espace.com.eg/company/jobs [following]
–2017-08-09 12:14:48-- http://espace.com.eg/company/jobs
Resolving espace.com.eg (espace.com.eg)… 97.107.136.109
Connecting to espace.com.eg (espace.com.eg)|97.107.136.109|:80… connected.
HTTP request sent, awaiting response… 302 Moved Temporarily
Location: https://espace.com.eg/company/jobs [following]
–2017-08-09 12:14:48-- https://espace.com.eg/company/jobs
Connecting to espace.com.eg (espace.com.eg)|97.107.136.109|:443… connected.
HTTP request sent, awaiting response… 200 OK

[jared.m]

Adding --webroot is unnecessary and not best practice in general. Renewal uses the same authorization method in use for issuance - this would be webroot in this case.

Webroot can follow https redirects - that’s not an issue.

[rg305]

But the redirect leads to a completely different domain (and IP).
Could be problematic if not considered in the design.

[jared.m]

Good point - noticed the domains, but the IP may be problematic.

[rg305]

Something has changed since the last working renewal.
Maybe the redirections.
Maybe the webroot folder was moved.
Maybe the acme-challenge alias was changed.
…

[schoen]

@jared.m

Adding --webroot is unnecessary and not best practice in general. Renewal uses the same authorization method in use for issuance - this would be webroot in this case.

I want to point out by way of agreement with this concern that certbot renew is designed to work in environments where you have perhaps dozens of different certificates which are obtained by all different authentication methods. In that case, specifying authentication information related to just one of the certificates would be a recipe for disaster (failed renewals) because it would be used for various certificates that it didn't actually apply to.

The intended design is indeed to specify the authentication parameters with certonly or run, and then have Certbot remember them for the future and supply them automatically with renew. If you need to change them, the recommended way is via a single (potentially --force-renewal) renewal with certonly and appropriate parameters, which should be saved.

[4m3ndy]

Here's my nginx configuration:

server {
listen 80 default_server;
server_name jobs.espace-technologies.com jobs.espace.com.eg;
return 301 https://$host$request_uri;
}
server {
listen 443 default_server;
server_name jobs.espace-technologies.com jobs.espace.com.eg;
ssl on;
ssl_certificate /etc/letsencrypt/live/jobs.espace-technologies.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/jobs.espace-technologies.com/privkey.pem;

    location =/{
    rewrite (.*) http://espace.com.eg/company/jobs redirect;
    }

    location ~ /.well-known {
            root /var/www/html;
            allow all;
    }

}

according to these configuration, we want to redirect the root location to go to espace.com.eg/jobs, but when it requires .well-known locations, it servers over https.

The webroot location wasn't changed, and the as you can see the configurations are correct.

[4m3ndy]

so This mean that It’s better to use certbot renew with out --webroot, did I get this right ?

[4m3ndy]

@jared.m

What’s the August 9th logs?

Yes do those a --dry-run test by me, just to make sure the configuration was right, and when I traced the log /var/log/letsencrypt/letsencrypt.log, the renewal dry-run was successful.

@jared.m

First note is that it’s recommended to run renewal twice daily

I'll try to update my cronjob with this, but the execution of the renewal cron during the 30 days before expiration wasn't unreasonable, that's why I'm trying to debug the environment, and I can see all the configurations are right till now.

[rg305]

try placing a test file at:
/var/www/html/acme-challenge/test.txt
and see if it is accessible via the Internet

[schoen]

Maybe also /var/www/html/.well-known/acme-challenge/test.txt for greater realism.

[rg305]

@4m3ndy

location ~ /.well-known {
root /var/www/html;

Doesn't that essentially remove the "/.well-known" portion form the URL?
mapping
http://fqdn/.well-known
to
/var/www/html
and
http://fqdn/.well-known/acme-challenge/test.txt
to
/var/www/html/acme-challenge/test.txt

[schoen]

We’ve definitely had people have trouble with that issue before. It might be more correct with root /var/www/html/.well-known in this case.

[4m3ndy]

@rg305

acme-challenge/test.txt

When /.well-known is requested by the acme challenge, it's created, and I can access secret.txt, so I think the root is fine

I have another question, If the post-hook did not successfully run, will the certificate be issued ?
as I got this in the log file

Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/jobs.espace-technologies.com/fullchain.pem (success)
Unable to find post-hook command nginx in the PATH.
(PATH is /usr/bin:/bin)

[rg305]

@4m3ndy

If the post-hook did not successfully run, will the certificate be issued ?

Yes, the cert was issued (renewed) and can be found at

@4m3ndy

/etc/letsencrypt/live/jobs.espace-technologies.com/

The post-hook simply restarts nginx to ensure the latest cert is in use.

I see that you have managed to get and use the latest cert:

crt.sh | 182708178