Automatic Renewal of certifications: what's the correct procedure?

marcoippolito · July 19, 2019, 3:29pm

I received this email:

"
Hello,

Your certificate (or certificates) for the names listed below will expire in 20 days (on 08 Aug 19 09:50 +0000). Please make sure to renew your certificate before then, or visitors to your website will encounter errors.

We recommend renewing certificates automatically when they have a third of their
total lifetime left. For Let’s Encrypt’s current 90-day certificates, that means
renewing 30 days before expiration

ggc.world
www.ggc.world
"

The version of my client is :

Ubuntu 18.04.01

marco@pc01:~$ certbot --version
certbot 0.31.0

I tried to renew the certificates in this way:

marco@pc01:~$ sudo certbot renew -d ggc.world -d www.ggc.world 
/etc/logrotate.d/certbot
usage: 
  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. 
certbot: error: unrecognized arguments: /etc/logrotate.d/certbot

In /etc/nginx/conf.d/default.conf file

server {
    listen 443 http2 default_server;
    server_name ggc.world www.ggc.world;
    ssl on;
    ssl_certificate /etc/ssl/certs/chained.pem;
    ssl_certificate_key /etc/ssl/private/domain.key;

This is the content of /etc/cron.d/certbot :

SHELL=/bin/sh
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin

0 */12 * * * root test -x /usr/bin/certbot -a \! -d /run/systemd/system && perl -e 
'sleep int(rand(43200))' && certbot -q renew

What’s the correct procedure to automatically renew the certificates?

Marco

JuergenAuer · July 19, 2019, 3:35pm

Hi @marcoippolito

use only

without a domain name. But your cron has already the correct command.

So it looks that your renew didn't work.

There is a check of your domain - some days old. Looks bad, two different ip addresses with different answers, timeout and blocking firewall.

What's the output of the command?

PS: Rechecked your domain - https://check-your-website.server-daten.de/?q=ggc.world

Now there is only one ip address. But http is blocked - you have to change that.

EmberHeartshine · July 19, 2019, 3:48pm

These two commands need to be on their own. (And usually certbot is run as root, but I don't know your setup so you may not need privilege elevation.)

marco@pc01:~$ sudo certbot renew
marco@pc01:~$ sudo logrotate -f certbot

If they must be on the same line, e.g. for cron, you can use the && construct (or a semicolon if you want the logs to rotate whether or not certbot successfully renews your certs) to run them on the same line.

marco@pc01:~$ sudo certbot renew && sudo logrotate -f certbot

Your cron job isn't running because Ubuntu uses systemd as its init system. (Take a peek at the comments in /etc/cron.d/certbot.) Also, there isn't a root command available for Ubuntu (did a quick search on the Ubuntu Packages site and I can't see anything that provides a root executable).

You'll have to manually create the cron job yourself by using sudo cron and entering in something like:

0 */12 * * * sleep $(awk 'BEGIN{;srand();print int(1+rand()*30);}')m;certbot renew && logrotate -f certbot

(The above must be entered into root's crontab!)

marcoippolito · July 19, 2019, 3:54pm

Hi @JuergenAuer.
Thanks for replying promptly.

I just checked my domain https://check-your-website.server-daten.de/?q=ggc.world
and it seems ok, apart from the expiration dates of the certificates.

How to unblock the http?

This is the entire /etc/nginx/conf.d/default.conf file, which hasn’t changed since when I got the certificates:

server {
    listen 443 http2 default_server;
    server_name ggc.world www.ggc.world;

    ssl on;
    ssl_certificate /etc/ssl/certs/chained.pem;
    ssl_certificate_key /etc/ssl/private/domain.key;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20-

draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DE
S:RSA+3DES:!MD5;
ssl_prefer_server_ciphers on;
ssl_session_cache shared:SSL:50m;
ssl_dhparam /etc/ssl/certs/dhparam.pem;
ssl_stapling on;
ssl_stapling_verify on;

    access_log /var/log/nginx/ggcworld-access.log combined;

   add_header Strict-Transport-Security "max-age=31536000";
   location = /favicon.ico { access_log off; log_not_found off; }
    location / {
        proxy_pass http://192.168.1.7:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

server {
    #listen 80 default server;
    listen [::]:80 default_server;
    server_name ggc.world www.ggc.world;
    return 443 https://$host$request_uri;

    access_log /var/log/nginx/ggcworld-access.log combined;

    add_header Strict-Transport-Security "max-age=31536000";
    location = /favicon.ico { access_log off; log_not_found off; }
    location / {
        proxy_pass http://192.168.1.7:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}


marco@pc01:~$ sudo certbot renew
[sudo] password for marco: 
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

No renewals were attempted.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

In /var/log/letsencrypt/letsencrypt.log :

2019-07-19 15:49:00,679:DEBUG:certbot.renewal:no renewal failures
2019-07-19 15:49:58,657:DEBUG:certbot.main:certbot version: 0.31.0

JuergenAuer · July 19, 2019, 3:59pm

It's your router or your firewall, not your webserver configuration.

Where do you see a working http / port 80?

That's (your new check) bad:

Domainname	Http-Status	Sec.	G
• http://ggc.world/
109.116.168.162	-2	1.097	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 109.116.168.162:80

• http://www.ggc.world/
109.116.168.162	-2	1.100	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 109.116.168.162:80

• https://ggc.world/
109.116.168.162	200	0.340	I
small content:


• https://www.ggc.world/
109.116.168.162	200	0.357	I
small content:


• http://ggc.world/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
109.116.168.162	-2	1.087	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 109.116.168.162:80
Visible Content:

• http://www.ggc.world/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
109.116.168.162	-2	1.100	V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 109.116.168.162:80

Connect failure -> http doesn't work.

Compare it with other domains (Grade E and better).

EmberHeartshine · July 19, 2019, 4:02pm

Looks like HTTPS works fine. You may want to correct your redirect block:

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name ggc.world www.ggc.world;
    return 301 https://$host$request_uri;
}

I disagree with @JuergenAuer, I don’t think this is related to a firewall. You just don’t have anything listening on port 80 with the config above.

JuergenAuer · July 19, 2019, 4:09pm

There

is a long topic. There you have used dns-01 validation, so port 80 isn't relevant.

So do that again, perhaps with the --manual - option.

But if you use --manual, you can't use an automatic cron job.

JuergenAuer · July 19, 2019, 4:18pm

Maybe correct.

@marcoippolito : Does your http work internal?

curl http://ggc.world/

from your webserver?

marcoippolito · July 19, 2019, 4:19pm

In /etc/nginx/conf.d/default.conf in the redirect block I modified the line:

return 443 https://$host$request_uri;

to

return 301 https://$host$request_uri;

I also checked the firewalls:

in my PC-WebServer:

marco@pc01:/etc/nginx/conf.d$ sudo ufw status
Status: inactive

and in the Internet Service Provider (Vodafone):

But still when checking https://check-your-website.server-daten.de/?q=ggc.world
Http-Status: -2

EmberHeartshine · July 19, 2019, 4:20pm

You also have to uncomment the listen line. Change

#listen 80 default server;

to

listen 80 default_server;

Don’t forget to reapply your config!

marco@pc01:~$ sudo nginx -t && sudo service nginx reload

marcoippolito · July 19, 2019, 4:24pm

Thanks a lot for your kind explanation.
I’m following now these hints to setup cron job on Ubuntu 18.04 : https://linux4one.com/how-to-set-cron-jobs-on-ubuntu-18-04/

EmberHeartshine · July 19, 2019, 4:27pm

If it's easier, I wrote a little bit about that here.

Quoted below:

EmberHeartshine:

You’ll have to manually create the cron job yourself by using sudo cron and entering in something like:
0 */12 * * * sleep $(awk 'BEGIN{;srand();print int(1+rand()*30);}')m;certbot renew && logrotate -f certbot
(The above must be entered into root 's crontab ! )

The above will run the update command every 12 hours (randomized between +1 and +30 minutes to save the servers a little headache) and update if needed. If there's an error, the logs won't rotate, and if the renewal succeeds, logs will be rotated.

marcoippolito · July 19, 2019, 4:30pm

marco@pc01:/etc/nginx/conf.d$ sudo nginx -t && sudo service nginx reload
nginx: [warn] the “ssl” directive is deprecated, use the “listen … ssl” directive instead in /etc/nginx/conf.d/default.conf:5
nginx: [emerg] invalid parameter “server” in /etc/nginx/conf.d/default.conf:30
nginx: configuration file /etc/nginx/nginx.conf test failed

line 30 : listen 80 default server;

EmberHeartshine · July 19, 2019, 4:32pm

You can remove the ssl on; line from your SSL config, and instead add it to the listen line:

listen 443 ssl http2 default_server;

Change default server to default_server. The underscore is important.

marcoippolito · July 19, 2019, 4:40pm

Now with the resulting /etc/nginx/conf.d/default.conf file :

server {
    listen 443 ssl http2 default_server;
    server_name ggc.world www.ggc.world;

    ssl_certificate /etc/ssl/certs/chained.pem;
    ssl_certificate_key /etc/ssl/private/domain.key;
    ssl_session_timeout 5m;
    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3;
    ssl_ciphers EECDH+CHACHA20:EECDH+CHACHA20- 
draft:EECDH+AES128:RSA+AES128:EECDH+AES256:RSA+AES256:EECDH+3DES:RSA+3DES:!MD5;
    ssl_prefer_server_ciphers on;
    ssl_session_cache shared:SSL:50m;
    ssl_dhparam /etc/ssl/certs/dhparam.pem;
    ssl_stapling on;
    ssl_stapling_verify on;

    access_log /var/log/nginx/ggcworld-access.log combined;

    add_header Strict-Transport-Security "max-age=31536000";
    location = /favicon.ico { access_log off; log_not_found off; }
    location / {
        proxy_pass http://192.168.1.7:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

server {
    listen 80 default_server;
    listen [::]:80 default_server;
    server_name ggc.world www.ggc.world;
    #return 443 https://$host$request_uri;
    return 301 https://$host$request_uri;

    access_log /var/log/nginx/ggcworld-access.log combined;

    add_header Strict-Transport-Security "max-age=31536000";
    location = /favicon.ico { access_log off; log_not_found off; }
    location / {
        proxy_pass http://192.168.1.7:8080;
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
    }
}

marco@pc01:/etc/nginx/conf.d$ sudo nginx -t && sudo service nginx reload
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

Checking with https://check-your-website.server-daten.de/?q=ggc.world

Http-Status : 301

Now the only thing to fix is the auto-renewal of certificates with sudo chron

EmberHeartshine · July 19, 2019, 4:47pm

Since you’re redirecting all traffic to HTTPS, you can remove these lines from your HTTP server (the one listening on 80):

add_header Strict-Transport-Security "max-age=31536000";
location = /favicon.ico { access_log off; log_not_found off; }
location / {
    proxy_pass http://192.168.1.7:8080;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection "upgrade";
}

Since you’re pushing traffic to the other server, headers never get sent and your proxy is never used. I don’t think it’ll actually cause any problems as-is, but it just looks cleaner and it’s easier to debug if things are as simple as possible. You don’t need to do that if you don’t want to.

marcoippolito · July 19, 2019, 4:55pm

Actually I do want to redirect all traffic to https.
Removing those lines -> HTTP_STATUS : 301

Now … the cron job…

JuergenAuer · July 19, 2019, 4:56pm

Great, a working port 80

Try

sudo certbot --nginx -d ggc.world -d www.ggc.world

to see, if that works. If yes, that changes your config file, so you can use renew.

PS: No, not the cron job. First the new certificate creation.

marcoippolito · July 19, 2019, 4:58pm

@EmberHeartshine

marco@pc01:/etc/cron.d$ su -
Password: 
root@pc01:~# sudo cron 0 */12 * * * sleep $(awk 'BEGIN{;srand();print    
int(1+rand()*30);}')m;certbot renew && logrotate -f certbot
cron: can't lock /var/run/crond.pid, otherpid may be 942: Resource temporarily 
unavailable
 Saving debug log to /var/log/letsencrypt/letsencrypt.log

 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

 No renewals were attempted.
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
 error: cannot stat certbot: No such file or directory
 root@pc01:~#

What do you mean as “entering into root’s crontab”?

marcoippolito · July 19, 2019, 5:16pm

@JuergenAuer
Unfortunately something again has to be fixed:

marco@pc01:/etc/nginx/conf.d$ sudo certbot --nginx -d ggc.world -d 
www.ggc.world
[sudo] password for marco: 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
The requested nginx plugin does not appear to be installed
marco@pc01:/etc/nginx/conf.d$