Update cert on NixOS / installing certbot

notdesmond · April 12, 2018, 5:58am

Hi there,

A vendor setup our NixOS for us and now the certificate is expiring. I have 2 question I need to ask:

How can I renew my certs? Since we are using NixOS. I’m not sure how to install certbot on NixOS and renew the certs?
How can I redo our certs, so that the email for cert expiry will be emailed to me?

Thanks.
Desmond

My domain is: portal.docdoc.com

I ran this command: sudo certbot renew

It produced this output: command not found (I don’t have the certbot command on the system)

My web server is (include version): nginx (don’t know which version)

The operating system my web server runs on is (include version): NixOS (don’t know which version)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

_az · April 12, 2018, 6:24am

https://nixos.org/nixos/manual/index.html#module-security-acme

That uses simpl_le under the hood, but other clients like Certbot and Lego are available from the Nix package manager. You should try the above though, since it is actually documented.

I would take care that you do things in the “NixOS way”, as it’s not quite the same as using regular Linux, and therefore the usual recommendations on how to use Certbot and other ACME clients may not be suitable for you specifically.

notdesmond · April 12, 2018, 6:45am

Hi _az,

Thanks for the help.

Yup, I’m trying to do it the NixOS way but I’m stuck. The NixOS is configured correctly. It seem that the simpl_le is broken (not sure how). Is that a way or a command for me to check this? Can I renew the certs manually? If simpl_le is broken, how can I fix it?

Thanks.

_az · April 12, 2018, 7:01am

Well, there is clear evidence that your nginx is not configured in the way that is suggested in the NixOS documentation.

They suggest that you need to have a special rule for /.well-known/acme-challenge on your port 80 virtual host:

# (from the NixOS documentation)
listen 80;
listen [::]:80;

location /.well-known/acme-challenge {
  root /var/www/challenges;
}

However, we see that your nginx server is missing this stanza in the port 80 server, because we receive an HTTPS redirect (and we shouldn't if the above was present):

$ curl -ikL portal.docdoc.com/.well-known/acme-challenge/xxx
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.1
Date: Thu, 12 Apr 2018 06:58:26 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://portal.docdoc.com/.well-known/acme-challenge/xxx

notdesmond · April 12, 2018, 7:14am

We have these codes:

    server {
      server_name portal.docdoc.com;
      listen 80;
      return 301 https://$server_name$request_uri;

      location /.well-known {
        root /var/www/acme/portal.docdoc.com;
      }
    }

Do I just add it in like this:

    server {
      server_name portal.docdoc.com;
      listen 80;
      return 301 https://$server_name$request_uri;

      location /.well-known {
        root /var/www/acme/portal.docdoc.com;
      }
    
      location /.well-known/acme-challenge {
        root /var/www/challenges;
      }
    }

But on the NixOS server there is no challenges directory.

_az · April 12, 2018, 7:16am

That is the problem . You have a server-wide redirect that is preventing the location stanza from doing anything.

Try (after getting rid of the existing return line):

  location / {
    return 301 https://$server_name$request_uri;
  }
  location /.well-known {
    root /var/www/acme/portal.docdoc.com;
  }

I'm not too sure about the ACME path, safest to try what your previous server administrator setup, and then repair it if it does not work. Best to check what your configuration.nix file says under security.acme.certs.

notdesmond · April 12, 2018, 7:23am

The security.acme.certs shows this webroot = "/var/www/acme/portal.docdoc.com";

So, after the update, how would I know if the cert will be renewed? It will expire on 14 April.

_az · April 12, 2018, 7:28am

In that case

location /.well-known {

should be

location /.well-known/acme-challenge {

Did you reload nginx? Your server is still doing the bad redirect.

I think you can force the renewal attempt by running

systemctl start acme-portal.docdoc.com

(However, not sure, never used NixOS).

notdesmond · April 12, 2018, 7:31am

Okay, I’m changing the location and am going to upload the settings to NixOS.

Then I’ll try the reload. Hopefully it works.

_az · April 12, 2018, 7:34am

I should add that you should be able to check the logs for the renewal using

journalctl -u acme-portal.docdoc.com

It might be worth looking at before you go trying again.

notdesmond · April 12, 2018, 7:46am

I think it worked and renewed. Not sure if I read the logs correctly.

This is from the log:

Apr 12 07:43:05 www systemd[1]: Starting Renew ACME Certificate for portal.docdoc.com... Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,111:DEBUG:simp_le:1371: ['-v', '-d', 'portal.docdoc.com', '--de Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,112:DEBUG:simp_le:367: Loading fullchain.pem Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,112:DEBUG:simp_le:367: Loading full.pem Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,113:DEBUG:simp_le:367: Loading key.pem Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,113:DEBUG:simp_le:367: Loading account_key.json Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,147:DEBUG:simp_le:1203: Existing SANs: [u'portal.docdoc.com'], Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,149:DEBUG:simp_le:1068: Certificate expires in 89 days, 22:19:5 Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,149:INFO:simp_le:1383: Certificates already exist and renewal i Apr 12 07:43:06 www systemd[1]: Started Renew ACME Certificate for portal.docdoc.com.

_az · April 12, 2018, 7:54am

So, the certificate DID renew: https://crt.sh/?id=392777144 (at least, I’m pretty sure)

However, your nginx server is still using an old certificate:

$ openssl s_client -connect portal.docdoc.com:443 -servername portal.docdoc.cm -showcerts 2>/dev/null | openssl x509 -noout -dates
notBefore=Jan 14 23:53:06 2018 GMT
notAfter=Apr 14 23:53:06 2018 GMT

You may need to reload nginx (though it should have done that by itself), also check what ssl_certificate your nginx config is pointing to, it may be pointing to a copy of the certificate that was not automatically updated by NixOS/simp_le.

notdesmond · April 12, 2018, 8:04am

You are talking about this correct?

server {
          server_name portal.docdoc.com;
          listen 443 ssl http2;
          ssl_certificate /var/lib/acme/portal.docdoc.com/full.pem;
          ssl_certificate_key /var/lib/acme/portal.docdoc.com/key.pem;
          ssl_dhparam /var/lib/acme/portal.docdoc.com/dhparam.pem;

          client_body_buffer_size 100m;

          location / {
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_pass http://localhost:8080/;
          }
        }

It is pointing to the correct files as the files have been changed. I’ve restarted nginx but it still have the same dates.

_az · April 12, 2018, 8:07am

What’s the output of this:

ls -la /var/lib/acme/portal.docdoc.com
openssl x509 -in /var/lib/acme/portal.docdoc.com/full.pem -noout -dates
openssl x509 -in /var/lib/acme/portal.docdoc.com/fullchain.pem -noout -dates

notdesmond · April 12, 2018, 8:11am

[desmond@www:~]$ sudo ls -la /var/lib/acme/portal.docdoc.com
total 36
drwx------ 2 root root 4096 Sep  7  2017 .
drwxr-xr-x 6 root root 4096 Sep  6  2017 ..
-rw-r--r-- 1 root root 3169 Sep  7  2017 account_key.json
-rw-r--r-- 1 root root  424 Sep  6  2017 dhparam.pem
-rwx------ 1 root root 4150 Apr 12 07:36 fullchain.pem
-rwx------ 1 root root 7422 Apr 12 07:36 full.pem
-rwx------ 1 root root 3271 Apr 12 07:36 key.pem


[desmond@www:~]$ sudo openssl x509 -in /var/lib/acme/portal.docdoc.com/full.pem -noout -dates
notBefore=Apr 12 06:36:18 2018 GMT
notAfter=Jul 11 06:36:18 2018 GMT


[desmond@www:~]$ sudo openssl x509 -in /var/lib/acme/portal.docdoc.com/fullchain.pem -noout -dates
notBefore=Apr 12 06:36:18 2018 GMT
notAfter=Jul 11 06:36:18 2018 GMT

_az · April 12, 2018, 8:13am

Edit: OK, so I had a typo in my previous openssl command.

-servername portal.docdoc.cm -

Whoops, should have been

-servername portal.docdoc.com

When we connect with the right arguments, we get the updated certificate, happily:

notBefore=Apr 12 06:36:18 2018 GMT
notAfter=Jul 11 06:36:18 2018 GMT

When connecting to an unknown SNI name (such as portal.docdoc.cm, we get the certificate for portalsandbox.docdoc.com, which is not renewed yet).

Problem solved, I think

notdesmond · April 12, 2018, 8:17am

[desmond@www:~]$ sudo systemctl restart nginx

[desmond@www:~]$ systemctl status nginx
● nginx.service - Nginx Web Server
   Loaded: loaded (/nix/store/l0b34js1499aij6a20zacmcfb6wy6n38-unit-nginx.service/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-04-12 08:16:10 UTC; 3s ago
  Process: 1328 ExecStartPre=/nix/store/lv71lq5izg78bzdg52irm-unit-script/bin/nginx-pre-start (code=exited, status=0/SUCCESS)
 Main PID: 1332 (nginx)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─1332 nginx: master process /nix/store/lisnkdnh24cfaz3j5zb6p-nginx-1.12.1/bin/nginx -c /nix/store/403bfqqpm09n
           └─1336 nginx: worker process

Apr 12 08:16:10 www systemd[1]: Starting Nginx Web Server...
Apr 12 08:16:10 www systemd[1]: Started Nginx Web Server.

[desmond@www:~]$ date
Thu Apr 12 08:16:30 UTC 2018

_az · April 12, 2018, 8:17am

My bad, see my previous post

notdesmond · April 12, 2018, 8:19am

Cool! Thank _az for all your help!

Topic		Replies	Views
Cert is expiring and don't know how to renew Help	21	1124	October 3, 2022
Can't renew certificate nginx reverse-proxy Help	46	3451	June 14, 2024
Nginx auto renewal Server	14	41521	May 25, 2017
Having trouble renewing Help	20	3436	April 26, 2018
Certificate Expiring in 20 days Help	25	1949	August 15, 2019

Update cert on NixOS / installing certbot

Related topics