Update cert on NixOS / installing certbot


#1

Hi there,

A vendor setup our NixOS for us and now the certificate is expiring. I have 2 question I need to ask:

  1. How can I renew my certs? Since we are using NixOS. I’m not sure how to install certbot on NixOS and renew the certs?
  2. How can I redo our certs, so that the email for cert expiry will be emailed to me?

Thanks.
Desmond

My domain is: portal.docdoc.com

I ran this command: sudo certbot renew

It produced this output: command not found (I don’t have the certbot command on the system)

My web server is (include version): nginx (don’t know which version)

The operating system my web server runs on is (include version): NixOS (don’t know which version)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no


#2

https://nixos.org/nixos/manual/index.html#module-security-acme

That uses simpl_le under the hood, but other clients like Certbot and Lego are available from the Nix package manager. You should try the above though, since it is actually documented.

I would take care that you do things in the “NixOS way”, as it’s not quite the same as using regular Linux, and therefore the usual recommendations on how to use Certbot and other ACME clients may not be suitable for you specifically.


#3

Hi _az,

Thanks for the help.

Yup, I’m trying to do it the NixOS way but I’m stuck. The NixOS is configured correctly. It seem that the simpl_le is broken (not sure how). Is that a way or a command for me to check this? Can I renew the certs manually? If simpl_le is broken, how can I fix it?

Thanks.


#4

Well, there is clear evidence that your nginx is not configured in the way that is suggested in the NixOS documentation.

They suggest that you need to have a special rule for /.well-known/acme-challenge on your port 80 virtual host:

# (from the NixOS documentation)
listen 80;
listen [::]:80;

location /.well-known/acme-challenge {
  root /var/www/challenges;
}

However, we see that your nginx server is missing this stanza in the port 80 server, because we receive an HTTPS redirect (and we shouldn’t if the above was present):

$ curl -ikL portal.docdoc.com/.well-known/acme-challenge/xxx
HTTP/1.1 301 Moved Permanently
Server: nginx/1.12.1
Date: Thu, 12 Apr 2018 06:58:26 GMT
Content-Type: text/html
Content-Length: 185
Connection: keep-alive
Location: https://portal.docdoc.com/.well-known/acme-challenge/xxx

#5

We have these codes:

    server {
      server_name portal.docdoc.com;
      listen 80;
      return 301 https://$server_name$request_uri;

      location /.well-known {
        root /var/www/acme/portal.docdoc.com;
      }
    }

Do I just add it in like this:

    server {
      server_name portal.docdoc.com;
      listen 80;
      return 301 https://$server_name$request_uri;

      location /.well-known {
        root /var/www/acme/portal.docdoc.com;
      }
    
      location /.well-known/acme-challenge {
        root /var/www/challenges;
      }
    }

But on the NixOS server there is no challenges directory.


#6

That is the problem :slight_smile: . You have a server-wide redirect that is preventing the location stanza from doing anything.

Try (after getting rid of the existing return line):

  location / {
    return 301 https://$server_name$request_uri;
  }
  location /.well-known {
    root /var/www/acme/portal.docdoc.com;
  }

I’m not too sure about the ACME path, safest to try what your previous server administrator setup, and then repair it if it does not work. Best to check what your configuration.nix file says under security.acme.certs.


#7

The security.acme.certs shows this webroot = "/var/www/acme/portal.docdoc.com";

So, after the update, how would I know if the cert will be renewed? It will expire on 14 April.


#8

In that case

location /.well-known {

should be

location /.well-known/acme-challenge {

Did you reload nginx? Your server is still doing the bad redirect.

I think you can force the renewal attempt by running

systemctl start acme-portal.docdoc.com

(However, not sure, never used NixOS).


#9

Okay, I’m changing the location and am going to upload the settings to NixOS.

Then I’ll try the reload. Hopefully it works.


#10

I should add that you should be able to check the logs for the renewal using

journalctl -u acme-portal.docdoc.com

It might be worth looking at before you go trying again.


#11

I think it worked and renewed. Not sure if I read the logs correctly.

This is from the log:

Apr 12 07:43:05 www systemd[1]: Starting Renew ACME Certificate for portal.docdoc.com... Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,111:DEBUG:simp_le:1371: ['-v', '-d', 'portal.docdoc.com', '--de Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,112:DEBUG:simp_le:367: Loading fullchain.pem Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,112:DEBUG:simp_le:367: Loading full.pem Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,113:DEBUG:simp_le:367: Loading key.pem Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,113:DEBUG:simp_le:367: Loading account_key.json Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,147:DEBUG:simp_le:1203: Existing SANs: [u'portal.docdoc.com'], Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,149:DEBUG:simp_le:1068: Certificate expires in 89 days, 22:19:5 Apr 12 07:43:06 www acme-portal.docdoc.com-start[32399]: 2018-04-12 07:43:06,149:INFO:simp_le:1383: Certificates already exist and renewal i Apr 12 07:43:06 www systemd[1]: Started Renew ACME Certificate for portal.docdoc.com.


#12

So, the certificate DID renew: https://crt.sh/?id=392777144 (at least, I’m pretty sure)

However, your nginx server is still using an old certificate:

$ openssl s_client -connect portal.docdoc.com:443 -servername portal.docdoc.cm -showcerts 2>/dev/null | openssl x509 -noout -dates
notBefore=Jan 14 23:53:06 2018 GMT
notAfter=Apr 14 23:53:06 2018 GMT

You may need to reload nginx (though it should have done that by itself), also check what ssl_certificate your nginx config is pointing to, it may be pointing to a copy of the certificate that was not automatically updated by NixOS/simp_le.


#13

You are talking about this correct?

server {
          server_name portal.docdoc.com;
          listen 443 ssl http2;
          ssl_certificate /var/lib/acme/portal.docdoc.com/full.pem;
          ssl_certificate_key /var/lib/acme/portal.docdoc.com/key.pem;
          ssl_dhparam /var/lib/acme/portal.docdoc.com/dhparam.pem;

          client_body_buffer_size 100m;

          location / {
            proxy_set_header X-Forwarded-For $remote_addr;
            proxy_pass http://localhost:8080/;
          }
        }

It is pointing to the correct files as the files have been changed. I’ve restarted nginx but it still have the same dates.


#14

What’s the output of this:

ls -la /var/lib/acme/portal.docdoc.com
openssl x509 -in /var/lib/acme/portal.docdoc.com/full.pem -noout -dates
openssl x509 -in /var/lib/acme/portal.docdoc.com/fullchain.pem -noout -dates

#15
[desmond@www:~]$ sudo ls -la /var/lib/acme/portal.docdoc.com
total 36
drwx------ 2 root root 4096 Sep  7  2017 .
drwxr-xr-x 6 root root 4096 Sep  6  2017 ..
-rw-r--r-- 1 root root 3169 Sep  7  2017 account_key.json
-rw-r--r-- 1 root root  424 Sep  6  2017 dhparam.pem
-rwx------ 1 root root 4150 Apr 12 07:36 fullchain.pem
-rwx------ 1 root root 7422 Apr 12 07:36 full.pem
-rwx------ 1 root root 3271 Apr 12 07:36 key.pem


[desmond@www:~]$ sudo openssl x509 -in /var/lib/acme/portal.docdoc.com/full.pem -noout -dates
notBefore=Apr 12 06:36:18 2018 GMT
notAfter=Jul 11 06:36:18 2018 GMT


[desmond@www:~]$ sudo openssl x509 -in /var/lib/acme/portal.docdoc.com/fullchain.pem -noout -dates
notBefore=Apr 12 06:36:18 2018 GMT
notAfter=Jul 11 06:36:18 2018 GMT

#16

Edit: OK, so I had a typo in my previous openssl command.

-servername portal.docdoc.cm -

Whoops, should have been

-servername portal.docdoc.com

When we connect with the right arguments, we get the updated certificate, happily:

notBefore=Apr 12 06:36:18 2018 GMT
notAfter=Jul 11 06:36:18 2018 GMT

When connecting to an unknown SNI name (such as portal.docdoc.cm, we get the certificate for portalsandbox.docdoc.com, which is not renewed yet).

Problem solved, I think :slight_smile:


#17
[desmond@www:~]$ sudo systemctl restart nginx

[desmond@www:~]$ systemctl status nginx
● nginx.service - Nginx Web Server
   Loaded: loaded (/nix/store/l0b34js1499aij6a20zacmcfb6wy6n38-unit-nginx.service/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2018-04-12 08:16:10 UTC; 3s ago
  Process: 1328 ExecStartPre=/nix/store/lv71lq5izg78bzdg52irm-unit-script/bin/nginx-pre-start (code=exited, status=0/SUCCESS)
 Main PID: 1332 (nginx)
    Tasks: 2 (limit: 4915)
   CGroup: /system.slice/nginx.service
           ├─1332 nginx: master process /nix/store/lisnkdnh24cfaz3j5zb6p-nginx-1.12.1/bin/nginx -c /nix/store/403bfqqpm09n
           └─1336 nginx: worker process

Apr 12 08:16:10 www systemd[1]: Starting Nginx Web Server...
Apr 12 08:16:10 www systemd[1]: Started Nginx Web Server.

[desmond@www:~]$ date
Thu Apr 12 08:16:30 UTC 2018

#18

My bad, see my previous post :slight_smile:


#19

Cool! Thank _az for all your help!


#20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.