Cert renewal with nginx as reverse proxy

Help
1

My domain is:
mx.rg47c.dk

I ran this command:
certbot-auto --nginx renew

It produced this output:
I cannot get it right now, as I have had too many failed authorizations recently in order to try to fix it
For another domain on the same reverse proxy, I get:

Failed authorization procedure. owncloud.rg47c.dk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://owncloud.rg47c.dk/.well-known/acme-challenge/IpDp_BMW1fYFHqYeU97T5VJ3-6d7CuL49_IetJkO5AQ: “\r\n404 Not Found\r\n<body bgcolor=“white”>\r\n

404 Not Found

\r\n
”

My web server is (include version):
nginx 1.10.2

The operating system my web server runs on is (include version):
CentOS Linux release 7.3.1611

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No

I use nginx as a reverse proxy in order to host multiple SSL sites on a single public IP.

So the config for my website looks like:

server {
listen 80;
server_name mx.rg47c.dk;
return 301 https://$server_name$request_uri;
}

server {
  listen 443 ssl http2;
  server_name    mx.rg47c.dk;
  ssl_certificate /etc/letsencrypt/live/mx.rg47c.dk/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/mx.rg47c.dk/privkey.pem;
  ssl_session_timeout 1d;
  ssl_session_cache shared:SSL:50m;
  ssl_session_tickets off;
  ssl_protocols TLSv1.1 TLSv1.2;
  ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!3DES:!MD5:!PSK';
  ssl_prefer_server_ciphers on;
  ssl_dhparam /etc/letsencrypt/dhparam.pem;
  add_header Strict-Transport-Security max-age=15768000;
  ssl_stapling on;
  ssl_stapling_verify on;
  ssl_trusted_certificate /etc/letsencrypt/live/mx.rg47c.dk/chain.pem;
  root /var/www/example.com/html;
  index index.php;
  access_log /var/log/nginx/mx.rg47c.dk.log;



  location / {
    proxy_pass https://192.168.xxx.yy:443/;
    proxy_set_header Proxy "";
    proxy_set_header Accept-Encoding "";
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    add_header Front-End-Https on;
    add_header Strict-Transport-Security "max-age=63072000; includeSubdomains; preload";
    client_max_body_size 1024M;
    proxy_read_timeout 600s;
    proxy_send_timeout 600s;
    proxy_connect_timeout 600s;
  }
}

The problem is, that certificate renewal has been working for over an year - but now it suddenly doesn’t work anymore.

Is there a way, where I can enable the " /.well-known/acme-challenge/" location to be on the local server, so that it isn’t redirected to the other internal server - so I can get things up and running again ?

2

The auth request is to HTTP.
Your vhost (shown) is only 443 SSL.

Yes, that is the ideal setup.

3

Change this to:

location / {
     return 301 https://$server_name$request_uri;
}

otherwise when Certbot adds the location for the Let's Encrypt domain validation, it "won't work" because your 301 takes precedence over the location blocks.

4

And then maybe also add this for the virtual host on port 80 ?

    location /.well-known/ {
       default_type "text/plain";
       root         /var/www/letsencrypt;
    }
5

In this case, it’s not necessary. What the Certbot nginx plugin does is temporarily insert a directive like:

location = /.well-known/acme-challenge/xxxxwhatever {
    default_type text/plain;
    return 200 "the authentication string";
}

So you can see why potentially it would have combined badly with your previous return statement.

6

I just noticed that the error is for a different hostname to your configuration, not mx.rg47c.dk. Do you have separate nginx configuration for that domain?

7

Hm… But this will probably break the possibility to auto-renew/request the certificate automatically.

I have a couple of other sites passing through this nginx reverse proxy, and it would be nice if the certbot-auto just could place the authentication reply in a file in the acme-challenge folder and thereby have the process running automatically (both renewal and also requests for new certificates)

Would it work, if I do something like:

server {
listen 80;
server_name mx.rg47c.dk;
root /var/www/mx.rg47c.dk/

location / {
    return 301 https://$server_name$request_uri;
    }

}

And then a .well-known folder was located at /var/www/mx.rg47c.dk/.well-known ?

I have multiple sites configured (different FQDN names), but the basic principle of the different configs are the same…

8

What I'm trying to say, is that certbot --nginx does not use a /.well-known directory on your disk at all.

It temporarily inserts a directive like I described above. I'm not suggesting that YOU write that directive, I'm just telling you that Certbot's nginx plugin modifies your nginx configuration in order to fulfill the challenge.

If you just make the location / { ... } fix, Certbot should start succeeding again on its own, as long as you sort out the mx vs owncloud thing.

9

Just tried it on another virtual host - with the suggested change:

server {
      listen         80;
      server_name    webmail.vangevej1.dk;

        root         /var/www/webmail.vangevej1.dk;

        location / {
            return 301 https://$server_name$request_uri;
        }

    }

Still gives problems with certbot-auto:

Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter ‘c’ to cancel): 9
Cert is due for renewal, auto-renewing…
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for webmail.vangevej1.dk
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. webmail.vangevej1.dk (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://webmail.vangevej1.dk/.well-known/acme-challenge/O_1O40-EHX39PHxeywP31PkqnFEi_pccGyInE2nLuLA: “\r\n404 Not Found\r\n<body bgcolor=“white”>\r\n

404 Not Found

\r\n
”

IMPORTANT NOTES:

10 
$ curl -i webmail.vangevej1.dk
HTTP/1.1 301 Moved Permanently
Date: Thu, 08 Nov 2018 10:36:44 GMT
Server: Apache/2.4.6 (CentOS) OpenSSL/1.0.1e-fips mod_wsgi/3.4 Python/2.7.5
Location: https://webmail.vangevej1.dk/

$ curl -ik https://webmail.vangevej1.dk
HTTP/2 200
server: nginx/1.10.2

You seem to have Apache running on port 80 and nginx on port 443.

Is this intended?

Certbot would not be able to perform authentication if it was doing it via nginx but Apache was listening on 80 instead.

You could try something wild like:

certbot-auto renew --cert-name webmail.vangevej1.dk -a apache -i nginx --dry-run

but it depends what you want to do.

11

Only nginx running on port 80:

netstat -lptn | grep ‘:80’

tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 32658/nginx: master

It must be the redirect that is giving the apache response…

12

Seems like a firewall issue - not sending the traffic to the right host when using port 80 … I will need to check that once I have physical access to the equipment …

13

I wonder - would it be possible to do the renewal via DNS validation instead ?

14

Hi @bipsendk

this is always possible. But your dns provider should support an API, so you can automate that.

Then you don't need a running webserver or an open port 80.

Certbot has a lot dns plugins, acme.sh has more then 50.

15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.