Nginx reverse proxy issues


#1

Hey there. I set up a server to teach myself Linux, so it has a hodgepodge of services spackled together, all behind all nginx reverse proxy. I have since become more proficient in administration, but this server is out of state, and not something I want to tear down and rebuild. Thus, I am stuck trying to solve this problem:

Situation:

I have an nginx reverse proxy running on CentOS that receives all traffic from 80 and 443. Traffic is not redirected to 443 on the reverse proxy. I have two specific services that I am focused on.

The command I am using to obtain the certs is:

certbot certonly --webroot -w /var/www/$FQDN -d $FQDN

Please find the relevant Nginx config files below. These are located on the reverse proxy itself.

Subdomain 1 - this one works:

# /etc/nginx/conf.d/sub1.domain.org.conf

server {
        listen 80;
        server_name sub1.domain.org;
        include conf.d/acme.inc;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://192.168.1.46;
                client_max_body_size 5G;
        }
}

and

# /etc/nginx/conf.d/acme.inc

location ^~ /.well-known/acme-challenge { 
        allow all; 
        alias /var/www/acme;
}

Subdomain 2 - this one doesn’t work:

However, when I do what, to my eyes is the same thing for my other service:

# /etc/nginx/conf.d/sub2.domain.org

server {
        listen 80;
        server_name sub2.domain.org;
        include conf.d/acme.inc;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://192.168.1.30;
                client_max_body_size 100M;
        }

}

… with the included acme.inc file as posted above.

The error message is as follows:

FailedChallenges: Failed authorization procedure. $FQDN (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://$FQDN/.well-known/acme-challenge/YGdiTdw6WXr4fKGoUgqQQth0RvafnKatcFo52jmejL0: "\r\n404 Not Found\r\n<body bgcolor=“white”>\r\n<h1>404 Not Found\r\n



I am not sure what the issue is. I have also compared the apache configs on the backends, and they are fairly similar, though they really should not matter since the reverse proxy is where Let’sEncrypt is looking.

Any help is appreciated. I am happy to provide other configuration files when requested. I have some time before I can try again, since I am rate-limited after several failed attempts… for the second day in a row. :frowning:

Thanks!


#2

As an addition (I felt adding this to the main post would be too bloated), I also tried the following block in each .conf file:

        location /.well-known {
            alias /var/www/$FQDN/.well-known;
}

Again, this worked for subdomain1 but not subdomain2.


#3

Which ACME client are you using, what command invocation are you using to issue the certificate, and what is the output of that invocation?


#4

Thanks for the response - definitely an oversight. I have edited my post with that information.


#5

To try remove the webroot as a factor that is involved in your issue, would you be willing to try doing this using the nginx authenticator? It may succeed, since it does not rely on your webroot or reverse proxy setup.

certbot certonly -d sub2.domain.org -a nginx --dry-run

#6

Thanks for the suggestion. I did the dry run (forgot I could do that, whoops!)

See below for output:

[root@nginxproxy letsencrypt]# sudo certbot certonly -d $FQDN -a nginx --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer None
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for $FQDN
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. $FQDN (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://$FQDN/.well-known/acme-challenge/roFTsWgYnkRsS3si-oRpzagN1LzAw3oXZVQrOvYF9v4: “\r\n404 Not Found\r\n<body bgcolor=“white”>\r\n<h1>404 Not Found\r\n


”

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: $FQDN
    Type: unauthorized
    Detail: Invalid response from
    http://$FQDN/.well-known/acme-challenge/roFTsWgYnkRsS3si-oRpzagN1LzAw3oXZVQrOvYF9v4:
    “\r\n404 Not Found\r\n<body
    bgcolor=“white”>\r\n<h1>404 Not
    Found\r\n


    ”

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.

  • Your account credentials have been saved in your Certbot
    configuration directory at /etc/letsencrypt. You should make a
    secure backup of this folder now. This configuration directory will
    also contain certificates and private keys obtained by Certbot so
    making regular backups of this folder is ideal.


#7

Well, that’s a pretty good sign of a major config (or port forwarding) issue I think.

Back to your webroot approach, I noticed a mismatch:

This should be -w /var/www/acme for any domain, right?

Can you create /var/www/acme/testfile and confirm it’s accessible in a browser for both domains?


#8

I am able to access it on the one that is able to obtain a cert, and not the one that isn’t - which makes perfect sense. But here shouldn’t be anything wrong with port forwarding.

My router takes in port 80 and 443, and directs them to the Nginx reverse proxy. The request for the .well-known directory stays on that server, so I can’t see how there would be any issue. Please correct me if I am wrong, of course.

What other things can I look into that could provide insight as to why this is a problem? I even made another block in the one that fails getting the cert:

location /testing {
                alias /var/www/sub2.domain.org/testing;
       }

and when I navigate to sub2.domain.org/testing/test, it doesn’t even remain on the proxy - it goes to the backend server and gives me that server’s 404 page.

I appreciate your help. :slight_smile:


#9

Can you post your full effective nginx config? You can generate it using:

nginx -T

Obfuscate the server_name directives if you want, but post it in its complete form.


#10

I don’t know if this is an accidental ommission, but the default nginx.conf only includes conf.d/*.conf (note the extension and the lack of extension in the above).


#11

Good catch on that, but that was a copy/paste error on that. :slight_smile:

Thanks for the insight on nginx -T - The late reply is because I spent a good little while searching through there to see if I could solve the issue. I couldn’t, but I did find other elements I had forgotten about (see: passthrough.conf - which was one of the many iterations I have gone through in trying to solve this on my own, on-and-off, for months).

I did obfuscate, and removed some entries which has personally identifiable information. I feel like I am baring just by posting this full config. This server has been a great learning bed for me, but the webserver is one of the first things I learned. Definitely know there is plenty of room for improvement if I had time to do it!

Also note - I strayed from the include conf.d/acme.inc; route and reverted to their own .well-known directories. The paths for both DO exist in my filesystem.

I made a pastebin, because the comments were breaking the readability: https://pastebin.com/AGuB0AY8 (edit: new link)

Thanks!


#12

Does your nginx config actually work (e.g. if you stop and start nginx)?

You have two incompatible binds on port 443. One under http {}, one under stream {}. When I try to start nginx with your config -> fatal error:

2019/02/01 05:16:27 [emerg] 3901#3901: bind() to 0.0.0.0:443 failed (98: Address already in use)

Multiple binds for 443 are acceptable if they are all inside http {}, but this is only because they are implemented using SNI. This doesn’t work for a stream {} server, since it’s just ordinary TCP.

Could this explain why your config changes (e.g. /testing) don’t seem to actually be effective?

Can you check /var/log/nginx/error.log for any complaining?

For what it’s worth, when I comment out your stream{}->server{} block, your configuration does work as expected, but not until then.


#13

You did it. :slight_smile: Thanks! I really appreciate that.

I always ran nginx -s reload rather than starting and stopping and yes - it turns out that broke the config (as well as a couple other things). Thanks for the guidance on how to at least get the service started again, haha.

I am going to have to read what you typed a little more closely tomorrow to understand that, but instead I got the cert issued and then had a ton of fixes to do on the backend server. I still need to copy over certs from frontend to backend it seems, but at least I am one step closer to having true passthrough set up.

Thanks again for helping this entry-level sysadmin - very appreciated!


#14

Streams can be complicated to implement (correctly).
From my experience, they are generally put into a server block or within the http block and both end up creating problems or breaking the config.
I’ve found that most (if not all) streams {} can/should be at placed at the very top level (equal to http {} block) to work correctly.
[above, and not connected to in any way with, the http {} section]