Nginx reverse proxy issues

createchange · February 1, 2019, 1:16am

Hey there. I set up a server to teach myself Linux, so it has a hodgepodge of services spackled together, all behind all nginx reverse proxy. I have since become more proficient in administration, but this server is out of state, and not something I want to tear down and rebuild. Thus, I am stuck trying to solve this problem:

Situation:

I have an nginx reverse proxy running on CentOS that receives all traffic from 80 and 443. Traffic is not redirected to 443 on the reverse proxy. I have two specific services that I am focused on.

The command I am using to obtain the certs is:

certbot certonly --webroot -w /var/www/$FQDN -d $FQDN

Please find the relevant Nginx config files below. These are located on the reverse proxy itself.

Subdomain 1 - this one works:

# /etc/nginx/conf.d/sub1.domain.org.conf

server {
        listen 80;
        server_name sub1.domain.org;
        include conf.d/acme.inc;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://192.168.1.46;
                client_max_body_size 5G;
        }
}

and

# /etc/nginx/conf.d/acme.inc

location ^~ /.well-known/acme-challenge { 
        allow all; 
        alias /var/www/acme;
}

Subdomain 2 - this one doesn't work:

However, when I do what, to my eyes is the same thing for my other service:

# /etc/nginx/conf.d/sub2.domain.org

server {
        listen 80;
        server_name sub2.domain.org;
        include conf.d/acme.inc;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://192.168.1.30;
                client_max_body_size 100M;
        }

}

... with the included acme.inc file as posted above.

The error message is as follows:

FailedChallenges: Failed authorization procedure. $FQDN (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://$FQDN/.well-known/acme-challenge/YGdiTdw6WXr4fKGoUgqQQth0RvafnKatcFo52jmejL0: "\r\n404 Not Found\r\n<body bgcolor="white">\r\n<h1>404 Not Found\r\n

I am not sure what the issue is. I have also compared the apache configs on the backends, and they are fairly similar, though they really should not matter since the reverse proxy is where Let'sEncrypt is looking.

Any help is appreciated. I am happy to provide other configuration files when requested. I have some time before I can try again, since I am rate-limited after several failed attempts... for the second day in a row.

Thanks!

createchange · February 1, 2019, 1:23am

As an addition (I felt adding this to the main post would be too bloated), I also tried the following block in each .conf file:

        location /.well-known {
            alias /var/www/$FQDN/.well-known;
}

Again, this worked for subdomain1 but not subdomain2.

_az · February 1, 2019, 1:40am

Which ACME client are you using, what command invocation are you using to issue the certificate, and what is the output of that invocation?

createchange · February 1, 2019, 2:07am

Thanks for the response - definitely an oversight. I have edited my post with that information.

_az · February 1, 2019, 2:12am

To try remove the webroot as a factor that is involved in your issue, would you be willing to try doing this using the nginx authenticator? It may succeed, since it does not rely on your webroot or reverse proxy setup.

certbot certonly -d sub2.domain.org -a nginx --dry-run

createchange · February 1, 2019, 2:16am

Thanks for the suggestion. I did the dry run (forgot I could do that, whoops!)

See below for output:

[root@nginxproxy letsencrypt]# sudo certbot certonly -d $FQDN -a nginx --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer None
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for $FQDN
Waiting for verification…
Cleaning up challenges
Failed authorization procedure. $FQDN (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://$FQDN/.well-known/acme-challenge/roFTsWgYnkRsS3si-oRpzagN1LzAw3oXZVQrOvYF9v4: “\r\n404 Not Found\r\n<body bgcolor=“white”>\r\n<h1>404 Not Found\r\n

”

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: $FQDN
Type: unauthorized
Detail: Invalid response from
http://$FQDN/.well-known/acme-challenge/roFTsWgYnkRsS3si-oRpzagN1LzAw3oXZVQrOvYF9v4:
“\r\n404 Not Found\r\n<body
bgcolor=“white”>\r\n<h1>404 Not
Found\r\n

”
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
Your account credentials have been saved in your Certbot
configuration directory at /etc/letsencrypt. You should make a
secure backup of this folder now. This configuration directory will
also contain certificates and private keys obtained by Certbot so
making regular backups of this folder is ideal.

_az · February 1, 2019, 2:30am

Well, that's a pretty good sign of a major config (or port forwarding) issue I think.

Back to your webroot approach, I noticed a mismatch:

This should be -w /var/www/acme for any domain, right?

Can you create /var/www/acme/testfile and confirm it's accessible in a browser for both domains?

createchange · February 1, 2019, 3:54am

I am able to access it on the one that is able to obtain a cert, and not the one that isn’t - which makes perfect sense. But here shouldn’t be anything wrong with port forwarding.

My router takes in port 80 and 443, and directs them to the Nginx reverse proxy. The request for the .well-known directory stays on that server, so I can’t see how there would be any issue. Please correct me if I am wrong, of course.

What other things can I look into that could provide insight as to why this is a problem? I even made another block in the one that fails getting the cert:

location /testing {
                alias /var/www/sub2.domain.org/testing;
       }

and when I navigate to sub2.domain.org/testing/test, it doesn’t even remain on the proxy - it goes to the backend server and gives me that server’s 404 page.

I appreciate your help.

_az · February 1, 2019, 4:25am

Can you post your full effective nginx config? You can generate it using:

nginx -T

Obfuscate the server_name directives if you want, but post it in its complete form.

_az · February 1, 2019, 4:28am

I don't know if this is an accidental ommission, but the default nginx.conf only includes conf.d/*.conf (note the extension and the lack of extension in the above).

createchange · February 1, 2019, 5:00am

Good catch on that, but that was a copy/paste error on that.

Thanks for the insight on nginx -T - The late reply is because I spent a good little while searching through there to see if I could solve the issue. I couldn't, but I did find other elements I had forgotten about (see: passthrough.conf - which was one of the many iterations I have gone through in trying to solve this on my own, on-and-off, for months).

I did obfuscate, and removed some entries which has personally identifiable information. I feel like I am baring just by posting this full config. This server has been a great learning bed for me, but the webserver is one of the first things I learned. Definitely know there is plenty of room for improvement if I had time to do it!

Also note - I strayed from the include conf.d/acme.inc; route and reverted to their own .well-known directories. The paths for both DO exist in my filesystem.

I made a pastebin, because the comments were breaking the readability: https://pastebin.com/AGuB0AY8 (edit: new link)

Thanks!

_az · February 1, 2019, 5:20am

Does your nginx config actually work (e.g. if you stop and start nginx)?

You have two incompatible binds on port 443. One under http {}, one under stream {}. When I try to start nginx with your config -> fatal error:

2019/02/01 05:16:27 [emerg] 3901#3901: bind() to 0.0.0.0:443 failed (98: Address already in use)

Multiple binds for 443 are acceptable if they are all inside http {}, but this is only because they are implemented using SNI. This doesn't work for a stream {} server, since it's just ordinary TCP.

Could this explain why your config changes (e.g. /testing) don't seem to actually be effective?

Can you check /var/log/nginx/error.log for any complaining?

For what it's worth, when I comment out your stream{}->server{} block, your configuration does work as expected, but not until then.

createchange · February 1, 2019, 6:08am

You did it. Thanks! I really appreciate that.

I always ran nginx -s reload rather than starting and stopping and yes - it turns out that broke the config (as well as a couple other things). Thanks for the guidance on how to at least get the service started again, haha.

I am going to have to read what you typed a little more closely tomorrow to understand that, but instead I got the cert issued and then had a ton of fixes to do on the backend server. I still need to copy over certs from frontend to backend it seems, but at least I am one step closer to having true passthrough set up.

Thanks again for helping this entry-level sysadmin - very appreciated!

rg305 · February 1, 2019, 10:47pm

Streams can be complicated to implement (correctly).
From my experience, they are generally put into a server block or within the http block and both end up creating problems or breaking the config.
I’ve found that most (if not all) streams {} can/should be at placed at the very top level (equal to http {} block) to work correctly.
[above, and not connected to in any way with, the http {} section]

system · March 3, 2019, 10:47pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
.well-known/acme-challenge/ fail 404 in nginx proxy Help	19	8583	January 12, 2020
Failing subdomain certificate request. 404 Help	6	2555	November 28, 2018
.well-known/acme-challenge/ 404 Server	4	1035	October 10, 2018
Certbot failed to authenticate some domains (authenticator: webroot) Help	7	460	September 12, 2024
Nginx Reverse Proxy Help	5	2051	October 24, 2017