# Nginx reverse proxy issues

**URL:** <https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898>\
**Category:** Server\
**Created:** [February 1, 2019, 1:16am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898 "2019-02-01T01:16:31Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 1:16am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/1 "2019-02-01T01:16:31Z")

</div>

Hey there. I set up a server to teach myself Linux, so it has a hodgepodge of services spackled together, all behind all nginx reverse proxy. I have since become more proficient in administration, but this server is out of state, and not something I want to tear down and rebuild. Thus, I am stuck trying to solve this problem:

## Situation:

I have an nginx reverse proxy running on CentOS that receives all traffic from 80 and 443. Traffic is not redirected to 443 on the reverse proxy. I have two specific services that I am focused on.

## The command I am using to obtain the certs is:

` certbot certonly --webroot -w /var/www/$FQDN -d $FQDN`

Please find the relevant Nginx config files below. These are located on the reverse proxy itself.

## Subdomain 1 - this one works:

```nohighlight
# /etc/nginx/conf.d/sub1.domain.org.conf

server {
        listen 80;
        server_name sub1.domain.org;
        include conf.d/acme.inc;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://192.168.1.46;
                client_max_body_size 5G;
        }
}

```

and

```nohighlight
# /etc/nginx/conf.d/acme.inc

location ^~ /.well-known/acme-challenge { 
        allow all; 
        alias /var/www/acme;
}

```

## Subdomain 2 - this one doesn't work:

However, when I do what, to my eyes is the same thing for my other service:

```nohighlight
# /etc/nginx/conf.d/sub2.domain.org

server {
        listen 80;
        server_name sub2.domain.org;
        include conf.d/acme.inc;

        location / {
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_pass http://192.168.1.30;
                client_max_body_size 100M;
        }

}

```

... with the included `acme.inc` file as posted above.

### The error message is as follows:

> FailedChallenges: Failed authorization procedure. $FQDN (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://$FQDN/.well-known/acme-challenge/YGdiTdw6WXr4fKGoUgqQQth0RvafnKatcFo52jmejL0: "\r\n404 Not Found\r\n\<body bgcolor="white"\>\r\n\<h1\>404 Not Found\r\n
> 
> * * *

* * *

I am not sure what the issue is. I have also compared the apache configs on the backends, and they are fairly similar, though they really should not matter since the reverse proxy is where Let'sEncrypt is looking.

Any help is appreciated. I am happy to provide other configuration files when requested. I have some time before I can try again, since I am rate-limited after several failed attempts... for the second day in a row. ☹

Thanks!

---

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 1:23am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/2 "2019-02-01T01:23:46Z")

</div>

As an addition (I felt adding this to the main post would be too bloated), I also tried the following block in each .conf file:

```auto
        location /.well-known {
            alias /var/www/$FQDN/.well-known;
}

```

Again, this worked for subdomain1 but not subdomain2.

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [February 1, 2019, 1:40am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/3 "2019-02-01T01:40:15Z")

</div>

Which ACME client are you using, what command invocation are you using to issue the certificate, and what is the output of that invocation?

---

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 2:07am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/4 "2019-02-01T02:07:33Z")

</div>

Thanks for the response - definitely an oversight. I have edited my post with that information.

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [February 1, 2019, 2:12am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/5 "2019-02-01T02:12:03Z")

</div>

To try remove the webroot as a factor that is involved in your issue, would you be willing to try doing this using the nginx authenticator? It may succeed, since it does not rely on your webroot or reverse proxy setup.

```
certbot certonly -d sub2.domain.org -a nginx --dry-run
```

---

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 2:16am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/6 "2019-02-01T02:16:50Z")

</div>

Thanks for the suggestion. I did the dry run (forgot I could do that, whoops!)

See below for output:

[root@nginxproxy letsencrypt]# sudo certbot certonly -d $FQDN -a nginx --dry-run  
Saving debug log to /var/log/letsencrypt/letsencrypt.log  
Plugins selected: Authenticator nginx, Installer None  
Starting new HTTPS connection (1): [acme-staging-v02.api.letsencrypt.org](http://acme-staging-v02.api.letsencrypt.org)  
Obtaining a new certificate  
Performing the following challenges:  
http-01 challenge for $FQDN  
Waiting for verification…  
Cleaning up challenges  
Failed authorization procedure. $FQDN (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://$FQDN/.well-known/acme-challenge/roFTsWgYnkRsS3si-oRpzagN1LzAw3oXZVQrOvYF9v4: “\r\n404 Not Found\r\n\<body bgcolor=“white”\>\r\n\<h1\>404 Not Found\r\n

* * *
”

IMPORTANT NOTES:

- The following errors were reported by the server:

- Your account credentials have been saved in your Certbot  
configuration directory at /etc/letsencrypt. You should make a  
secure backup of this folder now. This configuration directory will  
also contain certificates and private keys obtained by Certbot so  
making regular backups of this folder is ideal.

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [February 1, 2019, 2:30am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/7 "2019-02-01T02:30:07Z")

</div>

Well, that's a pretty good sign of a major config (or port forwarding) issue I think.

Back to your webroot approach, I noticed a mismatch:

> [@createchange](#):
>
> alias /var/www/acme;

> [@createchange](#):
>
> `certbot certonly --webroot -w /var/www/$FQDN -d $FQDN`

This should be `-w /var/www/acme` for any domain, right?

Can you create `/var/www/acme/testfile` and confirm it's accessible in a browser for both domains?

---

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 3:54am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/8 "2019-02-01T03:54:41Z")

</div>

I am able to access it on the one that is able to obtain a cert, and not the one that isn’t - which makes perfect sense. But here shouldn’t be anything wrong with port forwarding.

My router takes in port 80 and 443, and directs them to the Nginx reverse proxy. The request for the .well-known directory stays on that server, so I can’t see how there would be any issue. Please correct me if I am wrong, of course.

What other things can I look into that could provide insight as to why this is a problem? I even made another block in the one that fails getting the cert:

```auto
location /testing {
                alias /var/www/sub2.domain.org/testing;
       }

```

and when I navigate to [sub2.domain.org/testing/test](http://sub2.domain.org/testing/test), it doesn’t even remain on the proxy - it goes to the backend server and gives me that server’s 404 page.

I appreciate your help. 🙂

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [February 1, 2019, 4:25am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/9 "2019-02-01T04:25:39Z")

</div>

Can you post your full effective nginx config? You can generate it using:

```
nginx -T

```

Obfuscate the `server_name` directives if you want, but post it in its complete form.

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [February 1, 2019, 4:28am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/10 "2019-02-01T04:28:12Z")

</div>

> [@createchange](#):
>
> # /etc/nginx/conf.d/sub2.domain.org

I don't know if this is an accidental ommission, but the default `nginx.conf` only includes `conf.d/*.conf` (note the extension and the lack of extension in the above).

---

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 5:00am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/11 "2019-02-01T05:00:18Z")

</div>

> [@\_az](#):
>
> nginx -T

Good catch on that, but that was a copy/paste error on that. 🙂

Thanks for the insight on `nginx -T` - The late reply is because I spent a good little while searching through there to see if I could solve the issue. I couldn't, but I did find other elements I had forgotten about (see: `passthrough.conf` - which was one of the many iterations I have gone through in trying to solve this on my own, on-and-off, for months).

I did obfuscate, and removed some entries which has personally identifiable information. I feel like I am baring just by posting this full config. This server has been a great learning bed for me, but the webserver is one of the first things I learned. Definitely know there is plenty of room for improvement if I had time to do it!

Also note - I strayed from the `include conf.d/acme.inc;` route and reverted to their own `.well-known` directories. The paths for both DO exist in my filesystem.

I made a pastebin, because the comments were breaking the readability: [https://pastebin.com/AGuB0AY8](https://pastebin.com/AGuB0AY8) (edit: new link)

Thanks!

---

<div class="post-metadata">

**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\
**Post date:** [February 1, 2019, 5:20am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/12 "2019-02-01T05:20:47Z")

</div>

Does your nginx config actually work (e.g. if you stop and start nginx)?

You have two incompatible binds on port 443. One under `http {}`, one under `stream {}`. When I try to start nginx with your config -\> fatal error:

> 2019/02/01 05:16:27 [emerg] 3901#3901: bind() to 0.0.0.0:443 failed (98: Address already in use)

Multiple binds for 443 are acceptable if they are all inside `http {}`, but this is only because they are implemented using SNI. This doesn't work for a `stream {}` server, since it's just ordinary TCP.

Could this explain why your config changes (e.g. `/testing`) don't seem to actually be effective?

Can you check `/var/log/nginx/error.log` for any complaining?

For what it's worth, when I comment out your `stream{}->server{}` block, your configuration _does_ work as expected, but not until then.

---

<div class="post-metadata">

**Author:** ![createchange](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/createchange/32/29563_2.png) [@createchange](https://community.letsencrypt.org/u/createchange)\
**Post date:** [February 1, 2019, 6:08am UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/13 "2019-02-01T06:08:39Z")

</div>

You did it. 🙂 Thanks! I really appreciate that.

I always ran `nginx -s reload` rather than starting and stopping and yes - it turns out that broke the config (as well as a couple other things). Thanks for the guidance on how to at least get the service started again, haha.

I am going to have to read what you typed a little more closely tomorrow to understand that, but instead I got the cert issued and then had a ton of fixes to do on the backend server. I still need to copy over certs from frontend to backend it seems, but at least I am one step closer to having true passthrough set up.

Thanks again for helping this entry-level sysadmin - very appreciated!

---

<div class="post-metadata">

**Author:** ![rg305](https://sea3.discourse-cdn.com/letsencrypt/user_avatar/community.letsencrypt.org/rg305/32/91314_2.png) [@rg305](https://community.letsencrypt.org/u/rg305)\
**Post date:** [February 1, 2019, 10:47pm UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/14 "2019-02-01T22:47:51Z")

</div>

Streams can be complicated to implement (correctly).  
From my experience, they are generally put into a server block or within the http block and both end up creating problems or breaking the config.  
I’ve found that most (if not all) streams {} can/should be at placed at the very top level (equal to http {} block) to work correctly.  
[above, and not connected to in any way with, the http {} section]

---

<div class="post-metadata">

**Author:** ![system](https://global.discourse-cdn.com/letsencrypt/original/3X/c/a/ca6c06ea1ea201324bba7048c6841ce60236468d.png) [@system](https://community.letsencrypt.org/u/system)\
**Post date:** [March 3, 2019, 10:47pm UTC](https://community.letsencrypt.org/t/nginx-reverse-proxy-issues/84898/15 "2019-03-03T22:47:52Z")

</div>

Closed on March 3, 2019, 10:47pm UTC

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.