Nginx with multiple containers

kazyka · June 2, 2023, 6:20am

Hi, I am trying to learn to work with docker. Currently I am trying to setup a reverse proxy. However I am having some trouble.

Currently my folder structure is

2-multiple-example/
├── docker3
│   ├── data
│   │   ├── index.html
│   │   └── .well-known
│   │       └── acme-challenge
│   │           └── test
│   └── docker-compose.yml
├── docker4
│   ├── data
│   │   ├── index.html
│   │   └── .well-known
│   │       └── acme-challenge
│   │           └── test
│   └── docker-compose.yml
└── nginx
    ├── data1
    │   ├── index.html
    │   └── .well-known
    │       └── acme-challenge
    │           └── test
    ├── data2
    │   ├── index.html
    │   └── .well-known
    │       └── acme-challenge
    │           └── test
    ├── docker-compose.yml
    └── nginx.conf

In the nginx folder I have the following docker-compose.yml file content

version: "3.9"

networks:
 app-network:
   external:
     name: app-network

services:
  nginx:
    image: nginx:1.17.10
    environment:
      - PGID=1001
      - PUID=1001
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - /etc/letsencrypt:/etc/letsencrypt
      - ./data1:/usr/local/apache2/htdocs/data1
      - ./data2:/usr/local/apache2/htdocs/data2
    networks:
      - app-network
    depends_on:
      - docker1
      - docker2

  certbot:
    image: certbot/certbot
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - ./data1:/var/www/html/data1
      - ./data2:/var/www/html/data2
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"


  docker1:
    image: httpd:2.4
    ports:
      - "8500:80"
    volumes:
      - ./data1:/usr/local/apache2/htdocs
    networks:
      - app-network


  docker2:
    image: httpd:2.4
    ports:
      - "8501:80"
    volumes:
      - ./data2:/usr/local/apache2/htdocs
    networks:
      - app-network

and the content of the nginx.conf is

user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 1024;
}

http {

add_header 'Access-Control-Allow-Origin' '';

server {
    listen 80;
    listen [::]:80;

    server_name *.domain.com; 

  }

# Docker 1
  server {
    listen 80;
    server_name docker1.domain.com;


    location / {
      proxy_pass http://docker1:80;
      proxy_redirect off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Host $server_name;
    }

    location ~ /.well-known/acme-challenge/ {
        allow all;
        root /usr/local/apache2/htdocs/data1;
    }
  }

# Docker 1 https
  server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name docker1.domain.com;

    ssl_certificate /etc/letsencrypt/live/docker1.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/docker1.domain.com/privkey.pem;

    location / {
      proxy_pass http://docker1:80;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Host $server_name;
    }

  }


# Docker 2
  server {
    listen 80;
    server_name docker2.domain.com;


    location / {
      proxy_pass http://docker2:80;
      proxy_redirect off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Host $server_name;
    }

    location ~ /.well-known/acme-challenge/ {
        allow all;
        root /usr/local/apache2/htdocs/data2;
    }
  }

# Docker 2 https
  server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name docker2.domain.com;

    ssl_certificate /etc/letsencrypt/live/docker2.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/docker2.domain.com/privkey.pem;

    location / {
      proxy_pass http://docker2:80;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Host $server_name;
    }

  }

# Docker 3
  server {
    listen 80;
    server_name docker3.domain.com;


    location / {
      proxy_pass http://docker3:80;
      proxy_redirect off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Host $server_name;
    }

    location ~ /.well-known/acme-challenge/ {
        allow all;
        root /var/www/nginx-data;
    }
  }

# Docker 3 https
  # server {
  #   listen 443 ssl http2;
  #   listen [::]:443 ssl http2;

  #   server_name docker3.domain.com;

  #   ssl_certificate /etc/letsencrypt/live/docker3.domain.com/fullchain.pem;
  #   ssl_certificate_key /etc/letsencrypt/live/docker3.domain.com/privkey.pem;

  #   location / {
  #     proxy_pass http://docker3:80;
  #     proxy_set_header Host $host;
  #     proxy_set_header X-Real-IP $remote_addr;
  #     proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
  #     proxy_set_header X-Forwarded-Host $server_name;
  #   }
  # }

# Docker 4
  server {
    listen 80;
    server_name docker4.domain.com;


    location / {
      proxy_pass http://docker4:80;
      proxy_redirect off;
      proxy_set_header Host $host;
      proxy_set_header X-Real-IP $remote_addr;
      proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
      proxy_set_header X-Forwarded-Host $server_name;
    }

    location ~ /.well-known/acme-challenge/ {
        allow all;
        #root /usr/local/apache2/htdocs/data1;
        root /etc/letsencrypt/;
    }
  }


}

I have no problem creating a certificate for docker1 and docker2 with this command.

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/html/data2 \
  --email mymail@gmail.com \
  --agree-tos --no-eff-email \
  -d docker2.domain.com" certbot

however, when I try to create it for docker3 I can't

I can access docker1.domain.com and docker2.domain.com

This is the content for the docker-compose.yml file in docker3

version: "3.9"
services:
  docker3:
    container_name: docker3
    image: httpd:2.4
    ports:
      - "8502:80"
    networks:
      - app-network
    volumes:
      - ./data:/usr/local/apache2/htdocs

  certbot:
    image: certbot/certbot
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt
      - ./data:/var/www/html
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do if [ ! -d /etc/letsencrypt/live/docker3.domain.com/ ]; then certbot certonly --test-cert --webroot --webroot-path=/var/www/html -d docker3.domain.com --email mymail@gmail.com --agree-tos --no-eff-email; fi; certbot renew; sleep 12h & wait $${!}; done;'"


networks:
  app-network:
    external:
      name: app-network

The error I get is

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: docker3.domain.com
  Type:   unauthorized
  Detail: 87.104.29.157: Invalid response from http://docker3.domain.com/.well-known/acme-challenge/cNamimCNG4Ov8Dgc57jp-kVJzUk6ZpAVqKBDsbcPWxY: 404

which also makes sense since I can't access this myself.

However, I can access http://docker3.domain.com

In the folder for docker3 certbot and httpd:2.4 I can see that both the volume for certbot and httpd:2.4 exist e.g. /var/www/html and /usr/local/apache2/htdocs.

would love some help figuring this out.

rg305 · June 2, 2023, 7:32am

None of that adds up in my calculator.

kazyka · June 2, 2023, 7:45am

If I proxy_pass to the port defined in the docker e.g. "8500:80" and then in the nginx.conf "proxy_pass http://docker1:8500" I get error 502.

Or do you ask why I have docker4.domain.com in the nginx.conf file?

rg305 · June 2, 2023, 7:53am

None of it makes much sense to me.

When using HTTP-01 authentication, port 80 must reach the ACME client.

If you are going to run the ACME client within each container, then you would have to use four external IPs OR a reverse proxy [to share the one external IP between the four HTTP sites].

If you are going to run the ACME client outside each container, then you don't have to worry about them [at all] to obtain the certs.

If you are going to mix-and-match [some inside and some outside], then you will have to use a reverse proxy for the inside ones.

Even the initial information provided doesn't make it clear where all four docker containers exist.
Nor how the Internet can reach them all via HTTP.

kazyka · June 2, 2023, 9:41am

I am to just trying to understand how to setup a reverse-proxy.

The two docker containers docker1 and docker2 exists in /nginx/docker-compose.yml With this I also have a nginx container and certbot.

And then docker3 is located inside /docker3/docker-compose.yml.

What do you need from me to make it clear where all four docker containers exist?

Or how would you set this up?

I am quite new to this so and I feel that every guide I find have everything in one docker-compose.yml file. Shouldn't it be possible to split it up so you have multiple files for each container?

rg305 · June 2, 2023, 10:11am

The nginx container has no access to docker3 and 4:

kazyka:

    volumes:
      - ./nginx.conf:/etc/nginx/nginx.conf
      - /etc/letsencrypt:/etc/letsencrypt
      - ./data1:/usr/local/apache2/htdocs/data1
      - ./data2:/usr/local/apache2/htdocs/data2
    networks:
      - app-network
    depends_on:
      - docker1
      - docker2

rg305 · June 2, 2023, 10:14am

FYI:
This is not an nginx nor a docker forum.
You would do well by seeking help within the appropriate forum.

kazyka · June 2, 2023, 10:22am

I will try to look for help in the appropriate forums Thank you for your help

system · July 2, 2023, 10:23am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.