Certificate renewal when running a VPS with Docker and nginx

Help
1

This question is about how to renew certificates within a VPS that has multiple domains, with each domain being it's own Docker service. Nginx is used as a Docker service.

Each of the Docker services are a Python Flask app.

For the sake of example (because I have multides of domains and Docker services), let's assume I have 1 certificate, called alldomains.com, that covers 2 domains: domain1.com, domain2.com.

SSL is working fine, but I'm having issues with renewing the certificates. I don't want to stop nginx to renew the certificates.

My docker compose file is as follows:

services:

  nginx:
    build: ./nginx
    volumes:
      - /etc/letsencrypt:/etc/nginx/ssl:rw
      - /var/www/certbot:/var/www/certbot:ro
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443

  service1:
    container_name: service1
    image: service1_image
    hostname: service1
    restart: unless-stopped
    ports:
      - 5110:5110

  service2:
    container_name: service2
    image: service2_image
    hostname: service2
    restart: unless-stopped
    ports:
      - 5111:5111

My nginx default.conf is as follows:

server {
	listen 80;
	server_name domain1.com;
	location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
	return 301 https://domain1.com$request_uri;
}


server {
	listen 443 ssl;
	server_name domain1.com;
	client_max_body_size 4G;

	ssl_certificate /etc/nginx/ssl/live/alldomains.com/fullchain.pem;
	ssl_certificate_key /etc/nginx/ssl/live/alldomains.com/privkey.pem;
	include /etc/nginx/conf.d/ssl.conf;

	location / {
		proxy_pass http://service1:5110/;
	}
}


server {
	listen 80;
	server_name domain2.com
	location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
	return 301 https://domain2.com$request_uri;
}


server {
	listen 443 ssl;
	server_name domain2.com;
	client_max_body_size 4G;

	ssl_certificate /etc/nginx/ssl/live/alldomains.com/fullchain.pem;
	ssl_certificate_key /etc/nginx/ssl/live/alldomains.com/privkey.pem;
	include /etc/nginx/conf.d/ssl.conf;

	location / {
		proxy_pass http://service2:5111/;
	}
}

So basically, domain1.com goes to docker service1, and domain2.com goes to docker service2.

Now, upon certificate renewal, I get the following errors (note: I've hidden the IP address):

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/alldomains.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for alldomains.com and 2 more domains

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  
  Domain: domain1.com
  Type:   unauthorized
  Detail: (hidden): Invalid response from https://domain1.com/.well-known/acme-challenge/USj1Ff639loanlEx2yPMHQzHBnnYnkHpV4iRgl2B-WU: 404
  
  Domain: domain2.com
  Type:   unauthorized
  Detail: (hidden): Invalid response from https://domain2.com/.well-known/acme-challenge/e16_bDx7fImMRbe_dWsxHWL39kDv3zZZJ-M4IczwTjw: 404
  

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Failed to renew certificate alldomains.com with error: Some challenges have failed.
All renewals failed. The following certificates could not be renewed:

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
  /etc/letsencrypt/live/alldomains.com/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

It seems to me that the issue is nginx proxying each Let's Encrypt port 80 request, for each domain, into it's 443 server block, and then it's looking for a .well-known/acme-challenge folder within the respective Docker service (ie a Python Flask app)

How can I solve this so that the certificates are renewed? Thanks very much for any advice

1 Like
2

You should change your port 80 server blocks to look like below. That way HTTP Challenge requests won't get redirected to HTTPS and then sent to your proxy service.

But, I'm a little puzzled by this affecting only "renew". It would have affected getting your initial cert too. Is that just a terminology thing or did you do something different to get the first cert? Because maybe something further needs to be done in that case.

server {
	listen 80;
	server_name domain1.com;
	location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
    location / {
	    return 301 https://domain1.com$request_uri;
    }
}
3 Likes
3

Thanks for replying so quickly @MikeMcQ .

Actually, all of my port 80 server blocks already have the above code.

If I goto the following URL without the httpS: http://domain1.com/.well-known/acme-challenge/hello-world, I get redirected to the secured, httpS service. Same with domain2

So I wonder why it's not redirecting properly, in that it should not redirect to https

For the initial cert, I ran the following docker compose:

version: "3.9"

services:
  certbot-initial:
    image: certbot/certbot:latest # change to fixed version in production
    container_name: certbot-initial
    volumes:
      - /etc/letsencrypt:/etc/letsencrypt:rw
    ports:
      - "80:80"
    command: certonly --standalone --email email@email.com --agree-tos --preferred-challenges http --dry-run -d 'alldomains'

Thanks

4

No, they were missing a location / wrapper for the redirect. Without that nginx will redirect every HTTP request to HTTPS. Adding that means HTTP requests matching the other location block for ACME Challenge get processed in the port 80 server block. And, only all the other requests get redirected. It is not intuitive but that is how nginx works.

Yeah, the --standalone means Certbot starts up its own server that needs exclusive access to port 80. That's a poor choice when you have a web server (like nginx) to handle requests. Because, as you note, then nginx needs to be stopped.

Although, --dry-run won't get a production cert and these are not candidates for renewal

Use something like

certonly --webroot -w /var/www/certbot -d 'alldomains'

You can add --dry-run for tests. You also then need a way to restart your nginx after you get a new cert. I am not expert with containers so maybe they need some other option, but, if no containers were involved you could do

certonly --webroot -w /var/www/certbot -d 'alldomains' --deploy-hook 'cmd'

Where 'cmd' is whatever suitable perhaps sudo systemctl reload nginx
You could also just schedule all your nginx containers to reload once a day to pickup any fresh cert issued during the day.

6 Likes
5

@MikeMcQ , you are absolutely right and I apologize for missing the location wrappers.

That solved my problem.

And your point on using webroot is noted!

I want to sincerely thank you for taking your time to help, and especially on a Saturday. I wish you all the very best

3 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.