Certbot + Docker compose + NGINX + windows : Certificate generation fail

Hi,
I am sorry for posting a request on a topic that seems to have been addressed so many times, but all the workarounds I have seen did not help me. Therefore I would be extremely thankful if someone could help me with this issue:
I am (unfortunately) working on windows server, running multiple services that are dockerized, including an API, a proxy (NGINX) and a website (accessible through NGINX on port 80). I have followed different tutorials, including this one. So in my docker-compose file, I have added a certbot container that writes into a folder located in its container, which is mounted to a folder in windows. NGINX has also a location that is mounted to the same windows folder, and that is given to client addressing the service at /.well-known/acme-challenge/ . IIS is disabled as it is handled by nginx.

The docker-compose configuration is as follows (the part that concern my issue) (without the "2" for the domain):

networks:
    network:
        driver: bridge
services:
    nginx:
        image: nginx:latest
        ports:
        - "80:80"
        volumes:
        - "./nginx.conf:/etc/nginx/nginx.conf"
        - "./html:/usr/share/nginx/html"
        - ./certbot/www/:/var/www/certbot/:rw  
        networks:
        - network
        restart: always
    certbot:
        image: certbot/certbot:latest
        volumes:
        - ./certbot/www/:/var/www/certbot/:rw
        - ./certbot/conf/:/etc/letsencrypt/:rw
        networks:
        - network
 

And the nginx conf file is as follows (without the 2 for the domain):

worker_processes 1;
events {
    worker_connections 1024;
}

http {
    include /etc/nginx/mime.types;
    server {
        listen 80 ;
        server_name sensordatacrichton2.uk www.sensordatacrichton2.uk; 

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
       location / {
      root /usr/share/nginx/html;
      index index.html;
    } 
    }
}

When I run the command (without the 2 for the domain):

docker-compose run --rm certbot certonly --webroot --webroot-path /var/www/certbot/ --dry-run -v -d sensordatacrichton2.uk

some challenges fail:

 Using the webroot path /var/www/certbot for all unmatched domains.
Waiting for verification...
Challenge failed for domain sensordatacrichton2.uk
http-01 challenge for sensordatacrichton2.uk

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: domainname.uk
  Type:   connection
  Detail: <ip_address>: Fetching http://sensordatacrichton2.uk/.well-known/acme-challenge/ckjDC6YMCpeZ0XkXVmOxS3rpzab5jKvQU80phyEtWeI: Connection refused

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

So it seems that certbot is not able to write into the www/certbot folder. I have created a “.well-known” folder and a “acme-challenge” folder, then have added an index.html file there, which is accessible from here: http://sensordatacrichton2.uk/.well-known/acme-challenge/index.html
When I get into my nginx container, I can successfully create a file that then appears in windows, so I do not really understand how come certbot cannot do the same thing…
If you have any idea of how I could solve it, it would be deeply appreciated !
Many thanks in advance !

Also if needed, the route to let’s encrypt (I wonder if this is all due to a DNS issue as, although my nginx service is accessible from outside, I cannot access it from my local network (not even from the computer using localhost, not 0.0.0.0, nor 127.0.0.0)):

 1   10.0.2.253
 2    no-dns-yet.demon.co.uk [194.70.168.245]
 3    ae12-xcr1.lns.cw.net [195.2.28.26]
 4    ae1-xcr1.ltw.cw.net [195.2.24.125]
 5    cloudflare-gw-xcr1.ltw.cw.net [195.2.29.90]
 6    141.101.71.63
 7    172.65.32.248

My domain is:
sensordatacrichton2.uk without the “2”
I ran this command:
docker-compose run --rm certbot certonly --webroot --webroot-path /var/www/certbot/ --dry-run -v -d sensordatacrichton2.uk

It produced this output:

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: domainname.uk
Type: connection
Detail: <ip_address>: Fetching http://sensordatacrichton2.uk/.well-known/acme-challenge/ckjDC6YMCpeZ0XkXVmOxS3rpzab5jKvQU80phyEtWeI: Connection refused

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.
*

My web server is (include version):
Nginx (docker-compose: latest)

The operating system my web server runs on is (include version):
Docker-compose, running on windows

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
Docker certbot

No, that's not the right conclusion. A "Connection Refused" is simply that the LE Auth Server could not contact your domain using HTTP on port 80. The connection was, well, refused :slight_smile:

Even if Certbot wrote to the "wrong" place the LE Auth Server should still be able to reach your nginx. It would get a "404" Not Found error if the challenge token wasn't placed correctly.

Certbot is not the "thing" making the HTTP request that fails. Certbot prepares its cert request, makes the challenge token, and then requests the cert from Let's Encrypt ACME Server. The LE Auth server(s) are the ones making the requests to you over the public internet and various locations world-wide.

That all said, I cannot reproduce the "connection refused" using various testing tools and from various world-wide locations. This "feels like" you have a firewall blocking certain IP addresses that includes the LE Auth Servers.

Is that possible?

Oh, and, welcome to the community @bcouraud

6 Likes

Thank you so much for your quick reply and explanations @MikeMcQ !!
Hmm, that would be really weird (if the firewall was blocking certain IP addresses), especially since I have an inbound rule in the Firewall that allows all IPs on port 80.
But you are totally right, if a 404 would not result in this error message ("Connection Refused"), then the problem is somewhere else indeed.
Maybe that could come from the internet provider (?), in which case I will ask to check if they are blocking specific IPs.
Also, I would have expected to find somewhere a trace of the token, or something with the name "ckjDC6YMCpeZ0XkXVmOxS3rpzab5jKvQU80phyEtWeI", but I do not find any trace of this, and all subfolders from my "certbot" folder stay empty. That might be another problem, unless certbot removes it when the challenge is unsuccessful or when we run a dry-run ...?

I will check for IP filtering. Many thanks for your precious help and explanations...

1 Like

Is there any other network gear that looks at traffic? Like a router or other firewall device?

Interesting, the Let's Debug test site has no problem reaching you (see here). Although, because of the nature of that test only the primary Let's Encrypt Auth server makes an HTTP request. Two of the four secondary LE servers are not in the USA but usually failures from them are reported as "Secondary" failures in the message. Which is why it looks more like a failure from the Primary center. Still, maybe this particular failure omits the "Secondary" phrase and you have a geographic based firewall somewhere?

Because Let's Debug reaches you I wonder if you repeatedly get this failure. Do you? Because the IP addresses used by the LE Auth server farms rotate often.

You could try without "--dry-run" to use the LE production system. You would likely get different IP addresses. Avoid excessive tests as after like 5 failures/hour you will get blocked for an hour.

The other main thing to try is a different Certificate Authority. Buypass is the easiest and you just replace --dry-run and use --server https://api.buypass.com/acme/directory

You must replace --dry-run as that sets the LE staging environ

To debug just your challenge tokens you can add --debug-challenges -v to the command options. That will pause after making the challenge token files and wait for keyboard input before submitting the cert request to LE. I don't know how well that works in a container but I will leave that to you :slight_smile:

2 Likes

Also keep in mind that most docker compose commands assume you are running on linux or macOS, so when they specify a local mount path as a data volume their example will be part of a host linux/macOS filesystem e.g. your host data volume may in fact be C:\something\else, which can optionally be converted to /c/something/else/

As an aside, there's nothing particularly wrong about using docker on Windows but if you're going to do it them you need to fully understand host volume path mappings. Personally this whole setup sounds quite complicated to me, especially if you are relying on this for production. I'd suggest just setting up a normal linux VM and installing all this stuff (not containerized) would be simpler overall. Note also that you can achieve all of this just using IIS on windows, without docker or nginx.

[edit: re-reading this it sounds a bit grumpy, the point is just that you need to check you host volume mappings are correct.]

Step one is to prove http (TCP port 80) is reaching your host (windows) server without a firewall or router in the way.

3 Likes

All fair points @webprofusion re: structure. Just noting there is no trouble reaching their nginx server.

In fact, the only HTTP failure reaching it that I have seen is in their first post with the "refused" error.

You can test connection yourself just note the unusual way they obfuscated their domain name (see first post, second paragraph).

4 Likes

Thank you very much and sorry for the delay of my reply. I have just accessed the server...
So, good thing is that the problem is now solved thanks to you !
The issue was that I followed the steps proposed in most tutorials, which were missing a critical step:
the command

docker-compose run --rm certbot certonly --webroot --webroot-path /var/www/certbot/ --dry-run -v -d sensordatacrichton2.uk

failed to validate the challenges because NGINX was not started beforehand...
So everything was done correctly by the certbot (writing in the mounted folder was achieved successfully), but nginx wasn't started, so Let's Encrypt could not access it... And you could not see it because after these trials, I always re-started the whole docker-compose so my services would keep working in a degraded mode (http)...

Thank you so much to both of you for your great help, and I am so sorry for the mistake and for taken so much of your time... I will definitely send a message to the tutorials' authors so they add the necessary step : 'docker-compose up -d' then once all services are running (especially nginx, and that certbot is exiting), we can run the certbot again in a dry-run to check the challenges success !

And tumbs up for the certbot container as well, which makes it possible to get certificates on any platform...

3 Likes

Oops :slight_smile: Yes, that was important. No worries. Glad you got it sorted.

4 Likes

Can you describe how you solve this problem and what article/ tutorial you read. Thanks