Docker + Nginx + Certbot (Webroot) – Challenge Fails with 404 (No File Appears)

Dave.Sintra · March 24, 2025, 10:51am

Hi all,

I’ve been working to set up HTTPS using Certbot (with the webroot method) in a Dockerized environment, but I’m repeatedly hitting a wall with failed HTTP-01 challenges returning a 404.

Environment:

VPS (8GB RAM, 8 Cores)
Ubuntu 22.04
Docker + Docker Compose
Nginx (reverse proxy)
FastAPI + Gunicorn backend
Certbot (Docker image)
Let’s Encrypt for SSL

Docker Volumes:

I am using named Docker volumes for sharing challenge files:

volumes:
  postgres_data:
  certbot_data:
  certbot_logs:
  certbot_www:
  shared_data:

Relevant docker-compose.yml snippets:

nginx:
  image: nginx:latest
  container_name: nginx
  ports:
    - "80:80"
    - "443:443"
  volumes:
    - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
    - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
    - certbot_data:/etc/letsencrypt
    - certbot_logs:/var/log/letsencrypt
    - certbot_www:/var/www/certbot
entrypoint: ["/bin/sh", "/etc/nginx/entrypoint.sh"]

certbot:
  image: certbot/certbot
  container_name: certbot
  volumes:
    - certbot_data:/etc/letsencrypt
    - certbot_logs:/var/log/letsencrypt
    - certbot_www:/var/www/certbot
entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"

volumes:
  postgres_data:
  certbot_data:
  certbot_logs:
  certbot_www:
  shared_data:

default.conf for Nginx:


server {
    listen 80;
    server_name datainsightsai.org www.datainsightsai.org;

    location /.well-known/acme-challenge/ {
        alias /var/www/certbot/;
        try_files $uri =404;
    }
}

And my nginx/nginx.conf

# /nginx/nginx.conf
user  nginx;
worker_processes auto;
error_log  /var/log/nginx/error.log warn;
pid        /var/run/nginx.pid;
events {
    worker_connections 4096;
    use epoll;
    multi_accept on;
}

http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;
    sendfile        on;
    keepalive_timeout  65;
    server_tokens off;
    gzip on;
    gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;
    include /etc/nginx/conf.d/*.conf;
}

And my little entry script to ensure directories exist :

#!/bin/sh
# Ensure the Certbot challenge directory exists
# init-entrypoint.sh
mkdir -p /var/www/certbot/.well-known/acme-challenge
chown -R nginx:nginx /var/www/certbot
chmod -R 755 /var/www/certbot

# Start Nginx
exec nginx -g "daemon off;"

What Works:

Manual test files placed inside /var/www/certbot/.well-known/acme-challenge/ are served successfully via curl http://datainsightsai.org/.well-known/acme-challenge/testfile

Nginx is running fine, correctly proxies the FastAPI app on root, and serves test challenge files when placed manually.

What Fails:

When I run the command:

docker run --rm -it \
  -v certbot_data:/etc/letsencrypt \
  -v certbot_www:/var/www/certbot \
  certbot/certbot certonly --webroot -w /var/www/certbot \
  --email dave@datainsightsai.org \
  -d datainsightsai.org -d www.datainsightsai.org \
  --agree-tos --no-eff-email --debug --verbose

I consistently get:
Invalid response from http://datainsightsai.org/.well-known/acme-challenge/... : 404 Not Found

AND ... While watching the directory in real time using::
watch -n 0.5 "docker exec -it nginx ls -l /var/www/certbot/.well-known/acme-challenge/"
... no challenge files appear during the certbot run.

MY ERROR:

deployuser@datainsightsai:~/datainsightsai$ docker run --rm -it \
  -v certbot_data:/etc/letsencrypt \
  -v certbot_www:/var/www/certbot \
  certbot/certbot certonly --webroot -w /var/www/certbot \
  --email dave@datainsightsai.org \
  -d datainsightsai.org -d www.datainsightsai.org \
  --agree-tos --no-eff-email --debug --verbose
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Requesting a certificate for datainsightsai.org and www.datainsightsai.org
Performing the following challenges:
http-01 challenge for datainsightsai.org
http-01 challenge for www.datainsightsai.org
Using the webroot path /var/www/certbot for all unmatched domains.
Waiting for verification...
Challenge failed for domain datainsightsai.org
Challenge failed for domain www.datainsightsai.org
http-01 challenge for datainsightsai.org
http-01 challenge for www.datainsightsai.org

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: datainsightsai.org
  Type:   unauthorized
  Detail: 148.135.60.74: Invalid response from http://datainsightsai.org/.well-known/acme-challenge/I0RneGKj0JuV4iXdQLzqQ8ah9scuSeTt2J81lCZZJEo: 404

  Domain: www.datainsightsai.org
  Type:   unauthorized
  Detail: 148.135.60.74: Invalid response from http://www.datainsightsai.org/.well-known/acme-challenge/r8QFtJoxJHv4InYx57I7lhY08SHexrxQkZFdRLFJCx0: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Cleaning up challenges
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot', 'console_scripts', 'certbot')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1871, in main
    return config.func(config, plugins)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 1577, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/main.py", line 142, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 513, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 423, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/client.py", line 492, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/opt/certbot/src/certbot/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
deployuser@datainsightsai:~/datainsightsai$

Things I’ve Tried:

Ensuring volumes are correctly mounted and shared
Switching from root to alias in Nginx config
Using alternative webroot path (/var/www/certbot-test)
Testing access to challenge file manually via browser and curl
Verified that DNS is pointing correctly
Nginx logs show no errors for HTTP traffic on port 80
UFW ports 80 and 443 are open
Temporarily removed HTTPS from config to allow Certbot to complete first HTTP-01 and avoid conflicts.

Key Questions:

Where does Certbot actually place the challenge file? Should I see it in /var/www/certbot/.well-known/acme-challenge/ during the process?
** Is alias or root preferred for the challenge location in this case?**
Does Certbot create and delete the challenge files too fast for me to see?
Is there a better pattern for managing Certbot challenges in Dockerized setups like this?

I’ve read many similar posts, but none have resolved the issue. Any help or ideas would be hugely appreciated

Thanks for any help.

Dave

MikeMcQ · March 24, 2025, 1:15pm

Yes, with -w /var/www/certbot

root !

Maybe. If you can run Certbot interactively you could use --debug-challenges. Not sure how that would work given your container workflow

Managing containers is complicated and harder to debug.

Given you use Cloudflare as your DNS provider you might look at using a DNS Challenge instead. Let's Encrypt checks a TXT record in your DNS rather than using HTTP requests. See: Welcome to certbot-dns-cloudflare’s documentation! — certbot-dns-cloudflare 0 documentation

You might even just run Certbot in the host and share /etc/letsencrypt with whatever container(s) need the cert.

os11k · March 24, 2025, 2:32pm

Hi @Dave.Sintra !

I noticed the first issue in your configuration: /etc/nginx/entrypoint.sh isn’t mounted to the nginx container.

I just tried this code, which is basically a copy-paste of your nginx setup:

services:
  nginx:
    image: nginx:latest
    container_name: nginx
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./nginx/nginx.conf:/etc/nginx/nginx.conf:ro
      - ./nginx/default.conf:/etc/nginx/conf.d/default.conf:ro
      - certbot_data:/etc/letsencrypt
      - certbot_logs:/var/log/letsencrypt
      - certbot_www:/var/www/certbot
    entrypoint: ["/bin/sh", "/etc/nginx/entrypoint.sh"]

volumes:
  certbot_data:
  certbot_logs:
  certbot_www:

But the nginx container doesn’t start because it’s missing entrypoint.sh . Here’s what I got from the logs:

docker logs nginx
/bin/sh: 0: cannot open /etc/nginx/entrypoint.sh: No such file

To fix this, you need to add the following line to the docker-compose.yml under the nginx volumes:

      - ./nginx/entrypoint.sh:/etc/nginx/entrypoint.sh:ro

Dave.Sintra · March 25, 2025, 12:28pm

Thanks for your advice.
The crucial part was that docker adds the namespace so
I needed : docker run --rm -it
-v datainsightsai_certbot_data:/etc/letsencrypt \
and corresponding entries in the .yml file.

I hope that helps anyone else.

system · April 24, 2025, 12:28pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Cerbot challenge failed Help	7	5570	November 22, 2019
Trouble using certbot with nginx on docker Help	4	2912	May 10, 2018
Using Certbot in Docker Containers - Cannot Pass HTTP Challenge - The client lacks sufficient authorization Help	3	1734	June 8, 2017
Invalid response Docker Help	6	2844	September 18, 2019
Challenge failed when creating certificate: Unauthorized Help	10	3683	April 22, 2020

Docker + Nginx + Certbot (Webroot) – Challenge Fails with 404 (No File Appears)

What Works:

What Fails:

Things I’ve Tried:

Key Questions:

Related topics