Certbot failed to authenticate some domains (authenticator: webroot)

Good morning everyone...
I am stuck on creating certificates via certbot:
After running : sudo certbot certonly --webroot -w ./data/certbot/www -d prostatecanceratlas.org -d www.prostatecanceratlas.org

I see that certbot returns a 404 error:

Certbot failed to authenticate some domains (authenticator: webroot). The certificate authority reported these problems:
  Domain: prostatecanceratlas.org
  Type: unauthorized
  Detail: 2a01:4f8:d0a:27bd::1234: Invalid response from http://prostatecanceratlas.org/.well-known/acme-challenge/yiQBPmOxSiyjrR_E6dno1kMbebpyXCthfAJt-WM7Aok: 404

  Domain: www.prostatecanceratlas.org
  Type: unauthorized
  Detail: 2a01:4f8:d0a:27bd::1234: Invalid response from http://www.prostatecanceratlas.org/.well-known/acme-challenge/uVAq_RKl826CGcx18i6CGBdZWuGyqYVVGikmr5xtY60: 404

Hint: The certificate authority failed to download the temporary challenge files created by Certbot. Make sure that the listed domains serve their contents from the --webroot/-w path provided and that the created files can be downloaded from the Internet.

I don't know how to solve this problem....

I will leave you my nginx.config:

server {
    listen 80;
    server_name prostatecanceratlas.org/;
    server_tokens off;

    location /.well-known/acme-challenge/ {
      root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }
}

server {

  listen 443 ssl;
  server_name prostatecanceratlas.org;

  ssl_certificate /etc/letsencrypt/live/prostatecanceratlas.org-0001/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/prostatecanceratlas.org-0001/privkey.pem;

  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  sendfile on;
  default_type application/octet-stream;

  gzip on;
  gzip_http_version 1.1;
  gzip_disable      "MSIE [1-6]\.";
  gzip_min_length   256;
  gzip_vary         on;
  gzip_proxied      expired no-cache no-store private auth;
  gzip_types        text/plain text/css application/json application/javascript application/x-javascript text/xml application/xml application/xml+rss text/javascript;
  gzip_comp_level   9;
  client_max_body_size 64M;

  root /usr/share/nginx/html;

  location /api {
        proxy_pass http://pcaprofiler_api:6867/api;
  }

  location /static/ {
    alias /static/;
    add_header Content-disposition "attachment";
  }


  location / {
    try_files $uri @index;
  }


  location @index {
    add_header Cache-Control no-cache;
    expires 0;
    try_files /index.html =404;
  }


}

And this is my ini-letsencrypt.sh that i use to launch all:

#!/bin/bash

source .env

domains=(${WEB_DOMAIN} www.${WEB_DOMAIN})
rsa_key_size=4096
data_path="./data/certbot"
email="theurillat.team@gmail.com" # Adding a valid address is strongly recommended
staging=${CERT_STAGING} # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi


if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Creating dummy certificate for $domains ..."
path="/etc/letsencrypt/live/$domains"
mkdir -p "$data_path/conf/live/$domains"
docker compose run --rm --entrypoint "\
  openssl req -x509 -nodes -newkey rsa:$rsa_key_size -days 1\
    -keyout '$path/privkey.pem' \
    -out '$path/fullchain.pem' \
    -subj '/CN=localhost'" certbot
echo


echo "### Starting nginx ..."
docker compose up --force-recreate -d pcaprofiler_ui
echo

echo "### Deleting dummy certificate for $domains ..."
docker compose run --rm --entrypoint "\
  rm -Rf /etc/letsencrypt/live/$domains && \
  rm -Rf /etc/letsencrypt/archive/$domains && \
  rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
echo


echo "### Requesting Let's Encrypt certificate for $domains ..."
#Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

# Select appropriate email arg
case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

# Enable staging mode if needed
if [ $staging != "0" ]; then staging_arg="--staging"; fi

docker compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/certbot \
    $staging_arg \
    $email_arg \
    $domain_args \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo

echo "### Reloading nginx ..."
docker compose exec pcaprofiler_ui nginx -s reload

My web server is: nginx/1.20.2

The operating system my web server runs on is:
Distributor ID: Ubuntu
Description: Ubuntu 22.04.1 LTS
Release: 22.04
Codename: jammy

My hosting provider, if applicable, is: hetzner

I can login to a root shell on my machine.

Thanks all fro help!

Does this IP address belong to your server?

Is your webserver listening on IPv6?

2a01:4f8:d0a:27bd::1234 is my IPv6.

The think that is really strange is that if i run the same .sh (init-letsencrypt.sh) with another domain ("pleggit.eu") it work as i aspect!

Yes, it's your ipv6 according to your DNS.

Is your server answering to that ipv6?

You might want to check this: Module ngx_http_core_module

These do not match:

Also this vhost doesn't handle www subdomain.

Edit: Should've read further a bit. I see docker is involved, so the paths could match if mounts are properly configured. But I won't be of great help with docker…

Yhea the server responses to ipv6.

I see two different web serviers:

curl -Ii4 prostatecanceratlas.org
HTTP/1.1 301 Moved Permanently
Server: nginx <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< NGINX via IPv4
Date: Mon, 14 Aug 2023 12:23:42 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://prostatecanceratlas.org/

curl -Ii6 prostatecanceratlas.org
HTTP/1.1 200 OK
Date: Mon, 14 Aug 2023 12:23:47 GMT
Server: Apache <<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<<< APACHE via IPv6
Upgrade: h2c
Connection: Upgrade
Content-Type: text/html; charset=utf-8

Is it wrong? Should only nginx respond as ws?
Could the ws of APACHE be the DATABASE?

Maybe.
It is at least unexpected and more complicated than one would like.

I'd choose one - your choice.
But both IPv4 and IPv6 should be handled by the same ws.

I would not know anything about your system [other than what you have told us here].
But it could be that.
It could also be many other things; There is no way for anyone without access to your system to know.

Maby is my docker-compose.yml can help?

version: '3.8'
services: 
    traefik:
        image: 'traefik:v2.7'
        container_name: "traefik"
        command:
          - "--api.insecure=true"
          - "--providers.docker=true"
          - "--providers.docker.exposedbydefault=false"
          - "--entrypoints.web.address=:$PLUMBER_PORT"
        networks:
          - backend
        expose:
          - "$PLUMBER_PORT:$PLUMBER_PORT"
        labels:
          - "traefik.enable=false"
        volumes:
           - /var/run/docker.sock:/var/run/docker.sock
        restart: always
    plumber :
        env_file: ./.env
        depends_on:
          - mysqldb
        build: 
          context: ./PCaProfiler_R
          dockerfile: Dockerfile
        restart: unless-stopped
        environment:
          - TMPDIR=/pcaprofiler_R/shared/tmp
        networks:
          - backend
        volumes:
          - shared-content:/pcaprofiler_R/shared
        deploy:
           replicas : 2
           labels:
             - "traefik.enable=true"
             - "traefik.http.routers.plumber.rule=Host(`plumber.localhost`)"
             - "traefik.http.routers.plumber.entrypoints=web"
        command: bash -c "mkdir -p /pcaprofiler_R/shared/tmp && chmod -R 777 /pcaprofiler_R/shared/ && cd /pcaprofiler_R/ &&  Rscript serve.R"
    pcaprofiler_api:
        env_file: ./.env
        # command: tail -F anything # For maintainance
        depends_on:
          - mysqldb
        build: 
          context: ./PCaProfiler_API
          dockerfile: Dockerfile
        restart: unless-stopped
        expose:
          - $NODE_LOCAL_PORT:$NODE_DOCKER_PORT
        networks:
          - backend
        volumes:
          - shared-content:/pcaprofiler_api/shared/
          - ./PCaProfiler_API/RawData:/pcaprofiler_api/RawData
    pcaprofiler_ui:
        env_file: ./.env
        tty: true
        depends_on:
          - pcaprofiler_api
        build:
          context: ./PCaProfiler_UI
          dockerfile: Dockerfile
        ports:
          - $ANGULAR_LOCAL_PORT:$ANGULAR_DOCKER_PORT
          - 443:443
        networks:
          - frontend
          - backend
        volumes: 
          - ./data/certbot/conf:/etc/letsencrypt
          - ./data/certbot/www:/var/www/certbot
          - ./nginx/log:/var/log/nginx
          - ./static:/static/
    mysqldb:
        image: mysql:5.7
        restart: unless-stopped
        environment:
          - MYSQL_ROOT_PASSWORD=$MYSQLDB_ROOT_PASSWORD
          - MYSQL_DATABASE=$MYSQLDB_DATABASE
        expose:
          - $MYSQLDB_LOCAL_PORT:$MYSQLDB_DOCKER_PORT
        networks:
          - backend
        volumes:
          - db:/var/lib/mysql
    phpmyadmin:
        image: phpmyadmin
        restart: unless-stopped       
        ports:
          - "8080:80"
        environment:
            PMA_ARBITRARY : 1
            PMA_HOST : $MYSQLDB_HOST
            PMA_PORT : $MYSQLDB_LOCAL_PORT
        networks:
          - backend
    certbot:
      image: certbot/certbot
      restart: unless-stopped
      volumes:
        - ./data/certbot/conf:/etc/letsencrypt
        - ./data/certbot/www:/var/www/certbot
      entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
  
volumes:
    shared-content:
    db:
networks:
    backend:
    frontend:

You know that... Traefik Let's Encrypt Documentation - Traefik

Is possible that i reach the limit rate?
And why if i launch the same sh with the domain = pleggit.eu all works?

You would know that. The error message is pretty explicit if you do.

Also "failed to authenticate" has a rate limit, but it's a pretty fast one.

Normally you need an IPv6 listen like below. That Apache responds to IPv6 is odd so not sure this is needed or would help. A similar one for your listen 443 block too. And, you don't use a trailing slash on server name.

    listen 80;
    listen [::]:80;
    server_name prostatecanceratlas.org;

pleggit only has an A record in DNS (no IPv6 AAAA)

So a solution can be to delete the AAAA row in the dns setting... I'll try tomorrow!

Thank you @MikeMcQ and any other!
The problem was that i set IPv6 on my DNS

You should set IPv6 in your DNS, if you have the right one.