I got /.well-known/acme-challenge does not exist or is not a directory

[Khamonluckssa]

My domain is: portal.comp-moto.com

I ran this command: I run this init-letsencrypt.sh on server

It produced this output:

Creating dummy certificate for portal.comp-moto.com ...

Generating a RSA private key

............................................................................++++

........................................................................................................................................................................................................................................................................................................................++++

writing new private key to '/etc/letsencrypt/live/portal.comp-moto.com/privkey.pem'

---

Starting nginx ...

Recreating comp_moto_redis ... done

Recreating comp_moto_app ... done

Recreating comp_moto_web ... done

Deleting dummy certificate for portal.comp-moto.com ...

Requesting Let's Encrypt certificate for portal.comp-moto.com ...

/.well-known/acme-challenge does not exist or is not a directory

Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /tmp/tmpivsrd3cl/log or re-run Certbot with -v for more details.

Reloading nginx ...

2021/12/16 10:05:52 [notice] 12#12: signal process started

My web server is (include version): I use Nginx

The operating system my web server runs on is (include version): ubuntu 16.04

docker-compose file

version: "3"

volumes:
  data-uploads:
  packs:
services:
  app:
    build:
      context: .
      dockerfile: ./dockers/app/Dockerfile
    container_name: comp_moto_app
    restart: unless-stopped
    ports:
      - 3000:3000
    volumes:
      - .:/app
      - /app/node_modules
      - /app/public/assets
      - /app/public/packs
    depends_on:
      - redis
    environment:
      - REDIS_URL_CACHING=redis://redis@redis:6379/0
  db:
    image: "postgres:14.1-alpine"
    container_name: comp_moto_db
    environment:
      POSTGRES_USER: ${DOCKER_DB_USER}
      POSTGRES_PASSWORD: ${DOCKER_DB_PASSWORD}
      POSTGRES_DB: ${DOCKER_DB_NAME}
    volumes:
      - /docker_data/comp_moto_app/postgres:/var/lib/postgresql/data
    ports:
      - 5432:5432
  nginx:
    build:
      context: .
      dockerfile: ./dockers/web/Dockerfile
    container_name: comp_moto_web
    restart: unless-stopped
    ports:
      - 80:80
      - 443:443
    depends_on:
      - app
    volumes:
      - ./dockers/web/nginx.conf:/etc/nginx/conf.d/default.conf
      - ./public:/app/public
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
  certbot:
    image: certbot/certbot
    restart: unless-stopped
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
  redis:
    image: redis:6.2.6-alpine
    container_name: comp_moto_redis
    command: redis-server --requirepass ${REDIS_PASSWORD}
    volumes:
      - /docker_data/comp_moto/production/redis:/data
      - ./dockers/redis/redis.conf:/usr/local/etc/redis/redis.conf
    ports:
      - 6379:6379
  sidekiq:
    build:
      context: .
      dockerfile: ./dockers/sidekiq/Dockerfile
    container_name: comp_moto_sidekiq
    command: bundle exec sidekiq
    depends_on:
      - db
      - redis
    volumes:
       - data-uploads:/app/public/uploads

this is Nginx file

upstream rails_app {  
   server app:3000;
} 
server {  
  # define your domain  
  listen 80;
  server_name portal.comp-moto.com www.portal.comp-moto.com;
  server_tokens off;

  location /.well-known/acme-challenge/ {
    root /var/www/certbot;
  }
}
server {
  listen 443 ssl;
  server_name portal.comp-moto.com;
  server_tokens off;

  proxy_read_timeout 1800;
  proxy_connect_timeout 1800;
  proxy_send_timeout 1800; 

  client_max_body_size 20M;
  client_body_buffer_size 20M;
  client_body_timeout 1800;
  client_header_timeout 1800;
  keepalive_timeout 1800;
  send_timeout 1800;

  # define the public application root
  root   /app/public;
  index  index.html;
   
  ssl_certificate /etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/portal.comp-moto.com/privkey.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  return 301 https://www.$host$request_uri;
}

server {
  listen 443 ssl;
  server_name www.portal.comp-moto.com;
  server_tokens off;

  proxy_read_timeout 1800;
  proxy_connect_timeout 1800;
  proxy_send_timeout 1800; 
  
  client_max_body_size 20M;
  client_body_buffer_size 20M;
  client_body_timeout 1800;
  client_header_timeout 1800;
  keepalive_timeout 1800;
  send_timeout 1800;

  # define the public application root
  root   /app/public;
  index  index.html;

  # serve static (compiled) assets directly if they exist (for rails production)
  location ~ ^/(assets|images|javascripts|stylesheets|swfs|system)/ {
    try_files $uri @rails;

    access_log off;
    gzip_static on; # to serve pre-gzipped version

    expires max;
    add_header Cache-Control public;

    # Some browsers still send conditional-GET requests if there's a
    # Last-Modified header or an ETag header even if they haven't
    # reached the expiry date sent in the Expires header.
    add_header Last-Modified "";
    add_header ETag "";
    break;
  }

  # send non-static file requests to the app server
  location / {
    try_files $uri @rails;
  }

  ssl_certificate /etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/portal.comp-moto.com/privkey.pem;
  include /etc/letsencrypt/options-ssl-nginx.conf;
  ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

  location @rails {
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $http_host;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Forwarded-Ssl on;
    proxy_set_header  X-Forwarded-Port $server_port;
    proxy_set_header  X-Forwarded-Host $host;
    proxy_redirect off;
    proxy_pass http://rails_app;
  }
}

what I missed?

[rg305]

Hi @Khamonluckssa and welcome to the LE community forum

@Khamonluckssa

init-letsencrypt.sh

Please show that file.
IIRC, it was very poorly written and should NOT be used.

That said, we'll still have to address the problem:

@Khamonluckssa

/.well-known/acme-challenge does not exist or is not a directory

So, did you run the command within the Docker image?

[Khamonluckssa]

I ran this file and it goes well-known does not exist, this file should create certificate, isn't it?

#!/bin/bash

if ! [ -x "$(command -v docker-compose)" ]; then
  echo 'Error: docker-compose is not installed.' >&2
  exit 1
fi

domains=(portal.comp-moto.com www.portal.comp-moto.com)
rsa_key_size=4096
data_path="./data/certbot"
email="khamonluck.ang@gmail.com" # Adding a valid address is strongly recommended
staging=0 # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi


if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Creating dummy certificate for $domains ..."
path="/etc/letsencrypt/live/$domains"
mkdir -p "$data_path/conf/live/$domains"
docker-compose run --rm --entrypoint "\
  openssl req -x509 -nodes -newkey rsa:$rsa_key_size -days 1\
    -keyout '$path/privkey.pem' \
    -out '$path/fullchain.pem' \
    -subj '/CN=localhost'" certbot
echo


echo "### Starting nginx ..."
docker-compose up --force-recreate -d nginx
echo

echo "### Deleting dummy certificate for $domains ..."
docker-compose run --rm --entrypoint "\
  rm -Rf /etc/letsencrypt/live/$domains && \
  rm -Rf /etc/letsencrypt/archive/$domains && \
  rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
echo


echo "### Requesting Let's Encrypt certificate for $domains ..."
#Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

# Select appropriate email arg
case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

# Enable staging mode if needed
if [ $staging != "0" ]; then staging_arg="--staging"; fi

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /.well-known/acme-challenge \
    --staging \
    --email khamonluck.ang@gmail.com \
    -d portal.comp-moto.com www.portal.comp-moto.com \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo

echo "### Reloading nginx ..."
docker-compose exec nginx nginx -s reload

[MikeMcQ]

@Khamonluckssa

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /.well-known/acme-challenge \
    --staging \
    --email khamonluck.ang@gmail.com \
    -d portal.comp-moto.com www.portal.comp-moto.com \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --force-renewal" certbot
echo

The -w folder should be the folder named by the root setting in your nginx server block definition. Is that even accessible when running this script?

Also, this command looks wrong.

1. The line with -d should not have a space between domains. Either a comma and no spaces or a quoted string
2. --force-renewal is problem generally and especially in docker - soon will hit rate limits. Use --dry-run when testing
3. Not sure what that last certbot is - probably just remove it so end the command after --agree-tos with a double-quote and not a \

[Khamonluckssa]

now this is my script file

#!/bin/bash

if ! [ -x "$(command -v docker-compose)" ]; then
  echo 'Error: docker-compose is not installed.' >&2
  exit 1
fi

domains=(portal.comp-moto.com www.portal.comp-moto.com)
rsa_key_size=4096
data_path="./data/certbot"
email="khamonluck.ang@gmail.com" # Adding a valid address is strongly recommended
staging=0 # Set to 1 if you're testing your setup to avoid hitting request limits

if [ -d "$data_path" ]; then
  read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
  if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
    exit
  fi
fi


if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
  echo "### Downloading recommended TLS parameters ..."
  mkdir -p "$data_path/conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
  curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
  echo
fi

echo "### Creating dummy certificate for $domains ..."
path="/etc/letsencrypt/live/$domains"
mkdir -p "$data_path/conf/live/$domains"
docker-compose run --rm --entrypoint "\
  openssl req -x509 -nodes -newkey rsa:$rsa_key_size -days 1\
    -keyout '$path/privkey.pem' \
    -out '$path/fullchain.pem' \
    -subj '/CN=localhost'" certbot
echo


echo "### Starting nginx ..."
docker-compose up --force-recreate -d nginx
echo

echo "### Deleting dummy certificate for $domains ..."
docker-compose run --rm --entrypoint "\
  rm -Rf /etc/letsencrypt/live/$domains && \
  rm -Rf /etc/letsencrypt/archive/$domains && \
  rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
echo


echo "### Requesting Let's Encrypt certificate for $domains ..."
#Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
  domain_args="$domain_args -d $domain"
done

# Select appropriate email arg
case "$email" in
  "") email_arg="--register-unsafely-without-email" ;;
  *) email_arg="--email $email" ;;
esac

# Enable staging mode if needed
if [ $staging != "0" ]; then staging_arg="--staging"; fi

docker-compose run --rm --entrypoint "\
  certbot certonly --webroot -w /var/www/certbot \
    $staging_arg \
    $email_arg \
    $domain_args \
    --rsa-key-size $rsa_key_size \
    --agree-tos \
    --dry-run" 
echo

echo "### Reloading nginx ..."
docker-compose exec nginx nginx -s reload

and I got error here

### Creating dummy certificate for portal.comp-moto.com ...
Generating a RSA private key
..................................................................................................................................................................................................++++
............................................................................................................................++++
writing new private key to '/etc/letsencrypt/live/portal.comp-moto.com/privkey.pem'
-----

### Starting nginx ...
Recreating comp_moto_redis ... done
Recreating comp_moto_app   ... done
Recreating comp_moto_web   ... done

### Deleting dummy certificate for portal.comp-moto.com ...

### Requesting Let's Encrypt certificate for portal.comp-moto.com ...
Run a one-off command on a service.

For example:

    $ docker-compose run web python manage.py shell

By default, linked services will be started, unless they are already
running. If you do not want to start linked services, use
`docker-compose run --no-deps SERVICE COMMAND [ARGS...]`.

Usage:
    run [options] [-v VOLUME...] [-p PORT...] [-e KEY=VAL...] [-l KEY=VALUE...]
        SERVICE [COMMAND] [ARGS...]

Options:
    -d, --detach          Detached mode: Run container in the background, print
                          new container name.
    --name NAME           Assign a name to the container
    --entrypoint CMD      Override the entrypoint of the image.
    -e KEY=VAL            Set an environment variable (can be used multiple times)
    -l, --label KEY=VAL   Add or override a label (can be used multiple times)
    -u, --user=""         Run as specified username or uid
    --no-deps             Don't start linked services.
    --rm                  Remove container after run. Ignored in detached mode.
    -p, --publish=[]      Publish a container's port(s) to the host
    --service-ports       Run command with the service's ports enabled and mapped
                          to the host.
    --use-aliases         Use the service's network aliases in the network(s) the
                          container connects to.
    -v, --volume=[]       Bind mount a volume (default [])
    -T                    Disable pseudo-tty allocation. By default `docker-compose run`
                          allocates a TTY.
    -w, --workdir=""      Working directory inside the container

### Reloading nginx ...
Error response from daemon: Container 497a3f8898d68eb3c2f8df15a2acf6f6e73d2ae9e40b43d0b44f988b1f81151f is restarting, wait until the container is running

and my Nginx container log show this error

2021/12/17 04:36:09 [crit] 1#1: pread() "/etc/letsencrypt/options-ssl-nginx.conf" failed (21: Is a directory)
nginx: [crit] pread() "/etc/letsencrypt/options-ssl-nginx.conf" failed (21: Is a directory)

[Khamonluckssa]

I can not build docker because it will can not file letsencrypt file and I can not run my script, it wait until the container is running. what should I do?

[Khamonluckssa]

@rg305 How can I run within image? I'm a new for docker as well

[Khamonluckssa]

@MikeMcQ this is my script

[MikeMcQ]

oh my bad that trailing certbot was the container name you need that

[Khamonluckssa]

Can you help me with this? I got this @MikeMcQ

### Creating dummy certificate for portal.comp-moto.com ...
Generating a RSA private key
.....................................................................................+++++
............+++++
writing new private key to '/etc/letsencrypt/live/portal.comp-moto.com/privkey.pem'
-----

### Starting nginx ...
Recreating comp_moto_redis ... done
Recreating comp_moto_app   ... done
Recreating comp_moto_web   ... done

### Deleting dummy certificate for portal.comp-moto.com ...

### Requesting Let's Encrypt certificate for portal.comp-moto.com ...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for portal.comp-moto.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: portal.comp-moto.com
  Type:   unauthorized
  Detail: Invalid response from https://portal.comp-moto.com/.well-known/acme-challenge/o2oEWSywu0_dYpTEunVSIU3QnSmIEvfvX8y-KQQ--gg [103.86.51.196]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

### Reloading nginx ...
2021/12/17 05:23:04 [emerg] 12#12: cannot load certificate "/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

and this is my script

#!/bin/bash

if ! [ -x "$(command -v docker-compose)" ]; then
    echo 'Error: docker-compose is not installed.' >&2
      exit 1
    fi

    domains=(portal.comp-moto.com)
    rsa_key_size=4096
    data_path="./data/certbot"
    email="khamonluck.ang@gmail.com" # Adding a valid address is strongly recommended
    staging=0 # Set to 1 if you're testing your setup to avoid hitting request limits

    if [ -d "$data_path" ]; then
        read -p "Existing data found for $domains. Continue and replace existing certificate? (y/N) " decision
          if [ "$decision" != "Y" ] && [ "$decision" != "y" ]; then
          exit
            fi
          fi


          if [ ! -e "$data_path/conf/options-ssl-nginx.conf" ] || [ ! -e "$data_path/conf/ssl-dhparams.pem" ]; then
              echo "### Downloading recommended TLS parameters ..."
          mkdir -p "$data_path/conf"
            curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot-nginx/certbot_nginx/_internal/tls_configs/options-ssl-nginx.conf > "$data_path/conf/options-ssl-nginx.conf"
              curl -s https://raw.githubusercontent.com/certbot/certbot/master/certbot/certbot/ssl-dhparams.pem > "$data_path/conf/ssl-dhparams.pem"
                echo
        fi

        echo "### Creating dummy certificate for $domains ..."
        path="/etc/letsencrypt/live/$domains"
        mkdir -p "$data_path/conf/live/$domains"
      docker-compose run --rm --entrypoint "\
          openssl req -x509 -nodes -newkey rsa:1024 -days 1\
              -keyout '$path/privkey.pem' \
                  -out '$path/fullchain.pem' \
                -subj '/CN=localhost'" certbot
      echo


      echo "### Starting nginx ..."
    docker-compose up --force-recreate -d nginx
    echo

    echo "### Deleting dummy certificate for $domains ..."
  docker-compose run --rm --entrypoint "\
      rm -Rf /etc/letsencrypt/live/$domains && \
        rm -Rf /etc/letsencrypt/archive/$domains && \
          rm -Rf /etc/letsencrypt/renewal/$domains.conf" certbot
  echo


  echo "### Requesting Let's Encrypt certificate for $domains ..."
  #Join $domains to -d args
domain_args=""
for domain in "${domains[@]}"; do
    domain_args="$domain_args -d $domain"
  done

  # Select appropriate email arg
  case "$email" in
      "") email_arg="--register-unsafely-without-email" ;;
        *) email_arg="--email $email" ;;
      esac

      # Enable staging mode if needed
      if [ $staging != "0" ]; then staging_arg="--staging"; fi

      docker-compose run --rm --entrypoint "\
          certbot certonly --webroot -w /var/www/certbot \
        $staging_arg \
            $email_arg \
          $domain_args \
              --rsa-key-size $rsa_key_size \
            --agree-tos \
                --dry-run" certbot
      echo

      echo "### Reloading nginx ..."
      docker-compose exec nginx nginx -s reload

[rg305]

@MikeMcQ

Not sure what that last certbot is

That's part of the "docker-compose" and should remain.
I agree with your findings; as I mentioned in my initial post:

@rg305

it was very poorly written and should NOT be used.

[Khamonluckssa]

I still got this, I'm so confused, I pointed webroot already on Nginx.conf and on docker-compose(Certbot image)

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for portal.comp-moto.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: portal.comp-moto.com
  Type:   unauthorized
  Detail: Invalid response from https://portal.comp-moto.com/.well-known/acme-challenge/7uNbYHVfKwGS0b2g7gf4FFCj51W5vFNs3FLDgEeaGNY [103.86.51.196]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I ran this command

     docker-compose run --rm --entrypoint "\
          certbot certonly --webroot -w /var/www/certbot \
        --staging \
            --email khamonluck.ang@gmail.com \
          -dportal.comp-moto.com \
              --rsa-key-size 4096 \
            --agree-tos \
                --dry-run" certbot
      echo

[rg305]

You need to verfy the webroot:

@Khamonluckssa

/var/www/certbot

[Khamonluckssa]

@Khamonluckssa

Certbot failed to authenticate some domains (authenticator: webroot)

https://answers.webroot.com/Webroot/ukp.aspx?pid=17&app=vw&vw=1&solutionid=3654
Should do something like this?

[rg305]

No.
That is a completely different "webroot".

I'm talking about the use of webroot in the certbot command:

@Khamonluckssa

certbot certonly --webroot -w /var/www/certbot

Also, there is a space missing in this part of it:

@Khamonluckssa

-dportal.comp-moto.com

Which should be:
-d portal.comp-moto.com

[Khamonluckssa]

@Khamonluckssa

certbot certonly --webroot -w /var/www/certbot

I run certbot command inside certbot container and then I got this

/opt/certbot # certbot certonly --webroot -w /var/www/certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Please enter the domain name(s) you would like on your certificate (comma and/or
space separated) (Enter 'c' to cancel): portal.comp-moto.com
Requesting a certificate for portal.comp-moto.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: portal.comp-moto.com
  Type:   unauthorized
  Detail: Invalid response from https://portal.comp-moto.com/.well-known/acme-challenge/Gg4p39t4hYEVTTby41i166mNjwGq-1jh-KqGsyffIUg [103.86.51.196]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

[rg305]

@rg305

You need to verfy the webroot:

@Khamonluckssa

/var/www/certbot

Please show the nginx configuration.
nginx -T

[Khamonluckssa]

@rg305

nginx -T

# configuration file /etc/nginx/nginx.conf:

user  nginx;
worker_processes  auto;

error_log  /var/log/nginx/error.log notice;
pid        /var/run/nginx.pid;


events {
    worker_connections  1024;
}


http {
    include       /etc/nginx/mime.types;
    default_type  application/octet-stream;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  /var/log/nginx/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    keepalive_timeout  65;

    #gzip  on;

    include /etc/nginx/conf.d/*.conf;
}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf.d/default.conf:
upstream rails_app {  
   server app:3000;
} 
server {  
   # define your domain  
   listen 80;
   server_name portal.comp-moto.com;
   server_tokens off;

   location /.well-known/acme-challenge/ {
      root /var/www/certbot;
   }
}

server {
  listen 443 ssl;
  server_name portal.comp-moto.com;
  server_tokens off;

  proxy_read_timeout 1800;
  proxy_connect_timeout 1800;
  proxy_send_timeout 1800; 
  
  client_max_body_size 20M;
  client_body_buffer_size 20M;
  client_body_timeout 1800;
  client_header_timeout 1800;
  keepalive_timeout 1800;
  send_timeout 1800;

  # define the public application root
  root   /app/public;
  index  index.html;

  # serve static (compiled) assets directly if they exist (for rails production)
  location ~ ^/(assets|images|javascripts|stylesheets|swfs|system)/ {
    try_files $uri @rails;

    access_log off;
    gzip_static on; # to serve pre-gzipped version

    expires max;
    add_header Cache-Control public;

    # Some browsers still send conditional-GET requests if there's a
    # Last-Modified header or an ETag header even if they haven't
    # reached the expiry date sent in the Expires header.
    add_header Last-Modified "";
    add_header ETag "";
    break;
  }

  # send non-static file requests to the app server
  location / {
    try_files $uri @rails;
  }

  ssl_certificate /etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem;
  ssl_certificate_key /etc/letsencrypt/live/portal.comp-moto.com/privkey.pem;

  location @rails {
    proxy_set_header  X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header  Host $http_host;
    proxy_set_header  X-Forwarded-Proto $scheme;
    proxy_set_header  X-Forwarded-Ssl on;
    proxy_set_header  X-Forwarded-Port $server_port;
    proxy_set_header  X-Forwarded-Host $host;
    proxy_redirect off;
    proxy_pass http://rails_app;
  }
}

[rg305]

@Khamonluckssa

server {  
   # define your domain  
   listen 80;
   server_name portal.comp-moto.com;
   server_tokens off;

   location /.well-known/acme-challenge/ {
      root /var/www/certbot;
   }
}

That block is somehow NOT being reached:

curl -Ii portal.comp-moto.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 301 Moved Permanently
Date: Fri, 17 Dec 2021 06:52:26 GMT
Content-Type: text/html; charset=iso-8859-1
Content-Length: 278
Connection: keep-alive
Location: https://portal.comp-moto.com/.well-known/acme-challenge/Test_File-1234
Server: Nginx_Rc-Cr
etag: on
X-Cache-Status: BYPASS - 15m desktop

Something else is responding with a redirection.

[Khamonluckssa]

@MikeMcQ

dry-run

I did run certbot certonly --webroot -w /var/www/certbot inside Certbot container again, it was success

root@portal:~/comp_moto# docker exec -it comp_moto_certbot_1 /bin/sh
/opt/certbot #  certbot certonly --webroot -w /var/www/certbot
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Please enter the domain name(s) you would like on your certificate (comma and/or
space separated) (Enter 'c' to cancel): portal.comp-moto.com
Requesting a certificate for portal.comp-moto.com

Successfully received certificate.
Certificate is saved at: /etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem
Key is saved at:         /etc/letsencrypt/live/portal.comp-moto.com/privkey.pem
This certificate expires on 2022-03-17.
These files will be updated when the certificate renews.

NEXT STEPS:
- The certificate will need to be renewed before it expires. Certbot can automatically renew the certificate in the background, but you may need to take steps to enable that functionality. See https://certbot.org/renewal-setup for instructions.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
If you like Certbot, please consider supporting our work by:
 * Donating to ISRG / Let's Encrypt:   https://letsencrypt.org/donate
 * Donating to EFF:                    https://eff.org/donate-le
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

but when i run my script, it still the same error like this

root@portal:~/comp_moto# sudo ./init-letsencrypt.sh
Existing data found for portal.comp-moto.com. Continue and replace existing certificate? (y/N) y
### Creating dummy certificate for portal.comp-moto.com ...
Generating a RSA private key
.................+++++
.................................+++++
writing new private key to '/etc/letsencrypt/live/portal.comp-moto.com/privkey.pem'
-----

### Starting nginx ...
Recreating comp_moto_redis ... done
Recreating comp_moto_app   ... done
Recreating comp_moto_web   ... done

### Deleting dummy certificate for portal.comp-moto.com ...

### Requesting Let's Encrypt certificate for portal.comp-moto.com ...
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for portal.comp-moto.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: portal.comp-moto.com
  Type:   unauthorized
  Detail: During secondary validation: Invalid response from https://portal.comp-moto.com/.well-known/acme-challenge/N-TJnnIUhgo9qsU746hxk4K6XdPMedjNJj8uK78b8yQ [103.154.94.148]: "<!DOCTYPE html>\n<html>\n<head>\n  <title>The page you were looking for doesn't exist (404)</title>\n  <meta name=\"viewport\" content"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

### Reloading nginx ...
2021/12/17 10:03:03 [emerg] 12#12: cannot load certificate "/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/portal.comp-moto.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)