Certbot failing acme-challenge (unauthorized)

Help
1

Hi all,

I'm trying to setup the creation and renewal of ssl-certificates with nginx and Let's Encrypt within Docker Compose using the following tutorial: Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes | by Philipp | Medium

Unfortunately I am having troubles with generating the certificates as certbot fails to pass the acme-challenges. From the errors it seems that the location of the challenge is not reacheable. Therefore I have tried variations in the nginx.conf-file of the part that specifies where the challenge can be found (see below), but no success so far. From what I understand the ports (80, 443) are open as well.

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

Further details listed below. I have ommitted some parts, so let me know if you require more logs or information. Does anyone have an idea what else I can try? Many thanks for your help in advance!

My domain is:
xxxx.xxxx.com

I ran this command:
sudo ./init-letsencrypt.sh

The shell-script can be found here: https://raw.githubusercontent.com/wmnnd/nginx-certbot/master/init-letsencrypt.sh
And relevant params (keep staging=1 while testing):

domains=(xxxx.xxxx.com)
rsa_key_size=4096
data_path="./data/certbot"
email="xxx@xxxx.com" # Adding a valid address is strongly recommended
staging=1 # Set to 1 if you're testing your setup to avoid hitting request limits

nginx.conf:

upstream labeler {
    server web:8000;
}

server {
    listen 80 default_server;
    return 444;
}

server {
    listen 80;
    listen [::]:80;
    server_name xxxx.xxxx.com;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name xxxx.xxxx.com;

    # SSL
    ssl_certificate /etc/letsencrypt/live/xxxx.xxxx.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/xxxx.xxxx.com/privkey.pem;

    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    client_max_body_size 4G;
    keepalive_timeout 5;

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://labeler;
    }

}

docker-compose.yml:

version: "3.8"
   
services:
  (... db and web service ...)
  nginx:
    build:
      context: .
      dockerfile: Dockerfile.nginx
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    ports:
      - 80:80
      - 443:443
    depends_on:
      - web
  certbot:
    image: certbot/certbot
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
      - ./log/letsencrypt:/var/log/letsencrypt

It produced this output:

Downloading recommended TLS parameters ...

Creating dummy certificate for xxxx.xxxx.com ...

Creating network "labeler_ssl_default" with the default driver
Creating labeler_ssl_certbot_run ... done
Generating a RSA private key
.......................................................................++++
.........................................................++++
writing new private key to '/etc/letsencrypt/live/xxxx.xxxx.com/privkey.pem'
-----

### Starting nginx ...
Creating labeler_ssl_db_1 ... done
Creating labeler_ssl_web_1 ... done
Creating labeler_ssl_nginx_1 ... done

### Deleting dummy certificate for xxxx.xxxx.com ...
Creating labeler_ssl_certbot_run ... done

### Requesting Let's Encrypt certificate for xxxx.xxxx.com ...
Creating labeler_ssl_certbot_run ... done
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Would you be willing, once your first certificate is successfully issued, to
share your email address with the Electronic Frontier Foundation, a founding
partner of the Let's Encrypt project and the non-profit organization that
develops Certbot? We'd like to send you email about our work encrypting the web,
EFF news, campaigns, and ways to support digital freedom.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: N
Account registered.
Requesting a certificate for xxxx.xxxx.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: xxxx.xxxx.com
  Type:   unauthorized
  Detail: Invalid response from https://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA [xxx.xxx.xxx.xxx]: "\n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Not Found</title>\n</head>\n<body>\n  <h1>Not Found</h1><p>The requested resource"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
ERROR: 1

letsenscrypt-log (let me know if you need more as the actual log is larger):

2021-06-15 08:12:09,338:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/68900668 HTTP/1.1" 200 1628
2021-06-15 08:12:09,339:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Tue, 15 Jun 2021 08:12:09 GMT
Content-Type: application/json
Content-Length: 1628
Connection: keep-alive
Boulder-Requester: 19908234
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0002tjlTExLwv4QVd8EzEb6xOE1y8--YnxeWcgS8xc1kDXk
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "xxxx.xxxx.com"
  },
  "status": "invalid",
  "expires": "2021-06-22T08:12:03Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from https://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA [xxx.xxx.xxx.xxx]: \"\\n\u003c!doctype html\u003e\\n\u003chtml lang=\\\"en\\\"\u003e\\n\u003chead\u003e\\n  \u003ctitle\u003eNot Found\u003c/title\u003e\\n\u003c/head\u003e\\n\u003cbody\u003e\\n  \u003ch1\u003eNot Found\u003c/h1\u003e\u003cp\u003eThe requested resource\"",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/68900668/CJxahw",
      "token": "D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA",
      "validationRecord": [
        {
          "url": "http://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA",
          "hostname": "xxxx.xxxx.com",
          "port": "80",
          "addressesResolved": [
            "xxx.xxx.xxx.xxx"
          ],
          "addressUsed": "xxx.xxx.xxx.xxx"
        },
        {
          "url": "https://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA",
          "hostname": "xxxx.xxxx.com",
          "port": "443",
          "addressesResolved": [
            "xxx.xxx.xxx.xxx"
          ],
          "addressUsed": "xxx.xxx.xxx.xxx"
        }
      ],
      "validated": "2021-06-15T08:12:04Z"
    }
  ]
}
2021-06-15 08:12:09,340:DEBUG:acme.client:Storing nonce: 0002tjlTExLwv4QVd8EzEb6xOE1y8--YnxeWcgS8xc1kDXk
2021-06-15 08:12:09,340:INFO:certbot._internal.auth_handler:Challenge failed for domain xxxx.xxxx.com
2021-06-15 08:12:09,340:INFO:certbot._internal.auth_handler:http-01 challenge for xxxx.xxxx.com
2021-06-15 08:12:09,341:DEBUG:certbot.display.util:Notifying user: 
Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: xxxx.xxxx.com
  Type:   unauthorized
  Detail: Invalid response from https://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA [xxx.xxx.xxx.xxx]: "\n<!doctype html>\n<html lang=\"en\">\n<head>\n  <title>Not Found</title>\n</head>\n<body>\n  <h1>Not Found</h1><p>The requested resource"

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

nginx-log:

nginx_1    | /docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
nginx_1    | /docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
nginx_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
nginx_1    | 10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
nginx_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
nginx_1    | /docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
nginx_1    | /docker-entrypoint.sh: Configuration complete; ready for start up
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: using the "epoll" event method
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: nginx/1.21.0
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: built by gcc 10.2.1 20201203 (Alpine 10.2.1_pre1)
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: OS: Linux 5.4.0-48-generic
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: start worker processes
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: start worker process 21
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: start worker process 22
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: start worker process 23
nginx_1    | 2021/06/15 08:11:50 [notice] 1#1: start worker process 24
nginx_1    | 18.196.102.134 - - [15/Jun/2021:08:12:05 +0000] "GET /.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx_1    | 18.196.102.134 - - [15/Jun/2021:08:12:05 +0000] "GET /.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA HTTP/1.1" 404 179 "http://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx_1    | 66.133.109.36 - - [15/Jun/2021:08:12:05 +0000] "GET /.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx_1    | 3.143.223.150 - - [15/Jun/2021:08:12:05 +0000] "GET /.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA HTTP/1.1" 301 169 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx_1    | 3.143.223.150 - - [15/Jun/2021:08:12:06 +0000] "GET /.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA HTTP/1.1" 404 179 "http://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
nginx_1    | 66.133.109.36 - - [15/Jun/2021:08:12:06 +0000] "GET /.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA HTTP/1.1" 404 179 "http://xxxx.xxxx.com/.well-known/acme-challenge/D6IBfwJy0Ja8eahYrQxSYsd49YERRZ43wRrmxlN0IcA" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

My web server is (include version):
nginx/1.21.0

The operating system my web server runs on is (include version):
Alpine Linux 3.13

My hosting provider, if applicable, is:
transip

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.16.0

2

Since the failed request is to HTTPS, then the HTTP location block missed the request.
[also corroborated by the 301s in the log shown]
Syntax problem? [not likely, nginx would gripe]
Programmatic/logic/flow problem? [probably some overlap/oversight]
OR Did you forget to restart/reload nginx?
What says: nginx -t

If all still fails, just post the full nginx config here:
nginx -T
[more eyes can find the problem that much quicker]

1 Like
3

Hi rg305, thank you for your reply and suggestions. I hope it is indeed a Programmatic/logic/flow problem, but I cannot seem to figure where :slight_smile:

So the full nginx.conf:

upstream labeler {
    server web:8000;
}

server {
    listen 80 default_server;
    return 444;
}

server {
    listen 80;
    listen [::]:80;
    server_name xxxx.xxxx.com;

    location ^~ /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name xxxx.xxxx.com;

    # SSL
    ssl_certificate /etc/letsencrypt/live/xxxx.xxxx.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/xxxx.xxxx.com/privkey.pem;

    include /etc/letsencrypt/options-ssl-nginx.conf;
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem;

    client_max_body_size 4G;
    keepalive_timeout 5;

    location /static/ {
        alias /app/static/;
    }

    location /images/ {
        alias /app/images/;
    }

    location / {
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;
        proxy_redirect off;
        proxy_pass http://labeler;
    }

}

Running docker-compose exec nginx nginx -T is throwing errors as it cannot find the ssl_certificate and ssl_certificate_key:

2021/06/15 22:16:06 [emerg] 26#26: cannot load certificate "/etc/letsencrypt/live/xxxx.xxxx.com/fullchain.pem": BIO_new_file() failed 
(SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/xxxx.xxxx.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/xxxx.xxxx.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/xxxx.xxxx.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: configuration file /etc/nginx/nginx.conf test failed

Running nginx -T in a separate container with docker run --rm -it <nginx_container_name> nginx -T doesn't really work as it starts complaining about the upstream block (which requires another container to be up?). I've tried to change web:8000 into nginx:8000 and nginx:80 as well to doublecheck without success. I see it is also failing the conf-file at /etc/nginx/nginx.conf, which I have not touched.

/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: /etc/nginx/conf.d/default.conf is not a file or does not exist
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2021/06/15 21:42:47 [emerg] 1#1: host not found in upstream "web:8000" in /etc/nginx/conf.d/nginx.conf:2
nginx: [emerg] host not found in upstream "web:8000" in /etc/nginx/conf.d/nginx.conf:2
nginx: configuration file /etc/nginx/nginx.conf test failed

Also the full docker-compose.yml so you can see where web:8000 is coming from:

version: "3.8"
   
services:
  db:
    image: postgis/postgis
    volumes:
      - ./data/db:/var/lib/postgresql/data
    environment:
      - POSTGRES_DB=XXXX
      - POSTGRES_USER=XXXX
      - POSTGRES_PASSWORD=XXXX
      - POSTGRES_PORT=5432
    ports:
      - "7432:5432"
  web:
    build:
      context: .
      dockerfile: Dockerfile
    command: gunicorn labeler.wsgi:application --bind 0.0.0.0:8000
    volumes:
      - static_volume:/app/static
      - image_volume:/app/images
    expose:
      - 8000
    depends_on:
      - db
  nginx:
    build:
      context: .
      dockerfile: Dockerfile.nginx
    volumes:
      - static_volume:/app/static
      - image_volume:/app/images
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
    ports:
      - 80:80
      - 443:443
    depends_on:
      - web
  certbot:
    image: certbot/certbot
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot
      - ./log/letsencrypt:/var/log/letsencrypt

volumes:
  static_volume:
  image_volume:

Note that nginx was starting up fine without all the ssl-parameters and certbot.

4

If that is the full nginx config, it (is rather small and) doesn't appear to contain any coding errors.

I don't really see where the problem lies but I have some suggestions that might shine some light on the situation.

  1. place a unique text file in the expected challenge location and see if it can be accessed from the Internet.
    If that fails, place another unique text file in the root folder of the site and see if that can be reached from the Internet.
    If that fails, then this is not the root folder OR this is not the nginx server handling those requests.
  2. ensure the nginx container has sufficient access to all the correct volumes
    If all that seems right, try rebuilding a new testing-nginx container and go step by step and validate that each step works before proceeding to the next one.
    If that fails, then the steps are incomplete or wrong.
    If that works, then the initial nginx container missed a step or there was a TYPO or something else...

If all fails, try using an alternate installation procedure or test this same procedure on another server.
[something has to become evident along the way]

1 Like
5

Due to the redirect being scoped to the entire server rather than to a location, it always takes priority and the /.well-known/acme-challenge/ location ends up being ignored.

You could try one of two things:

  1. Limit the redirect's scope to a location:
    location ^~ /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$server_name$request_uri;
    }
  1. ... or copy the /.well-known/acme-challenge/ block to your HTTPS server as well, so it matches post-redirect as well.
2 Likes
6

Thank you for your replies again! I tried the suggestions of _az and both options work well :slight_smile: Was not aware how the priority of redirects worked in nginx.conf, appreciate your explanation!

1 Like
7

So it was a

One we BOTH missed!

Thanks @_az for your :eyes:

1 Like
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.