Certbot Failed to Authenticate some Domains

My domain is: ubccoursenotify.com

My web server is (include version): Ubuntu 21.10.0

The operating system my web server runs on is (include version): Linux

My hosting provider, if applicable, is: LINODE

I can login to a root shell on my machine (yes or no, or I don't know): YES

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): NO

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot docker images "latest"

Hey guys,

I'm following this guide: Nginx and Let’s Encrypt with Docker in Less Than 5 Minutes | by Philipp | Medium

and have done the following:

  1. Created my docker-compose.yml file
version: "3"

services:
  web:
    image: weswwn/project-notify:1.9
    ports:
      - "8080:8080"
  nginx:
    image: nginx:latest
    command: "/bin/sh -c 'while :; do sleep 6h & wait $${!}; nginx -s reload; done & nginx -g \"daemon off;\"'"
    ports:
      - "80:80"
      - "443:443"
    volumes:
      - ./data/nginx:/etc/nginx/conf.d
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/cerbot
  certbot:
    image: certbot/certbot
    entrypoint: "/bin/sh -c 'trap exit TERM; while :; do certbot renew; sleep 12h & wait $${!}; done;'"
    volumes:
      - ./data/certbot/conf:/etc/letsencrypt
      - ./data/certbot/www:/var/www/certbot

  1. Created my nginx app.conf file (which lives in data/nginx/app.conf):
server {
    listen 80;
    server_name ubccoursenotify.com;

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }    
}
server {
    listen 443 ssl;
    server_name ubccoursenotify.com;

    ssl_certificate /etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ubccoursenotify.com/privkey.pem;
    
    location / {
        proxy_pass http://ubccoursenotify.com; #for demo purposes
    }
}
  1. Installed the letsencrypt script in the guide that i'm following and ran it.

I'm running into this problem:

  Domain: ubccoursenotify.com
  Type:   unauthorized
  Detail: Invalid response from http://ubccoursenotify.com/.well-known/acme-challenge/X4BbLw2SHhFegVgbN4B4GLunf2oI8aBiqdLiYpxVqQk [2600:3c01::f03c:93ff:fe9b:e23f]: 404

  Domain: www.ubccoursenotify.com
  Type:   unauthorized
  Detail: Invalid response from http://www.ubccoursenotify.com/.well-known/acme-challenge/t0JiAtsg4FWldYLcqQtwDw4JEjv-I_vtC8-Ek8qAqAs [2600:3c01::f03c:93ff:fe9b:e23f]: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

### Reloading nginx ...
2022/03/29 00:56:37 [emerg] 11#11: cannot load certificate "/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

My domains are set up as the following:

Please help!!

Nginx isn't running because it doesnt have certificates right now. You need a self signed certificate to bootstrap it.

See the " The Chicken or the Egg " section in the tutorial you linked.

(That's not a very good tutorial, I'd say)

1 Like

Yea, I've followed that part. I installed the script and ran it. That's where I'm running into the problem. The full message is:

### Starting nginx ...
Recreating root_nginx_1 ... done

### Deleting dummy certificate for ubccoursenotify.com ...
Creating root_certbot_run ... done

### Requesting Let's Encrypt certificate for ubccoursenotify.com ...
Creating root_certbot_run ... done
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for ubccoursenotify.com and www.ubccoursenotify.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: ubccoursenotify.com
  Type:   unauthorized
  Detail: Invalid response from http://ubccoursenotify.com/.well-known/acme-challenge/Gi9IuG2DMd0LC_KK1r13lztjsLhM2Cn6ngDKl_1Cn1k [2600:3c01::f03c:93ff:fe9b:e23f]: 404

  Domain: www.ubccoursenotify.com
  Type:   unauthorized
  Detail: Invalid response from http://www.ubccoursenotify.com/.well-known/acme-challenge/Fnqpr-RjhzeqmSQdcEjYMFiaQ4GshzlKD6q4H-YOiSs [2600:3c01::f03c:93ff:fe9b:e23f]: 404

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

### Reloading nginx ...
2022/03/29 01:06:19 [emerg] 11#11: cannot load certificate "/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)
nginx: [emerg] cannot load certificate "/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem": BIO_new_file() failed (SSL: error:02001002:system library:fopen:No such file or directory:fopen('/etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem','r') error:2006D080:BIO routines:BIO_new_file:no such file)

What's inside live in there?

1 Like

I believe its empty right now:

root@localhost:~# cd data

root@localhost:~/data# ls

**certbot** **nginx**

root@localhost:~/data# cd certbot/

root@localhost:~/data/certbot# ls

**conf** **www**

root@localhost:~/data/certbot# ls -l

total 8

drwxr-xr-x 8 root root 4096 Mar 29 01:09 **conf**

drwxr-xr-x 2 root root 4096 Mar 29 01:09 **www**

root@localhost:~/data/certbot# cd conf

root@localhost:~/data/certbot/conf# ls

**accounts** **csr** **keys** **live** options-ssl-nginx.conf **renewal** **renewal-hooks** ssl-dhparams.pem

root@localhost:~/data/certbot/conf# ls -l

total 32

drwx------ 4 root root 4096 Mar 27 22:41 **accounts**

drwxr-xr-x 2 root root 4096 Mar 29 01:09 **csr**

drwx------ 2 root root 4096 Mar 29 01:09 **keys**

drwxr-xr-x 2 root root 4096 Mar 29 01:09 **live**

-rw-r--r-- 1 root root 721 Mar 27 22:28 options-ssl-nginx.conf

drwxr-xr-x 2 root root 4096 Mar 27 22:38 **renewal**

drwxr-xr-x 5 root root 4096 Mar 27 22:19 **renewal-hooks**

-rw-r--r-- 1 root root 424 Mar 27 22:28 ssl-dhparams.pem

root@localhost:~/data/certbot/conf# cd live

root@localhost:~/data/certbot/conf/live# ls

root@localhost:~/data/certbot/conf/live# ls

root@localhost:~/data/certbot/conf/live#

Would you be available to hop on discord to help me out?

That's a problem. That script is supposed to put a self signed certificate in there.

1 Like

From my understanding it creates the self-signed certificate then deletes it once nginx is up and running.

From here I think thats where it happens:

## Starting nginx ...
Recreating root_nginx_1 ... done

### Deleting dummy certificate for ubccoursenotify.com ...
Creating root_certbot_run ... done

That's soooo messed up. (It works, but it's risky)

So... Why isn't validation working for you?

You should add --staging and --debug-challenges to your certbot command (i don't know if it's that script or you type it yourself) and check that you can get the proper file on ubccoursenotify.com/.well-known/acme-challenge/SOMESTRING (certbot will ask your permission to proceed after writing it. Leave it hanging and check if you can get the file)

(Check the docs because typing uncommon options from memory isn't something anybody should be doing)

1 Like

Thanks for the suggestions. If anybody else has any clue please let me know!

1 Like

It looks like it's working now, isn't it?

% curl -IL ubccoursenotify.com/
HTTP/1.1 301 Moved Permanently
Server: nginx/1.21.6
Date: Tue, 29 Mar 2022 04:08:19 GMT
Content-Type: text/html
Content-Length: 169
Connection: keep-alive
Location: https://ubccoursenotify.com/

HTTP/1.1 200 OK
Server: nginx/1.21.6
Date: Tue, 29 Mar 2022 04:08:20 GMT
Content-Type: text/html; charset=UTF-8
Content-Length: 2247
Connection: keep-alive
X-Powered-By: Express
Access-Control-Allow-Origin: *
Accept-Ranges: bytes
Cache-Control: public, max-age=0
Last-Modified: Sun, 27 Mar 2022 22:43:30 GMT
ETag: W/"8c7-17fcd8c7a50"

%
2 Likes

Create a file called test under /var/www/certbot then check you can browse to http://www.ubccoursenotify.com/.well-known/acme-challenge/test - if you can't do that then http validation is not going to work.

All the config here is trying to do is setup that path as the place for certbot to write http challenge response file to, then it asks certbot to use that path during validation.

The one issue I can forsee is that using /var/www/certbot as the webroot would surely mean that certbot would create /var/www/cerbot/.well-known/acme-challenge? This would make the resulting url http://www.ubccoursenotify.com/.well-known/acme-challenge/.well-known/acme-challenge/ which would be incorrect. I feel the guide would have been corrected if this was the case though.

What happens if you change app.conf to say

location /.well-known/acme-challenge/ {
        root /var/www/certbot/.well-known/acme-challenge;
    }
3 Likes

It will break.

The root directive applies to the / path. So if you have a root for a subpath, you need to use a directory matching the subpath, inside the root.

The alias directive, on the other hand...

2 Likes

Thank you for looking into this! Your comment caused me to look into this portion of the code and I tried to do:

server {
    listen 80;
    server_name ubccoursenotify.com;

    location ^~/.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

    location / {
        return 301 https://$host$request_uri;
    }    
}

server {
    listen 443 ssl;
    server_name ubccoursenotify.com;

    ssl_certificate /etc/letsencrypt/live/ubccoursenotify.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/ubccoursenotify.com/privkey.pem;
    
    location / {
        proxy_pass http://ubccoursenotify.com:8080;
    }
}

And this worked. I dont understand how changing ^~/.well-known/acme-challenge/ worked. Any insite on this? I looked it up and know that "The tilde instructs nginx to perform a case-sensitive regular expression match, instead of a straight string comparison."

Any idea on why this is the case?

You probably also want your server_name to be ubccoursenotify.com www.ubccoursenotify.com; because I note the cert you are trying to get uses both.

I believe the behavior of ^~ causes the rule to take precedence over other matching rules (such as other regular expressions), see Nginx Location Priority: Configuring NGINX and NGINX Plus as a Web Server | NGINX Plus

2 Likes

Yea, it worked when I changed:

location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

to:

location ^~/.well-known/acme-challenge/ {
        root /var/www/certbot;
    }
2 Likes

that wasn't supposed to happen. are you sure that's the only thing you changed?

2 Likes

Yea,

I left everything else the same :thinking:

Am confused about it as well. I'll double check my configuration files again but I'm pretty sure nothing else changed.

You might find this interesting: Module ngx_http_core_module

2 Likes