Certbot failed to authenticate some domains (authenticator: webroot)

[lejaregg]

Hi! I'm new here and to the realm of self-hosting, networking/security, and Linux in general. I had some time off, so I decided to start a self-hosting project with a Raspberry Pi 3 B+ I had laying around. I've got everything set up in a Docker containers and now my next step is to set certain ported apps (i.e. http://localhost:XXXX) to be securely accessible (as opposed to port forwarding) through a domain that I've purchased via IONOS. I'm following this guide to presumably do so using nginx: How to Set Up letsencrypt with Nginx on Docker. I've scraped through every post here and what returned on Google with a similar issue or even directly related to IONOS, but they don't seem to solve my problem or it's simply not clicking. I've tried removing A and AAAA, but then it's another error where Type: dns. I was hoping someone could hand hold this newbie into figuring out why certbot isn't working for me. Please let me know if this should be posted elsewhere or if I need to provide anything additional!

My domain is: lejaregg.io

I ran this command:
docker compose run --rm certbot certonly --webroot --webroot-path /var/www/certbot/ --dry-run -d lejaregg.io

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for lejaregg.io

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: lejaregg.io
Type: unauthorized
Detail: 2607:f1c0:100f:f000::200: Invalid response from http://lejaregg.io/.well-known/acme-challenge/YwaM0LTmLUkSUOi7EMwLMmzugAxVag16aIuQYN-5-7Y: 204

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: lejaregg.io
Type: dns
Detail: no valid A records found for lejaregg.io; no valid AAAA records found for lejaregg.io

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

My web server is (include version): Sorry, I'm not certain.

The operating system my web server runs on is (include version): DietPi v8.25.1

My hosting provider, if applicable, is: For my domain? IONOS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): I don't think so?

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): Running it contained, so latest?

[rg305]

Hi @lejaregg, and welcome to the LE community forum

@lejaregg

--webroot-path /var/www/certbot/

Are you sure that is the correct path?
Those instructions show:

    location /.well-known/acme-challenge/ {
        root /var/www/certbot;
    }

Did you do that?
Has this worked for you before?

[9peppe]

@lejaregg

Detail: no valid A records found for lejaregg.io; no valid AAAA records found for lejaregg.io

@lejaregg

Detail: 2607:f1c0:100f:f000::200: Invalid response from http://lejaregg.io/.well-known/acme-challenge/YwaM0LTmLUkSUOi7EMwLMmzugAxVag16aIuQYN-5-7Y: 204

These are different.

Can you show us the relevant docker-compose.yml and tell us what webserver you're using and how you're running it?

(I found the compose file, I don't see how nginx is supposed to read the certificate file)

[lejaregg]

Quickest replies I've seen on a forum yet lol.

@rg305 Thanks!

Are you sure that is the correct path?

I believe so, as it was set in the app.conf file set in Step 3. /etc/nginx/conf.d/app.conf was created within the directory I executed the docker compose. Are you saying the "root /var/www/certbot" should actually be in ~/ versus the other directory?

Has this worked for you before?

No, I haven't been able to get it to work yet.

@9peppe

Can you show us the relevant docker-compose.yml

 version: '3'

services:
  webserver:
    image: nginx:latest
    ports:
      - 80:80
      - 443:443
    restart: always
    volumes:
      - ./nginx/conf/:/etc/nginx/conf.d/:ro
      - ./certbot/www/:/var/www/certbot/:ro
  certbot:
    image: certbot/certbot:latest
    volumes:
      - ./certbot/www/:/var/www/certbot/:rw
      - ./certbot/conf/:/etc/letsencrypt/:rwv

and tell us what webserver you're using and how you're running it?

Apologies if I am not understanding here... but do you mean my Raspberry Pi 3 B+? Or what app (localhost:XXXX) I'm trying to forward to the domain?

[9peppe]

@lejaregg

    volumes:
      - ./nginx/conf/:/etc/nginx/conf.d/:ro
      - ./certbot/www/:/var/www/certbot/:ro

this is the issue I was referring to, nginx should be able to read ./certbot/conf/, but this is a problem for the future.

Right now you have another issue. You expect nginx to respond, but your website runs on apache:

% curl -IL http://lejaregg.io/df
HTTP/1.1 404 Not Found
Content-Type: text/html
Connection: keep-alive
Keep-Alive: timeout=15
Date: Tue, 02 Jan 2024 22:25:58 GMT
Server: Apache

[lejaregg]

I'm so sorry... I'm not understanding what the root cause is here from your statement. Can nginx only communicate to certain types of webservers?

[9peppe]

nginx can do whatever you ask it to do. but right now you're not using nginx.

[Bruce5051]

Side note: HTTPS is not connecting properly, but Port 443 is Open.

$ nmap -Pn -p80,443 lejaregg.io
Starting Nmap 7.80 ( https://nmap.org ) at 2024-01-02 23:12 UTC
Nmap scan report for lejaregg.io (74.208.236.77)
Host is up (0.065s latency).
Other addresses for lejaregg.io (not scanned): 2607:f1c0:100f:f000::200
rDNS record for 74.208.236.77: 74-208-236-77.elastic-ssl.ui-r.com

PORT    STATE SERVICE
80/tcp  open  http
443/tcp open  https

Nmap done: 1 IP address (1 host up) scanned in 0.13 seconds

$ curl -Ii https://lejaregg.io/.well-known/acme-challenge/sometestfile
curl: (35) error:0A000438:SSL routines::tlsv1 alert internal error

$ curl -Ii http://lejaregg.io/.well-known/acme-challenge/sometestfile
HTTP/1.1 204
Connection: keep-alive
Keep-Alive: timeout=15
Server: nginx
Date: Tue, 02 Jan 2024 23:13:05 GMT
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Cache-Control: no-cache, no-store, max-age=0, must-revalidate
Pragma: no-cache
Expires: 0
X-Frame-Options: DENY

And https://letsdebug.net/lejaregg.io/1756064 gets OK.

[lejaregg]

@9peppe

nginx can do whatever you ask it to do. but right now you're not using nginx.

By not using nginx, do you mean from my domain's perspective? If host perspective, I've tried running the yml and then initiating the script, but the result is the same.

@Bruce5051

Side note: HTTPS is not connecting properly, but Port 443 is Open.

I had opted for user managed SSL set up in IONOS and Digicert has issued me certificates/keys, but have yet to figure out how to use them in the communication between my RPi and the domain. I'm assuming that may be why it's not connecting properly? That and there's no content on that domain whatsoever right now. Would I need to this with NGINX?

[MikeMcQ]

@lejaregg

By not using nginx, do you mean from my domain's perspective?

Yes, that is what was meant. The Let's Encrypt servers use the public DNS system to locate your IP and make an HTTP request for the HTTP Challenge.

Your system was being handled by Apache so the --nginx plugin would not work.

But, I see nginx is now responding to HTTP requests so it might now work. What happens if you try it now?

[MikeMcQ]

Huh. You have both Apache and some kind of nginx based system responding to different HTTP requests to your domain.

Can you explain more how you have your system configured? It is possible the 'nginx' system is some kind of firewall or routing device based on that. Do you know why we see both Apache and nginx responding?

The responses are also unusual but are consistent between IPv4 and IPv6 (sometimes people get these wrong).

# "Home page" request responded by Apache
curl -I4 http://lejaregg.io
HTTP/1.1 200 OK
Server: Apache

# A partial incorrect URL gets an HTTP 300 (odd) from Apache
curl -I4 http://lejaregg.io/.well-known/acme-
HTTP/1.1 300 Multiple Choices
Server: Apache

# For some reason this exact URL gets a 301 redirect from nginx
curl -I4 http://lejaregg.io/.well-known/acme-challenge
HTTP/1.1 301 Moved Permanently
Server: nginx
Location: http://lejaregg.io/.well-known/acme-challenge/

# ACME HTTP Challenge URL with sample token value gets 204 from nginx
# 204 No Content should really be 404 Not Found in this example
curl -I4 http://lejaregg.io/.well-known/acme-challenge/Test404
HTTP/1.1 204
Server: nginx

[lejaregg]

@MikeMcQ I am not entirely sure why that is happening lol. Again, absolute noob here. I decided to take what @9peppe said about not running Nginx and ran with it, so maybe you caught this as I was messing with it today?

With that being said... I ended up scrapping everything, starting over, and tried a different approach. I ended up starting up an Nginx container first, exposed that container's 80 and 443 ports, set up Cloudfare with my domain, replaced the name servers within IONOS with Cloudfare's, and created the Lets Encrypt certificate within Nginx GUI via Cloudfare API token. It all appears to be working as expected being that I was able to securely connect to another docker (self hosting app).