root@ip-72-167-33-188:/var/www# chmod 755 cozybroadcast.stream
root@ip-72-167-33-188:/var/www# ls -l
total 8
drwxr-xr-x 2 www-data www-data 4096 May 1 19:51 cozybroadcast.stream
drwxr-xr-x 2 root root 4096 May 1 08:54 html
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# chmod 644 test
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# ls -l
total 4
-rw-r--r-- 1 root root 5 May 1 19:51 test
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# sudo apachectl -k graceful
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# sudo certbot certonly --apache -d "cozybroadcast.stream,www.cozybroadcast.stream" --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Simulating a certificate request for cozybroadcast.stream and www.cozybroadcast.stream
Performing the following challenges:
http-01 challenge for cozybroadcast.stream
http-01 challenge for www.cozybroadcast.stream
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain www.cozybroadcast.stream
Challenge failed for domain cozybroadcast.stream
http-01 challenge for www.cozybroadcast.stream
http-01 challenge for cozybroadcast.stream
Cleaning up challenges
Some challenges have failed.
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: www.cozybroadcast.stream
Type: unauthorized
Detail: Invalid response from
http://www.cozybroadcast.stream/.well-known/acme-challenge/RBhJbuLJI6Rr5-JrjpZTdwnGpBsMkjENUxKHg0fswzs
[72.167.33.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
2.0//EN\">\n<html><head>\n<title>403
Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"
Domain: cozybroadcast.stream
Type: unauthorized
Detail: Invalid response from
http://cozybroadcast.stream/.well-known/acme-challenge/NlW4--MBZFEfipF8s-VN2LVQBPJmM8Nq6hmbGRLyzMQ
[72.167.33.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
2.0//EN\">\n<html><head>\n<title>403
Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
While certbot is paused, look at the permissions of the .well-known and .well-known/acme-challenge directories as well as the two authentication files that are created.
Thank you so much for your patience and perseverance here.
I have my guesses, but I want to get you a real, definitive explanation going forward of exactly what's happening, so I'm calling in heavy reinforcements. They may take some time to respond (probably not today), so please continue to be patient. Please try to leave things in the existing state if at all possible.
I would really like to understand the root of this issue so that I have a complete understanding going forward. I have a strong feeling that it might have something to do with the user running the certbot process and the default permissions associated. Maybe a umask and/or chgrp involved?
update:
The problem turned out to be the restricted group and other permissions of the /etc/letsencrypt directory itself.
edit:
These posts from below seem prudent:
Keep in mind that the log shown is from Osiris's own system though, not that of the OP.
Note that the Apache plugin doesn't actually put the challenge file into the DocumentRoot directory, but uses a RewriteRule to redirect it to /var/lib/letsencrypt/.
2021-05-02 10:39:22,681:DEBUG:certbot_apache._internal.http_01:writing a pre config file with text:
RewriteEngine on
RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]
2021-05-02 10:39:22,682:DEBUG:certbot_apache._internal.http_01:writing a post config file with text:
<Directory /var/lib/letsencrypt/http_challenges>
Require all granted
</Directory>
<Location /.well-known/acme-challenge>
Require all granted
</Location>
that's expected. Certbot may add a temporary redirect and / or a temporary location. If your system blocks these directories, you will get a 403. The "original" /.well-known/ in your file system may not exist.
By the way, the --debug-challenges is also a very powerfull debugging tool, which enables you to really pry your webserver with requests during the time the challenge is available and see what pops up in the Apache logs when you do so.
So the redirected directory might be inaccessible too?
I did notice the strict permissions of the webroot itself and the files created inside resulted in anything inside the webroot being inaccessible. Fixing the permissions helped there. I didn't notice the redirect though, which would certainly explain the missing .well-known.
I just wonder how all this happened in the first place.
But looked in the wrong direction Also: Apache access and error logs!
When debugging strange stuff, I usually run a tail -f /var/log/apache2/access.log and/or tail -f /var/log/apache2/error.log in one or two separate terminals, so I can see the output of the logs directly on my screen. And while checking that, request something from Apache.. And directly see what's going on within Apache.
With a 403 yielded, I would think the target of the redirect must have been inaccessible to apache rather than non-existent or a 404 would have been yielded.