I deleted my ssl cert and now certbot won't let me get another one

Ok :slight_smile:

root@ip-72-167-33-188:/var/www# chmod 755 cozybroadcast.stream
root@ip-72-167-33-188:/var/www# ls -l
total 8
drwxr-xr-x 2 www-data www-data 4096 May  1 19:51 cozybroadcast.stream
drwxr-xr-x 2 root     root     4096 May  1 08:54 html
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# chmod 644 test
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# ls -l
total 4
-rw-r--r-- 1 root root 5 May  1 19:51 test
2 Likes

Guess what? :upside_down_face:

3 Likes

Once more, with feeling...

Let's test it:

sudo certbot certonly --apache -d "cozybroadcast.stream,www.cozybroadcast.stream" --dry-run

If that works, this should do it:

sudo certbot --apache -d "cozybroadcast.stream,www.cozybroadcast.stream"

3 Likes

It appears still to be acting up:

root@ip-72-167-33-188:/var/www/cozybroadcast.stream# sudo apachectl -k graceful
root@ip-72-167-33-188:/var/www/cozybroadcast.stream# sudo certbot certonly --apache -d "cozybroadcast.stream,www.cozybroadcast.stream" --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Simulating a certificate request for cozybroadcast.stream and www.cozybroadcast.stream
Performing the following challenges:
http-01 challenge for cozybroadcast.stream
http-01 challenge for www.cozybroadcast.stream
Enabled Apache rewrite module
Waiting for verification...
Challenge failed for domain www.cozybroadcast.stream
Challenge failed for domain cozybroadcast.stream
http-01 challenge for www.cozybroadcast.stream
http-01 challenge for cozybroadcast.stream
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.cozybroadcast.stream
   Type:   unauthorized
   Detail: Invalid response from
   http://www.cozybroadcast.stream/.well-known/acme-challenge/RBhJbuLJI6Rr5-JrjpZTdwnGpBsMkjENUxKHg0fswzs
   [72.167.33.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>403
   Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

   Domain: cozybroadcast.stream
   Type:   unauthorized
   Detail: Invalid response from
   http://cozybroadcast.stream/.well-known/acme-challenge/NlW4--MBZFEfipF8s-VN2LVQBPJmM8Nq6hmbGRLyzMQ
   [72.167.33.188]: "<!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML
   2.0//EN\">\n<html><head>\n<title>403
   Forbidden</title>\n</head><body>\n<h1>Forbidden</h1>\n<p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
2 Likes

Run this:

sudo certbot certonly --apache -d "cozybroadcast.stream,www.cozybroadcast.stream" --dry-run --debug-challenges

While certbot is paused, look at the permissions of the .well-known and .well-known/acme-challenge directories as well as the two authentication files that are created.

3 Likes

Not sure if I'm doing this right. so I ran that command, then when it paused, I opened up a second terminal and did this:

root@ip-72-167-33-188:/var/www/cozybroadcast.stream# ls -l .well-known
ls: cannot access '.well-known': No such file or directory
2 Likes

Thank you so much for your patience and perseverance here.

:pray:

I have my guesses, but I want to get you a real, definitive explanation going forward of exactly what's happening, so I'm calling in heavy reinforcements. They may take some time to respond (probably not today), so please continue to be patient. Please try to leave things in the existing state if at all possible.

@certbot-devs

I would really like to understand the root of this issue so that I have a complete understanding going forward. I have a strong feeling that it might have something to do with the user running the certbot process and the default permissions associated. Maybe a umask and/or chgrp involved?

update:

The problem turned out to be the restricted group and other permissions of the /etc/letsencrypt directory itself.

edit:

These posts from below seem prudent:

Keep in mind that the log shown is from Osiris's own system though, not that of the OP.

3 Likes

ok thank you. I'm happy to do a screen share of some sort, too.

3 Likes

Please check your Apache error log: it should shed some light on why the error is presented.

4 Likes

I'm fairly positive that it's permission-related in the OS. I just don't know exactly where things went sideways. :confused:

3 Likes

Note that the Apache plugin doesn't actually put the challenge file into the DocumentRoot directory, but uses a RewriteRule to redirect it to /var/lib/letsencrypt/.

4 Likes

Any idea why apache would be yielding a 403 forbidden? It even did so in my manual testing. I'm really stumped on this one.

3 Likes

From the certbot debug log:

2021-05-02 10:39:22,681:DEBUG:certbot_apache._internal.http_01:writing a pre config file with text:
         RewriteEngine on
        RewriteRule ^/\.well-known/acme-challenge/([A-Za-z0-9-_=]+)$ /var/lib/letsencrypt/http_challenges/$1 [END]

2021-05-02 10:39:22,682:DEBUG:certbot_apache._internal.http_01:writing a post config file with text:
         <Directory /var/lib/letsencrypt/http_challenges>
            Require all granted
        </Directory>
        <Location /.well-known/acme-challenge>
            Require all granted
        </Location>

There's where you should look :slight_smile:

5 Likes

Hi @zeesteve1

that's expected. Certbot may add a temporary redirect and / or a temporary location. If your system blocks these directories, you will get a 403. The "original" /.well-known/ in your file system may not exist.

Run your command with -vvv and share the result.

PS: Ah, @Osiris has already shared such a log.

3 Likes

By the way, the --debug-challenges is also a very powerfull debugging tool, which enables you to really pry your webserver with requests during the time the challenge is available and see what pops up in the Apache logs when you do so.

3 Likes

So the redirected directory might be inaccessible too?

I did notice the strict permissions of the webroot itself and the files created inside resulted in anything inside the webroot being inaccessible. Fixing the permissions helped there. I didn't notice the redirect though, which would certainly explain the missing .well-known.

I just wonder how all this happened in the first place.

3 Likes

I had him run --debug-challenges here:

3 Likes

But looked in the wrong direction :grinning_face_with_smiling_eyes: Also: Apache access and error logs!

When debugging strange stuff, I usually run a tail -f /var/log/apache2/access.log and/or tail -f /var/log/apache2/error.log in one or two separate terminals, so I can see the output of the logs directly on my screen. And while checking that, request something from Apache.. And directly see what's going on within Apache.

3 Likes

Evidently that is so. :sweat_smile:

3 Likes

With a 403 yielded, I would think the target of the redirect must have been inaccessible to apache rather than non-existent or a 404 would have been yielded.

3 Likes