Certbot working, now fails

Help
1

My domain is: signal.wetleaves.com

I ran this command: sudo certbot renew --force-renewal

It produced this output:

Renewing an existing certificate for signal.wetleaves.com

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
  Domain: signal.wetleaves.com
  Type:   connection
  Detail: 204.93.98.249: Fetching http://signal.wetleaves.com/.well-known/acme-challenge/laE4TFx48-IXIQLaCd0jS-m7BHYOun9OkDjEi9l-0Sk: Connection refused

Hint: The Certificate Authority failed to download the temporary challenge files created by Certbot. Ensure that the listed domains serve their content from the provided --webroot-path/-w and that files created there can be downloaded from the internet.

My web server is (include version): apache2, 2.4.29

The operating system my web server runs on is (include version): Ubuntu Linux 18.04.6

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): Kinda, using webmin, but I can stumble around in a terminal window when required.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):certbot 2.6.0

================

There is nothing in the folder /var/www/html/.well-known/acme-challenge/ except .htaccess which consists of:

AuthType None
Require all granted
Satisfy any

There seem to be no errors in any apache logs. If Apache is refusing the connection, shouldn't I see an error in the log? The external firewall passes all 80 and 443 to the server successfully. An index.html file placed in .well-known/acme-challenge/ is served correctly, but its flipped to 443 instead of port 80. I dont know how to change that.

This thing worked flawlessly for several renewal periods, and suddenly cant renew. Any help or advice would be appreciated. I can access the server from outside my firewall....but the certificate authority seems unable to access it.

2

Don't use this. It won't magically overcome problems with issuance, and it's a near-guaranteed path to busting a rate limit.

Your server is refusing connections on port 80. Fix that. I'm also not able to connect, and letsdebug.net is having the same problem:
Let's Debug

7 Likes
3

How have you tested this?
Everyone else sees:

curl -Ii signal.wetleaves.com
curl: (56) Recv failure: Connection reset by peer

curl -Ii https://signal.wetleaves.com
curl: (7) Failed to connect to signal.wetleaves.com port 443: Connection timed out
4 Likes
4

Yes. Which means something between Apache and the public internet is blocking the connection. Most often a firewall. If you are residential ISP then check that your router settings still allow connections. And, make sure your ISP is not blocking ports 80 and 443.

Served correctly even from the public internet? I think not :slight_smile:
Try a cell phone with wifi off so you use the carrier public internet and try reaching that test file. Almost certainly won't work based on what we all see

4 Likes
5

Is it possible for "lets debug" to tell me what the source IP is? It tells me the destination (my IP of course) but not source. Could it be 172.104.24.29 ? I run lets debug and the UFW log shows (among a bunch of other inscrutable stuff)

May 27 11:09:13 signal kernel: [10695887.543250] [UFW AUDIT] IN=enp0s31f6 OUT= MAC=40:8d:5c:c3:62:00:30:23:03:db:a6:d8:08:00 SRC=192.168.1.1 DST=192.168.1.120 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=3297 DF PROTO=TCP SPT=36268 DPT=10000 WINDOW=64240 RES=0x00 SYN URGP=0 
May 27 11:09:32 signal kernel: [10695906.633842] [UFW AUDIT] IN=enp0s31f6 OUT= MAC=40:8d:5c:c3:62:00:30:23:03:db:a6:d8:08:00 SRC=192.168.1.1 DST=192.168.1.120 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=18931 DF PROTO=TCP SPT=49790 DPT=10000 WINDOW=64240 RES=0x00 SYN URGP=0 
May 27 11:09:54 signal kernel: [10695928.940557] [UFW AUDIT] IN=enp0s31f6 OUT= MAC=40:8d:5c:c3:62:00:30:23:03:db:a6:d8:08:00 SRC=192.168.1.1 DST=192.168.1.120 LEN=60 TOS=0x00 PREC=0x00 TTL=63 ID=12500 DF PROTO=TCP SPT=49482 DPT=10000 WINDOW=64240 RES=0x00 SYN URGP=0

Apache log shows

192.168.1.1 - - [27/May/2023:11:09:20 -0400] "GET /.well-known/acme-challenge/ HTTP/1.1" 403 5165 "-" "Mozilla/5.0 (X11; Ubuntu; Linux x86_64; rv:109.0) Gecko/20100101 Firefox/112.0"
172.104.24.29 - - [27/May/2023:11:09:52 -0400] "GET / HTTP/1.1" 403 5113 "-" "Go-http-client/1.1"

So this seems to me to show Apache sending a 403 forbidden back to 172.104.24.29, is that correct?

Lets debug appears to be looking for a randomly named challenge file (seems different each time I rerun the test). So of course Apache is going to send back a 403, right? Since that file is not there?

I placed a index.html file in /var/www/html/.well-known/acme-challenge/, and I am able to view it from Verizon wireless. But its on 443, I do not know how to force it to 80, the browsers have gotten very brutal about forcing SSL.

Im lost at this point. I appreciate yall's help.

6

That's the destination IP address of letsdebug.net, so it makes sense if that would also be the source IP address.

3 Likes
7

I can see your index.html file using HTTPS as you say. So, that's improved.

But, the Let's Encrypt server will use HTTP. Your system is still not allowing HTTP requests. You could try using 'http://...' explicitly in your browser it might honor that.

curl -I -m8 http://signal.wetleaves.com/.well-known/acme-challenge/ForumTest123
curl: (7) Failed to connect to signal.wetleaves.com port 80 after 21 ms: Connection refused

Your last cert was issued Mar21. What has changed since then? New router? New firewall or router settings? New Apache config? ISP now blocking port 80?

It looks like you made some changes around that time as you got several certs in a short period of time. Your prior history showed a more consistent 60 day renewal

image
image2085×753 134 KB

6 Likes
8

Unrelated to actual LE challenge requests:

3 Likes
9

I spent some time making the pre and post hook scripts log what they were doing. But AFAICR, I made no changes to Apache2 or the FW. The pre and post hook scripts would open the FW before the renewal and then close it up again afterwards.

10

Please help me understand....if letsdebug is looking for a randomly named file, it seems to me that it would never be successful, because that folder is always empty except while certbot actually runs. From what I can see in the cerbot logs, the certbot process creates then deletes the challenge file regardless of whether or not it was successfully accessed.

So how is letsdebug expected to ever find this randomly named file on my system? Its never going to be there, unless something runs on my system to create it. Can someone help me understand this part of the letsdebug process?

11

Followup, possibly successful....letsdebug is now getting a timeout instead of a refusal. To me, this indicates I have fixed the apache problem, and letsdebug is now asking the server for a file that doesnt exist, as I stated above. Is my assumption right or wrong?

I added port 80 to the apache ports.conf, which was not there before....

12

Certbot successfully renewed just now. Thanks much to the team for helping me TS this failure. I learned much in the process, and yalls help was enlightening.

I do however wish that the letsdebug process would allow use of a predetermined file. Like I said, if its looking for a random fie, it seems it would never succeed. Maybe I am missing the whole point. Regardless, yall's help and letsdebug got my problem fixed.

1 Like
13

Let's Debug sees a 404 file not found-reply as a "success", as in, it can reach the webserver on port 80. As nothing normally is present in the /.well-known/acme-challenge/ directory, it cannot get any file there anyway, as even if there would be a non-human generated file there, it would be quite hard to guess as that file would be random too.

The process of Let's Debug is fully automated and doesn't involve human steps like "Please put a filename with name whatever in the directory /.well-known/acme-challenge/". This might be something to consider adding to some "advanced" version of Let's Debug to tackle certain webserver configuration errors. Like a documentation page extending the default behaviour explaining the user could perhaps put the file "lets-debug" into the /.well-known/acme-challenge/ folder. But that shouldn't be required for normal, automated use.

Maybe @_az is interested in implementing such a thing :slight_smile:

3 Likes
14

Just adding some color to what Osiris said ...

There are two HTTP challenge tests made by Let's Debug. The first looks for exactly this:

http://signal.wetleaves.com/.well-known/acme-challenge/letsdebug-test

This is expected to fail with a 404 Not Found because people don't usually have this file in that location. But, you could certainly create one and then this test will show a 200 OK when successful.

The second HTTP test is Let's Debug using the Let's Encrypt Staging system. Let's Debug requests a cert from Staging but of course this will also fail because Let's Debug cannot place the needed file on your server. This is the random file you see.

If the "failure" is expected the test is considered a success. Unexpected failures are things like timeouts or multiple IP addresses responding differently and others.

These tests, along with other DNS checks and such, help a lot to identify communications or common config problems with servers and domain names.

You can see detailed results of Let's Debug using its Verbose Output link at the bottom of the summary page. Such as below. In your case, if you dynamically open and close your firewall will always fail with a timeout so this test is not that useful in your case.

But, you could use certbot renew --dry-run for testing. This uses the Let's Encrypt Staging system and won't disturb existing certs. It should run your hooks so a failure points to an actual problem with your certbot request

2 Likes