Certbot renew failed; (a) why is my server unreachable? (b) can I renew without reaching it?

Help
1

I have a problem where apparently I can only get to ports 80 and 443 on my home Ubuntu server from inside my ISP's firewall. I can reach my home web server from my house, I can reach it from my employer who also uses the same ISP, but nobody else in the world can reach http://www.enchanter.net (or https). All other ports on my server appear to be reachable (email, my Docker server running Valheim, &c.) I've contacted my ISP and they say they are not blocking any ports to me. I have not set up any firewall on my home server. "ufw status" says "inactive".

So, first question: how can I figure out where I'm being blocked? How exactly do I figure out if my ISP has some firewall block that's preventing people from getting to my home web server, or whether it's some wrong setting on Ubuntu?

Second question: is there a way that I can get a certbot renewal without certbot having to connect in to my server (for now)?

My domain is:
enchanter.net

I ran this command:
certbot renew

It produced this output:
Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: enchanter.net
Type: connection
Detail: 216.53.249.115: Fetching http://enchanter.net/.well-known/acme-challenge/6UTZjRFbe42EHHiGUUb7-KwSwOu9ms4zzotpzi62NVI: Timeout during connect (likely firewall problem)

Domain: www.enchanter.net
Type: connection
Detail: 216.53.249.115: Fetching http://www.enchanter.net/.well-known/acme-challenge/_RDGPKot5cxbRWX774Nspho55AbboY8stZ2DmStku5g: Timeout during connect (likely firewall problem)

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

My web server is (include version):
Apache 2.4.41

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:
Ubuntu 20.04.4 LTS

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.27.0

2

Can you actually visit your website?

Are you blocking connections because of geography?

1 Like
3

I can visit my web site (running in a server in my closet) when I'm at home, or when I'm at a local company who uses my same ISP. I can't reach it when I'm on any other network. (The server has a static IP and for the past several years it's been reachable from the Internet. It's only been in the past few weeks that specifically only ports 80 and 443 appear to be blocked.)

I'm not blocking any connections (as far as I know).

Oddly, I am getting just a few hits from other sources who I don't think are part of my ISP, such as 198.235.24.146, 35.223.217.26, and 66.249.64.123.

I just can't figure out what could be blocking (only) ports 80 and 443 to my server. It could be the ISP's outer firewall, but then that would be affecting all of their other customers. It could be my router, but I haven't done anything on it that should cause this. It could be my Ubuntu server itself, but I don't see any firewall on it.

4

Well, I don't know if there's much we can help you with other than saying to work with your ISP. In theory, routing packets to your system correctly is exactly what you're paying them for. If you can determine exactly where the cutoff is between systems that can and can't connect, that might help. Maybe try some cloud providers with regions available near you, spin up a VM on them, and see whether each of them can connect?

I can confirm that it doesn't work from my house (in the Northeast of the US).

Only by using a DNS challenge. (See the Challenge Types documentation page.) That's the typical way to do it for "internal" sites that aren't intended to be publicly accessible. Most people find it more challenging to do it that way, because it involves being able to update the DNS TXT records automatically from a script (or doing a manual process every couple months), but sometimes it's the easiest approach. Of course, even if you have a certificate, if your users can't get to your site it won't do you much good.

2 Likes
5

"routing packets to your system correctly is exactly what you're paying them for" - that's a very good point; thank you for that. I'll contact them again. (I'm on uncertain ground as they seem to think that letting me run my own services is a privilege and not part of the contract, but I'll go to a business support contract if I need to.)

How do I use a DNS challenge? The Challenge Types page you linked says it's an option, but I can't find anything telling me how to make certbot use it. I see a lot of outdated documentation out there telling me I need to download and run various scripts, but this is just a one-off until I get my site accessible again.

"Of course, even if you have a certificate, if your users can't get to your site it won't do you much good" - true, but while my cert is expired my browsers and email clients are complaining when I try to use my web and mail servers from home, which is why I'm hoping this is the easier problem to solve.

6

https://eff-certbot.readthedocs.io/en/stable/using.html#dns-plugins

1 Like
7

Woohoo, I figured out how to renew my certificate using a DNS challenge! Posting the steps here in case they help anyone else (or me, in three months).

I used this command to first do a dry run:
certbot -d enchanter.net -d www.enchanter.net --manual --preferred-challenges dns certonly --dry-run

This told me to create a DNS TXT record under the name _acme-challenge.enchanter.net with a specific value, and then another DNS TXT record under the name _acme-challenge.www.enchanter.net with a different value.

My domain registrar is Afraid.org, and that site makes it easy to add new DNS records. Type "TXT", subdomain "_acme-challenge" and "_acme-challenge.www" (because the domain is automatically appended), and I put the strings into "Destination" with quotes around them.

Then it was a matter of going to "Dig (DNS lookup)", entering "_acme-challenge.enchanter.net", and refreshing occasionally until the TXT record appeared with the correct value. Same for the other name. When they both showed up properly, I could press Enter on the certbot script, and then it confirmed that the dry run was successful.

I re-ran without --dry-run, did the same steps, the updated TXT records were visible within a few minutes, and then certbot automatically wrote my new certificate to the right place. All I had to do was apachectl restart and service dovecot restart and now I'm up and running with a renewed certificate! Thank you for your help!

Now to ask my ISP for help figuring out why I appear to be blocked ...

1 Like
8

For a one-off, you may just want manual mode, with --manual --preferred-challenges dns, which will tell you the TXT entry to create in your DNS.

For automated use, you want a plugin based on your DNS provider. Some make it easy, some make it more challenging.

4 Likes
9

Glad you figured it out at the time I was trying to type up how to do it. :slight_smile:

You may want to put a reminder on your calendar (or whatever you use to tell yourself to do things) in 7 to 8 weeks from now, just to ensure that you don't forget, rather than waiting for this cert to expire again.

Yeah, a lot of residential ISPs don't really want to handle "server" traffic since they want to charge for business-class service. If this server is that important to you, you might want to make that move. It wouldn't surprise me if they actually are blocking it because you're on a residential connection, even if the person you were talking to before didn't know about it.

3 Likes
10

Certbot used to send me reminders by email, a few weeks before my certificate expired. I guess when my site became unreachable, it stopped sending me reminders.

11

It's not related to your site being unreachable, it's just that they've had some issues with sending the reminders lately.

Per their documentation page, they'll "do our best" to send them and recommend to "only use these expiry notices as a warning to check on your automation."

4 Likes
12

I hope this doesn't come off tooooo negatively, but...
Manually obtaining a cert is a false sense of accomplishment.
In fact, I see it as a step backwards - it doesn't even attempt to address the problem at hand:
There are plenty of IPs on the Internet that can't reach your site at all.
Manually enabling TLS on that unreachable site really doesn't move you closer to a solution.

I would confirm with the ISP that they are NOT blocking HTTP [some have been known to do that].
Once HTTP access has been restored, I would switch the authentication back to something automated.
If you manage to produce a working DNS authentication, that is well enough.
If that proves too dificult, simply revert to HTTP authentication - which has worked for you in the past.

Again, we are here to help - I just think the current path is deviating from the true desired outcome.

2 Likes
13

rg305, this isn't about a sense of accomplishment. :slight_smile:

Renewing my certificate allows me to continue using my web and imap servers here at home. The problem of other people not being able to reach my web site is actually more of an serious annoyance than a serious problem, and trying to convince my ISP that this is worth their time to help me solve (rather than telling me "go away until you can prove it's not being blocked on your side") might be a serious challenge.

(bonus: if my certificate weren't working, there's more of a chance my ISP would tell me 'your certificate is expired, no one can use your site anyway, go away')

Using certbot with DNS authentication is (I hope) a one-time thing until I can get outside access restored to my site, after which I'll be able to resume using certbot renew.

I appreciate your help!

1 Like
14

can they reach your smtp server, is it on another machine, or are you losing mail?

15

I'm receiving email. My smtp server (and my Valheim server, and my Minecraft server) is reachable from outside my ISP's network.

That's the weird part of it. Only ports 80 and 443 are affected, and only from outside my ISP's network.

2 Likes
16

So... the challenge is now "beat them at their own game!"
Challenge accepted!

1 Like
18

I tried from three different and large ISPs [within the South Florida Area] and they all can't reach your site:
image

image
image760Ɨ424 15.8 KB 

curl -Ii enchanter.net
curl: (56) Recv failure: Connection reset by peer

ping enchanter.net
PING enchanter.net (216.53.249.115) 56(84) bytes of data.
^C
--- enchanter.net ping statistics ---
9 packets transmitted, 0 received, 100% packet loss, time 8193ms


traceroute -I enchanter.net
traceroute to enchanter.net (216.53.249.115), 30 hops max, 60 byte packets
 1  [redacted] [redacted]   1.267 ms  1.452 ms  1.453 ms
 2  [redacted] [redacted]   1.987 ms  2.944 ms  3.187 ms
 3  * * *
 4  99.167.38.110 (99.167.38.110)  6.812 ms  7.180 ms  12.603 ms
 5  12.242.116.11 (12.242.116.11)  18.959 ms  19.282 ms  19.726 ms
 6  be7018.ccr21.mia03.atlas.cogentco.com (154.54.10.13)  20.184 ms  11.907 ms  11.033 ms
 7  be3401.ccr22.mia01.atlas.cogentco.com (154.54.47.29)  11.028 ms  11.178 ms  11.549 ms
 8  be2320.rcr21.tpa01.atlas.cogentco.com (154.54.5.86)  18.018 ms  18.319 ms  18.856 ms
 9  be3638.rcr52.mco01.atlas.cogentco.com (154.24.19.129)  20.477 ms  20.793 ms  21.097 ms
10  be2442.nr61.b005287-0.mco01.atlas.cogentco.com (154.24.54.118)  23.080 ms  23.254 ms  23.876 ms
11  38.142.102.98 (38.142.102.98)  23.437 ms  23.864 ms  24.411 ms
12  mn-d01-02-ten6-1.smartcitytelecom.com (66.90.9.125)  25.401 ms  25.542 ms  26.188 ms
13  mn-a07-01-vlan559.smartcitytelecom.com (66.90.8.142)  27.589 ms  27.944 ms  28.560 ms
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
1 Like
19

That absolutely screams that they must be blocking them on purpose for residential connections, especially if you haven't changed anything on your side.

3 Likes
20

Addendum:

After I renewed my certificate via a DNS challenge, last weekend I emailed my ISP's tech support to ask them to help me figure out why no traffic was reaching my web server.

I checked Monday evening and found that now my site is reachable. So it must have been something on their end that they fixed.

All's well that ends well!

3 Likes
21

I do agree on that statement...
But has it ended well?
Is your certificate set for autorenewal?

1 Like