IP got probably banned

Help
1

Hello,

We are experiencing problems with renewing/getting new certificates.
I believe, that our IP might be banned and will explain further after filling out the standard form:

My domain is: enobyte.com
Subdomain: matrix.enobyte.com

I ran this command: certbot renew

It produced this output:

[...]

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/matrix.enobyte.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Attempting to renew cert (matrix.enobyte.com) from /etc/letsencrypt/renewal/matrix.enobyte.com.conf produced an unexpected error: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable. Skipping.

[...]

My web server is (include version): nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20.04.5 LTS

My hosting provider, if applicable, is: -

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.40.0

Since the Command fails with a network unreachable error, I also tried to run curl:

# curl -v4 https://acme-v02.api.letsencrypt.org
*   Trying 172.65.32.248:443...
* TCP_NODELAY set
* connect to 172.65.32.248 port 443 failed: Connection timed out
* Failed to connect to acme-v02.api.letsencrypt.org port 443: Connection timed out
* Closing connection 0
curl: (28) Failed to connect to acme-v02.api.letsencrypt.org port 443: Connection timed out

Without the -4 curl also tries to use IPv6, which is configured on the server, but not routed.
If I change the IP of the server from 162.55.72.150 to 162.55.72.146, which we also own, the curl-command works. Since all DNS entries are for the .150 IP nothing else does though.
This leads me to believe, that our IP 162.55.72.150 was blacklisted although I don't know why.

The server is our SSL termination proxy, so it hosts all domain and sub-domain certificates, except a few that have their own dedicated servers. So maybe there are too many certificates hosted on that single IP, which creates too many requests?

I'd be thankful for any input into how to get our IP un-banned or what else this might be.

Best regards

1 Like
2

Welcome to the community @Enobyte

A timeout and unreachable are not the kinds of errors seen with blocked IP's.

I get timeouts trying to reach your domain from my own test server. I realize this is inbound to you while you are showing outbound connection problems. But, should I be able to reach your domain with IP ending in .150? Because if so then it is a generalized comms routing / config problem.

6 Likes
3

Using this online tool https://check-host.net/ is showing several locations around the world having Connection timed out (including the USA) while others are OK
Permanent link to this check report

From my IPv4 location I cannot connect

$ curl -Ii http://matrix.enobyte.com/.well-known/acme-challenge/sometestfile
curl: (28) Failed to connect to matrix.enobyte.com port 80 after 129931 ms: Connection timed out
3 Likes
4

Let's Debug results
https://letsdebug.net/enobyte.com/1375212

Supplemental, both matrix.enobyte.com and enobyte.com have the same IPv4 Address; it's fine.

$ nslookup -q=a matrix.enobyte.com ns1.first-ns.de.
Server:         ns1.first-ns.de.
Address:        213.239.242.238#53

Name:   matrix.enobyte.com
Address: 162.55.72.150

$ nslookup -q=a enobyte.com ns1.first-ns.de.
Server:         ns1.first-ns.de.
Address:        213.239.242.238#53

Name:   enobyte.com
Address: 162.55.72.150
2 Likes
5

Dear @MikeMcQ and @Bruce5051,

Thank you for your fast responses.
Usually the IP itself should forward to the default server (redirect 302 to https://academy.enobyte.com)
We also did notice that there are some connectivity issues which we have yet to completely understand (since a simple reload can fix them) but the issue with letsencrypt has been prevalent for quite some time. Since it did get better by changing the IP of the server I thought it must be related to some sort of restriction placed on the IP.
I also tested the same curl command with other sites (e.g. t-online.de, heise.de) to make sure it's not just the outgoing http/s connection that's not working. These sites were reachable without a problem.

Before posting I also ran letsdebug.net on enobyte.com and it reported no errors.
But now I get the same results as @Bruce5051.
Additionally two more servers with their own IP but on the same network (162.55.72.148 and 162.55.72.149) are starting to have issues, so I'm suspecting our network is degrading right now and I will have to look into that first.
The weird thing is that the letsencrypt issues started popping up about a week ago and just now the whole network is degrading. So I'm not sure that's related, but no point in trying to figure out just one failing connection, if all of them started to fail.

As for both URLs having the same IP:
That's intentional. As I mentioned that's our SSL termination proxy which will forward all requests to different servers inside a private part of the network depending on the incoming URL used.

3 Likes
6

That would be my guess as well.

3 Likes
7

Side note that is an old version of Certbot - Certbot 2.3.0 Release

2 Likes
8

What shows?:
sudo traceroute -4 -T -p 443 acme-v02.api.letsencrypt.org

3 Likes
9

Ok, I got a few things back running and letsdebug now shows a working result for enobyte.com (Let's Debug)
The issue with the SSL termination proxy and letsencrypt remains.

Thanks for letting me know. I updated certbot, but the problem remains the same:

:~# certbot --version
certbot 2.3.0

:~# certbot renew
[...]
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/matrix.enobyte.com.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Failed to renew certificate matrix.enobyte.com with error: Requesting acme-v02.api.letsencrypt.org/directory: Network is unreachable
[...]

The result from the SSL termination proxy with the .150 IP is:

:~# traceroute -4 -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  static.145.72.55.162.clients.your-server.de (162.55.72.145)  0.419 ms  0.406 ms  0.396 ms
 2  core32.hel1.hetzner.com (213.239.245.169)  0.499 ms  0.491 ms core24.fsn1.hetzner.com (213.239.245.165)  12.181 ms
 3  core3.sto.hetzner.com (213.239.245.70)  7.520 ms core3.sto.hetzner.com (213.239.224.17)  11.250 ms  11.288 ms
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

If I change the IP briefly to .146 it works:

:~# traceroute -4 -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  static.145.72.55.162.clients.your-server.de (162.55.72.145)  0.352 ms  0.342 ms  0.356 ms
 2  core32.hel1.hetzner.com (213.239.245.169)  1.318 ms core24.fsn1.hetzner.com (213.239.245.165)  4.809 ms  4.806 ms
 3  core3.sto.hetzner.com (213.239.245.70)  6.907 ms core3.sto.hetzner.com (213.239.252.226)  6.354 ms core3.sto.hetzner.com (213.239.245.70)  6.901 ms
 4  * * *
 5  172.65.32.248 (172.65.32.248)  26.455 ms  26.941 ms  25.625 ms

So it's directly related to the IP, which is why I initially thought that this IP might be banned by a firewall.
But looking at the results from the traceroute I'd guess that our hosting provider (Hetzner) isn't routing correctly?

10

That could be the problem.
If you can NAT all outbound connections to IP 172.65.32.248 via the .146 IP, that would "workaround" the problem.
Of course, you might still have the (lack of a functional) IPv6 problem.
OR
Other (remaining) problems...

3 Likes
11

I was actually already considering doing that, but it's not really solving the underlying issue:
Something in the line is blocking that connection when it shouldn't.

Also I forgot to mention, that the setup (with getting certificates from letsencrypt, the non-routed IPv6 and all) worked for more than a year. This all suddenly stopped working properly about a month ago. But Hetzner claims that they didn't change anything on their end (but still sent us new net-masks that we had to use from then on out, because the previous ones prevented two servers from seeing each other when they could before for over a year).

In any case, I opened a new ticket with Hetzner and am awaiting their response.
Thank you all for your input so far and I'll keep you posted.

3 Likes
12

162.55.72.150

This IP isn't blocked by Let's Encrypt. I see requests from that IP in our staging environment as recently as 2023-02-15 5:43:10.345 PM +0000

5 Likes
13

Dear @mcpherrinm,
Thank you for letting us know! That is pretty interesting, because it implies that the return channel is being blocked and not the outgoing one.

2 Likes
14

That is difficult to determine from seeing only one end of the conversation.

We only know for sure that the round-trip is not equal.
.150 fails
.146 works

But the failure could be in either direction OR both.

4 Likes
15

wonder if there is a way to make a proxy that cause triangle routing so test just one side of path.
like client sent packet by proxy, and proxy do tcp as if it's client, and LE reply back to client

4 Likes
16

NAT forward without changing the source IP.
A real ISP would frown on that kind of IP source spoofing...
But there aren't many real ISPs out there - too many are just out for a quick buck.
So, probably just about any cloud VM system will fit that bill.

4 Likes
17

Just a quick update:
Hetzner hasn't responded yet and I'll be on a conference today and tomorrow, so I'll not be able to try anything too involved like the last two suggestions (also not too sure if I feel comfortable actually spoofing an IP). But I'll keep at it and let you know what worked, although right now I'm considering just changing the DNS entries to a new IP (.146) and considering the .150 burned, since we are getting no useful responses from Hetzner when it comes to their routing (due to other issues we have talked to them a lot about routing recently).

5 Likes
18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.