Possible IP blocklist?

AntoineG · August 3, 2022, 3:06am

Sure, here's the result

> ping -c 10 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 56(84) bytes of data.
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=49 time=149 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=5 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=6 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=7 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=8 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=9 ttl=49 time=130 ms
64 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=10 ttl=49 time=130 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9322ms
rtt min/avg/max/mdev = 129.770/131.746/148.683/5.645 ms

rg305 · August 3, 2022, 3:09am

So... either:

100%
0%

hmm...

Sounds like possible routing or congestion issues.
15 hops is a lot on the Internet.

rg305 · August 3, 2022, 3:10am

[I'm surprised it didn't choose an IPv6 address]

rg305 · August 3, 2022, 3:11am

What shows?:
ping -c 2 -s 1492 acme-v02.api.letsencrypt.org
ping -c 2 -s 1400 acme-v02.api.letsencrypt.org

AntoineG · August 3, 2022, 3:13am

$ nslookup acme-v02.api.letsencrypt.org
Server:		127.0.0.53
Address:	127.0.0.53#53

Non-authoritative answer:
acme-v02.api.letsencrypt.org	canonical name = prod.api.letsencrypt.org.
prod.api.letsencrypt.org	canonical name = ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
Name:	ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 172.65.32.248
Name:	ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com
Address: 2606:4700:60:0:f53d:5624:85c7:3a2c

$  ping -c 2 -s 1492 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 1492(1520) bytes of data.
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=49 time=244 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
2 packets transmitted, 1 received, 50% packet loss, time 1001ms
rtt min/avg/max/mdev = 244.394/244.394/244.394/0.000 ms

$ ping -c 2 -s 1400 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 1400(1428) bytes of data.
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=49 time=245 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=49 time=245 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1001ms
rtt min/avg/max/mdev = 244.581/244.625/244.669/0.044 ms

rg305 · August 3, 2022, 3:23am

Let's increase these numbers:

Also, it's strange that is doesn't prefer the IPv6 number:

rg305 · August 3, 2022, 3:25am

I get:

ping -c 10 -s 1492 acme-v02.api.letsencrypt.org
PING acme-v02.api.letsencrypt.org(2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c)) 1492 data bytes
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=1 ttl=57 time=9.68 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=2 ttl=57 time=8.66 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=3 ttl=57 time=8.29 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=4 ttl=57 time=8.96 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=5 ttl=57 time=10.2 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=6 ttl=57 time=8.24 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=7 ttl=57 time=9.07 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=8 ttl=57 time=8.21 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=9 ttl=57 time=10.9 ms
1500 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=10 ttl=57 time=7.77 ms

--- acme-v02.api.letsencrypt.org ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 7.778/9.009/10.927/0.955 ms

AntoineG · August 3, 2022, 3:40am

$ ping -c 10 -s 1492 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 1492(1520) bytes of data.
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=49 time=154 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=49 time=154 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=49 time=152 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=49 time=148 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=5 ttl=49 time=161 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=6 ttl=49 time=150 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=7 ttl=49 time=148 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=8 ttl=49 time=154 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=9 ttl=49 time=156 ms
1500 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=10 ttl=49 time=152 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9012ms
rtt min/avg/max/mdev = 147.633/152.760/160.840/3.750 ms

$ ping -c 10 -s 1400 acme-v02.api.letsencrypt.org
PING ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com (172.65.32.248) 1400(1428) bytes of data.
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=1 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=2 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=3 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=4 ttl=49 time=149 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=5 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=6 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=7 ttl=49 time=151 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=8 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=9 ttl=49 time=148 ms
1408 bytes from 172.65.32.248 (172.65.32.248): icmp_seq=10 ttl=49 time=148 ms

--- ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com ping statistics ---
10 packets transmitted, 10 received, 0% packet loss, time 9013ms
rtt min/avg/max/mdev = 147.605/148.118/150.620/0.967 ms

rg305 · August 3, 2022, 3:43am

hmm...
Then the problem seems to be transient/temporary.

Try again:
certbot --apache

rg305 · August 3, 2022, 3:47am

That is still troubling me ...

Show:
curl -6 https://acme-v02.api.letsencrypt.org/
curl -6 https://ipv6.org/

MikeMcQ · August 3, 2022, 3:50am

My ping needs ping -6 to use IPv6. Maybe version or distro difference?

AntoineG · August 3, 2022, 3:51am

That seems to be an issue. Both respond with an issue

$ curl -6 https://acme-v02.api.letsencrypt.org/
curl: (7) Couldn't connect to server
$ curl -6 https://ipv6.org/
curl: (7) Couldn't connect to server

I also tried to do these 2 requests on another of our working server (101.132.227.218). It also have the same errors but can still do certbot --apache without any problems.

rg305 · August 3, 2022, 3:53am

IPv6 seems to be broken:

Please show:
ifconfig | grep -Ei 'add|inet'

AntoineG · August 3, 2022, 3:54am

$ ifconfig | grep -Ei 'add|inet'
br-aadddda641e6: flags=4099<UP,BROADCAST,MULTICAST>  mtu 1500
        inet 172.18.0.1  netmask 255.255.0.0  broadcast 172.18.255.255
        inet6 fe80::42:59ff:fefd:3fa7  prefixlen 64  scopeid 0x20<link>
        inet 172.19.0.1  netmask 255.255.0.0  broadcast 172.19.255.255
        inet6 fe80::42:f6ff:fef5:5568  prefixlen 64  scopeid 0x20<link>
        inet 172.17.0.1  netmask 255.255.0.0  broadcast 172.17.255.255
        inet6 fe80::42:9dff:fecd:c004  prefixlen 64  scopeid 0x20<link>
        inet 172.16.62.211  netmask 255.255.240.0  broadcast 172.16.63.255
        inet6 fe80::216:3eff:fe10:3c89  prefixlen 64  scopeid 0x20<link>
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        inet6 fe80::646f:bff:fe87:22e3  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::11:7ff:feaf:b4a9  prefixlen 64  scopeid 0x20<link>
        inet6 fe80::18aa:8dff:fe2f:e2c0  prefixlen 64  scopeid 0x20<link>

rg305 · August 3, 2022, 3:57am

All those IPv6 addresses are ~~private~~ local ("fe80::").

Please show:
curl -k https://172.65.32.248/

rg305 · August 3, 2022, 3:59am

And let's have a look at the LE log file, after:
certbot --apache -vv

AntoineG · August 3, 2022, 4:03am

On that one I finally have a difference between our working server and the one with an issue. The working server is returning the html content of the page. whereas or problem server returns a timeout:

$ curl -k https://172.65.32.248/
curl: (35) OpenSSL SSL_connect: Connection reset by peer in connection to 172.65.32.248:443

rg305 · August 3, 2022, 4:04am

OK,. we have a constantly failing test!
Now to find the why and the where ...

Is there a firewall?

rg305 · August 3, 2022, 4:06am

Also, try:
curl -k https://8.8.8.8/

AntoineG · August 3, 2022, 4:07am

We're using ufw on the server side but there might some additional settings from the provider (Aliyun)

Here's the status of ufw:

$ ufw status
Status: active

To                         Action      From
--                         ------      ----
22/tcp                     ALLOW       Anywhere                  
22                         ALLOW       Anywhere                  
80/tcp                     ALLOW       Anywhere                  
443/tcp                    ALLOW       Anywhere                  
10000/tcp                  ALLOW       Anywhere                  
10000                      ALLOW       Anywhere                  
1000/tcp                   ALLOW       Anywhere                  
465                        ALLOW       Anywhere                  
8989                       ALLOW       Anywhere                  
9200                       ALLOW       Anywhere                  
22/tcp (v6)                ALLOW       Anywhere (v6)             
22 (v6)                    ALLOW       Anywhere (v6)             
80/tcp (v6)                ALLOW       Anywhere (v6)             
443/tcp (v6)               ALLOW       Anywhere (v6)             
10000/tcp (v6)             ALLOW       Anywhere (v6)             
10000 (v6)                 ALLOW       Anywhere (v6)             
1000/tcp (v6)              ALLOW       Anywhere (v6)             
465 (v6)                   ALLOW       Anywhere (v6)             
8989 (v6)                  ALLOW       Anywhere (v6)             
9200 (v6)                  ALLOW       Anywhere (v6)

Topic		Replies	Views
Receiving tons of Acme Challenges with a successful and active domain name Help	16	1611	March 6, 2022
Unknown problem accessing Let's Encrypt API Help	4	237	July 19, 2024
Lets Encrypt blocked on 2 IP's I'm using Help	4	389	March 28, 2024
I think my Server IP is blocked or something Help	11	573	September 15, 2023
Possibly Blocked by Let’s Encrypt Help	5	102	June 22, 2025

Possible IP blocklist?

Related topics