Sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org FAILURE

Help
1

In the same AWS VPC, test server can access the -T -p 443 acme-v02.api.letsencrypt.org but the live servrer CANT ?

My domain is: payments.smartconcepts.co.zm

I ran this command: sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org

It produced this output:
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 * * *
2 * * *

28 * * *
29 * * *
30 * * *

My web server is (include version): nginx

The operating system my web server runs on is (include version): ubuntu 20.04

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.3.0

2

If the first hop of your traceroute is timing out, this is likely to be a local problem with the routing of your server.

  1. Can you traceroute to any other website?
  2. What's ip route say?
4 Likes
3

Here is what I get from my location doing sudo traceroute -T -p 443 payments.smartconcepts.co.zm

$ sudo traceroute -T -p 443 payments.smartconcepts.co.zm
traceroute to payments.smartconcepts.co.zm (18.170.236.29), 30 hops max, 60 byte packets
 1  192.168.1.1 (192.168.1.1)  0.189 ms  0.149 ms  0.219 ms
 2  96.120.60.137 (96.120.60.137)  7.744 ms  7.721 ms  7.703 ms
 3  162.151.125.157 (162.151.125.157)  9.828 ms  9.810 ms  20.604 ms
 4  68.85.243.154 (68.85.243.154)  13.707 ms  13.659 ms  13.619 ms
 5  96.216.60.245 (96.216.60.245)  13.578 ms  13.549 ms  13.510 ms
 6  ae-69-ar01.troutdale.or.bverton.comcast.net (68.85.243.197)  13.923 ms  13.153 ms  17.793 ms
 7  be-36221-cs02.seattle.wa.ibone.comcast.net (68.86.93.53)  21.120 ms  12.181 ms be-36231-cs03.seattle.wa.ibone.comcast.net (68.86.93.57)  12.077 ms
 8  be-2413-pe13.seattle.wa.ibone.comcast.net (96.110.44.94)  12.003 ms be-2113-pe13.seattle.wa.ibone.comcast.net (96.110.44.82)  12.333 ms be-2413-pe13.seattle.wa.ibone.comcast.net (96.110.44.94)  12.229 ms
 9  ae-9.a02.sttlwa01.us.bb.gin.ntt.net (129.250.66.105)  12.238 ms  12.182 ms  17.571 ms
10  ae-2.r24.sttlwa01.us.bb.gin.ntt.net (129.250.2.72)  17.515 ms  17.483 ms  20.472 ms
11  ae-11.r20.nwrknj03.us.bb.gin.ntt.net (129.250.6.176)  81.294 ms  81.264 ms  82.027 ms
12  ae-9.r20.londen12.uk.bb.gin.ntt.net (129.250.6.146)  146.615 ms  147.712 ms  147.321 ms
13  ae-0.a02.londen12.uk.bb.gin.ntt.net (129.250.3.213)  167.101 ms  154.230 ms  147.372 ms
14  212.119.4.66 (212.119.4.66)  152.378 ms  152.315 ms  174.230 ms
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  52.94.35.105 (52.94.35.105)  153.232 ms 52.94.35.117 (52.94.35.117)  164.276 ms 52.94.35.37 (52.94.35.37)  152.076 ms
25  52.94.35.52 (52.94.35.52)  143.749 ms 52.94.35.44 (52.94.35.44)  152.939 ms 52.94.35.106 (52.94.35.106)  151.278 ms
26  * * *
27  52.94.33.82 (52.94.33.82)  152.437 ms * *
28  * * *
29  * * *
30  * * *

$ nslookup -q=all payments.smartconcepts.co.zm
unknown query type: all
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   payments.smartconcepts.co.zm
Address: 18.170.236.29

$ ping payments.smartconcepts.co.zm
PING payments.smartconcepts.co.zm (18.170.236.29) 56(84) bytes of data.
^C
--- payments.smartconcepts.co.zm ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3049ms
1 Like
4

TRACEROUTE TO GOOGLE IS FINE

ubuntu@ip-10-100-1-108:~$ sudo traceroute -T -p 443 google.com
traceroute to google.com (142.250.187.238), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 100.65.8.129 (100.65.8.129) 0.714 ms 100.65.10.65 (100.65.10.65) 1.178 ms 100.65.11.129 (100.65.11.129) 0.696 ms
8 52.94.33.85 (52.94.33.85) 3.028 ms 52.94.33.81 (52.94.33.81) 3.015 ms *
9 52.94.33.206 (52.94.33.206) 2.384 ms 15.230.158.248 (15.230.158.248) 1.842 ms 15.230.159.30 (15.230.159.30) 2.367 ms
10 54.239.101.47 (54.239.101.47) 2.874 ms 150.222.65.69 (150.222.65.69) 1.955 ms 52.95.61.31 (52.95.61.31) 1.999 ms
11 * 100.91.13.96 (100.91.13.96) 3.309 ms 100.91.13.124 (100.91.13.124) 2.671 ms
12 100.100.6.9 (100.100.6.9) 3.352 ms 54.239.43.197 (54.239.43.197) 2.321 ms 54.239.43.179 (54.239.43.179) 2.723 ms
13 100.91.203.104 (100.91.203.104) 2.928 ms 100.91.203.98 (100.91.203.98) 3.006 ms 100.91.203.54 (100.91.203.54) 2.168 ms
14 100.91.203.133 (100.91.203.133) 2.497 ms 100.100.74.67 (100.100.74.67) 6.852 ms 100.91.203.85 (100.91.203.85) 3.058 ms
15 100.100.2.62 (100.100.2.62) 3.015 ms 100.100.2.56 (100.100.2.56) 4.129 ms 100.91.201.68 (100.91.201.68) 3.105 ms
16 99.82.181.139 (99.82.181.139) 2.942 ms 2.132 ms 2.911 ms
17 216.239.41.193 (216.239.41.193) 5.015 ms 100.91.209.87 (100.91.209.87) 2.531 ms 100.91.209.35 (100.91.209.35) 2.077 ms
18 142.251.54.47 (142.251.54.47) 1.974 ms 2.867 ms *
19 100.100.81.198 (100.100.81.198) 114.129 ms 100.100.89.6 (100.100.89.6) 4.212 ms 100.100.80.6 (100.100.80.6) 2.449 ms
20 100.100.73.131 (100.100.73.131) 2.291 ms 100.100.89.3 (100.100.89.3) 2.432 ms 100.100.89.195 (100.100.89.195) 2.413 ms
21 100.100.2.14 (100.100.2.14) 2.921 ms 100.100.2.24 (100.100.2.24) 2.417 ms 100.100.2.28 (100.100.2.28) 3.180 ms
22 lhr25s34-in-f14.1e100.net (142.250.187.238) 3.016 ms * *

5

#ip route
default via 10.100.1.1 dev eth0 proto dhcp src 10.100.1.108 metric 100
10.100.1.0/24 dev eth0 proto kernel scope link src 10.100.1.108
10.100.1.1 dev eth0 proto dhcp scope link src 10.100.1.108 metric 100

I can traceroute to other sites except the acme-v02.api.letsencrypt.org

TRACEROUTE TO BELOW ALSO WORKS exceot acme-v02

ubuntu@ip-10-100-1-108:~$ traceroute letsencrypt.org
traceroute to letsencrypt.org (3.72.140.173), 30 hops max, 60 byte packets
1 ec2-52-56-0-159.eu-west-2.compute.amazonaws.com (52.56.0.159) 9.423 ms ec2-52-56-0-147.eu-west-2.compute.amazonaws.com (52.56.0.147) 8.228 ms ec2-52-56-0-153.eu-west-2.compute.amazonaws.com (52.56.0.153) 5.809 ms
2 100.65.18.224 (100.65.18.224) 6.446 ms 100.65.18.64 (100.65.18.64) 5.544 ms 100.65.19.240 (100.65.19.240) 1.365 ms
3 * 100.66.8.234 (100.66.8.234) 1.065 ms 100.66.8.154 (100.66.8.154) 5.826 ms
4 100.66.11.0 (100.66.11.0) 1.348 ms 100.66.10.66 (100.66.10.66) 4.770 ms 100.66.11.232 (100.66.11.232) 3.846 ms
5 100.66.6.167 (100.66.6.167) 7.607 ms 100.66.7.37 (100.66.7.37) 4.154 ms 100.66.7.135 (100.66.7.135) 29.019 ms
6 100.66.4.251 (100.66.4.251) 7.288 ms 100.66.4.35 (100.66.4.35) 1.709 ms 100.66.4.83 (100.66.4.83) 7.491 ms
7 100.65.10.129 (100.65.10.129) 0.950 ms 100.65.8.1 (100.65.8.1) 1.124 ms 0.632 ms
8 15.230.158.99 (15.230.158.99) 7.913 ms 15.230.158.243 (15.230.158.243) 1.266 ms 52.94.33.81 (52.94.33.81) 2.646 ms
9 52.94.33.202 (52.94.33.202) 1.501 ms 15.230.158.138 (15.230.158.138) 4.934 ms 52.94.33.208 (52.94.33.208) 1.866 ms
10 150.222.65.77 (150.222.65.77) 1.249 ms 54.239.101.125 (54.239.101.125) 2.254 ms 52.95.61.89 (52.95.61.89) 1.444 ms
11 100.92.24.89 (100.92.24.89) 16.786 ms 100.92.24.19 (100.92.24.19) 16.846 ms 100.92.24.37 (100.92.24.37) 15.400 ms
12 100.100.8.71 (100.100.8.71) 33.566 ms 100.100.8.117 (100.100.8.117) 32.496 ms 100.100.8.49 (100.100.8.49) 15.620 ms
13 100.100.64.72 (100.100.64.72) 18.000 ms 100.100.81.72 (100.100.81.72) 17.226 ms 100.100.93.72 (100.100.93.72) 14.974 ms
14 100.100.64.79 (100.100.64.79) 15.269 ms 100.100.73.207 (100.100.73.207) 19.074 ms 100.100.85.143 (100.100.85.143) 19.455 ms
15 100.100.14.98 (100.100.14.98) 17.064 ms 100.100.14.116 (100.100.14.116) 15.270 ms 100.100.14.88 (100.100.14.88) 13.829 ms
16 100.100.16.60 (100.100.16.60) 17.361 ms 100.100.16.20 (100.100.16.20) 17.033 ms 100.100.16.34 (100.100.16.34) 14.589 ms
17 100.95.4.9 (100.95.4.9) 17.810 ms 100.95.20.4 (100.95.20.4) 16.904 ms 100.95.20.6 (100.95.20.6) 22.398 ms
18 240.0.92.8 (240.0.92.8) 16.880 ms 240.0.92.11 (240.0.92.11) 17.909 ms 240.0.92.10 (240.0.92.10) 15.453 ms
19 240.0.92.25 (240.0.92.25) 16.596 ms 240.0.92.29 (240.0.92.29) 17.316 ms 240.0.92.31 (240.0.92.31) 17.353 ms
20 240.0.92.31 (240.0.92.31) 15.527 ms 240.0.92.23 (240.0.92.23) 18.289 ms 240.0.92.22 (240.0.92.22) 13.928 ms
21 242.1.94.161 (242.1.94.161) 25.733 ms 242.1.95.17 (242.1.95.17) 18.781 ms *
22 * * *
23 * * *
24 * * *
25 * * *
26 * * *
27 * * *
28 * * *
29 * * *
30 * * *
ubuntu@ip-10-100-1-108:~$

6

Can you show the route table on the router?:

2 Likes
7

default via 10.100.1.1 dev eth0 proto dhcp src 10.100.1.108 metric 100
10.100.1.0/24 dev eth0 proto kernel scope link src 10.100.1.108
10.100.1.1 dev eth0 proto dhcp scope link src 10.100.1.108 metric 100

What is confusing me is...why is it that other sites are reachable? I can reach :

ubuntu@ip-10-100-1-108:~$ traceroute letsencrypt.org
traceroute to letsencrypt.org (3.72.140.173), 30 hops max, 60 byte packets
1 ec2-52-56-0-159.eu-west-2.compute.amazonaws.com (52.56.0.159) 9.423 ms ec2-52-56-0-147.eu-west-2.compute.amazonaws.com (52.56.0.147) 8.228 ms ec2-52-56-0-153.eu-west-2.compute.amazonaws.com (52.56.0.153) 5.809 ms

Then the Security Groups in AWS are allowing traffic outside.

8

I am thinking aloud. Is it POSSIBLE that letsencrypt has BLOCKED my IP to this server acme-v02.api.letsencrypt.org.

I have been doing experiments with DNS Zone names and changing the DNS name to another IP address on test server.

Is there such a thing as LetsEncrypt BLOCKING My ip 18.170.236.29 (payments.smartconcepts.co.zm ) from raching the certificate because i can reach else where like google.com, etc etc

9

Your traceroute stopped on hop one.
LE is NOT that powerful - LOL

Because they are in other networks.
Your network routing problem seems to overlap IP 172.65.32.248
Which is usually caused by an incorrect "internal" route to 172.16.0.0/12 [or firewall block of]
being defined as 172.16.0.0/[something smaller than 12 and bigger than 7]
Like: 172.16.0.0/9
Which is used as 172.0.0.0/9
IP range 172.0.0.0 to 172.127.255.255

3 Likes
10

Have you used LetsEncrypt from that IP address before or is this a new IP? LetsEncrypt does block IPs, but it's usually tied to abusive behavior or participation in a coordinated attack. It is more likely that you inherited a blocked IP than an IP was suddenly blocked.

More often than not, issues like yours have been due to networking problems within datacenters or their network providers.

2 Likes
12

Might be a red herring. Their traceroute to google.com timed out for 6 hops and then worked :person_shrugging:.

But each machine on this VPC NAT gateway would have the same outgoing IP, right?

OP says that their other server in the same VPC can access the ACME endpoint. If they all share one outbound IP, I don't see how it could be an IP address block either.

No idea what the problem would be at all.

3 Likes
13

That's possible...
But the LE path was all 30 lines of only "* * *"

1 Like
14

Show these route tests:
traceroute -m 5 172.8.1.1
traceroute -m 5 172.16.1.1
traceroute -m 5 172.32.1.1
traceroute -m 5 172.64.1.1
traceroute -m 5 172.128.1.1

1 Like
15

Thanks Team for pointing me in right direction. I have noticed one difference between test server and live server. Live server has vpn routes that are in the range 172.x.x.x/16 which are not in the test server. I have Security group on AWS that allows traffic from 172.x.x.x/16 to reach live server but the securiry group on test server does need these vpn networks. I see an overlap though only of 172.x.x.x/16.

Let me investigate these and also do the route tests as suggested by rg305. My brain is opening up a littlle bit I think. Looks like a routing issue now.

But who can check IP BLOCK on LE so we rule out this thinking ?

2 Likes
16

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 172.8.1.1

traceroute to 172.8.1.1 (172.8.1.1), 5 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 172.16.1.1

traceroute to 172.16.1.1 (172.16.1.1), 5 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 172.32.1.1

traceroute to 172.32.1.1 (172.32.1.1), 5 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 172.64.1.1

traceroute to 172.64.1.1 (172.64.1.1), 5 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 172.128.1.1

traceroute to 172.128.1.1 (172.128.1.1), 5 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 172.65.32.248

traceroute to 172.65.32.248 (172.65.32.248), 5 hops max, 60 byte packets

1 * * *

2 * * *

3 * * *

4 * * *

5 * * *

ubuntu@ip-10-100-1-108:~$ traceroute -m 5 8.8.8.8

traceroute to 8.8.8.8 (8.8.8.8), 5 hops max, 60 byte packets

1 * * ec2-52-56-0-159.eu-west-2.compute.amazonaws.com (52.56.0.159) 0.807 ms

2 100.65.19.0 (100.65.19.0) 0.904 ms * *

3 100.66.8.246 (100.66.8.246) 8.599 ms 100.66.8.228 (100.66.8.228) 6.765 ms 100.66.8.196 (100.66.8.196) 6.724 ms

4 100.66.10.192 (100.66.10.192) 3.608 ms 100.66.11.64 (100.66.11.64) 6.823 ms 100.66.11.134 (100.66.11.134) 7.712 ms

5 100.66.7.33 (100.66.7.33) 3.674 ms 100.66.6.47 (100.66.6.47) 3.830 ms 100.66.7.207 (100.66.7.207) 1.625 ms

ubuntu@ip-10-100-1-108:~$

17

TEAM WE ARE GOOD. Because of overlap, I have added a very specifuc route to LE but the network route 172.0.0.0/16 points somewhere else. :slight_smile: You guys are geniuses.

ubuntu@ip-10-100-1-108:~$ sudo traceroute -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
1 * * *
2 * * *
3 * * *
4 * * *
5 * * *
6 * * *
7 100.65.11.193 (100.65.11.193) 5.873 ms 100.65.9.129 (100.65.9.129) 1.260 ms 100.65.8.97 (100.65.8.97) 1.391 ms
8 15.230.158.101 (15.230.158.101) 2.357 ms 15.230.158.127 (15.230.158.127) 4.708 ms 15.230.158.123 (15.230.158.123) 4.698 ms
9 15.230.158.136 (15.230.158.136) 3.471 ms 15.230.158.114 (15.230.158.114) 2.816 ms 52.94.33.194 (52.94.33.194) 1.687 ms
10 54.239.101.67 (54.239.101.67) 3.533 ms 150.222.65.41 (150.222.65.41) 6.069 ms 150.222.65.87 (150.222.65.87) 1.953 ms
11 100.91.11.184 (100.91.11.184) 3.931 ms 100.91.11.190 (100.91.11.190) 3.487 ms 100.91.211.33 (100.91.211.33) 8.792 ms
12 100.100.6.75 (100.100.6.75) 2.384 ms 100.91.210.8 (100.91.210.8) 2.451 ms 100.91.210.42 (100.91.210.42) 2.481 ms
13 100.91.203.120 (100.91.203.120) 2.178 ms 100.100.86.6 (100.100.86.6) 2.595 ms 3.292 ms
14 100.100.80.131 (100.100.80.131) 2.410 ms 100.100.88.195 (100.100.88.195) 2.470 ms 100.91.203.17 (100.91.203.17) 3.045 ms
15 100.91.201.88 (100.91.201.88) 2.536 ms 100.100.2.72 (100.100.2.72) 7.509 ms 100.91.201.16 (100.91.201.16) 2.191 ms
16 100.91.201.107 (100.91.201.107) 3.121 ms 99.83.89.19 (99.83.89.19) 2.574 ms 100.91.201.131 (100.91.201.131) 2.988 ms
17 172.70.87.2 (172.70.87.2) 9.909 ms 100.91.209.57 (100.91.209.57) 2.046 ms 100.91.209.37 (100.91.209.37) 26.498 ms
18 100.100.6.99 (100.100.6.99) 2.422 ms 172.65.32.248 (172.65.32.248) 3.203 ms 100.100.6.93 (100.100.6.93) 5.060 ms

2 Likes
18

reserver ips range are 172.16.0.0/12 (172.16.0.0~172.31.255.255) I think someone use nonstandard IP range as if they are internal IPs

3 Likes
22

@orangepizza

That should be: 172.16.0.0/12 [255.240.0.0] (172.16.0.0-172.31.255.255)
[which doesn't overlap with LE IP 172.65.32.248]

This is what they used: 172.0.0.0/16 [255.255.0.0] (172.0.0.0-172.0.255.255)
[which also doesn't overlap with LE IP 172.65.32.248]

So, the reason it failed is still unexplained.
But the more direct route seemed to work around that misconfigured route.
Of course, the newly added route might still leave out IPs 172.32.0.0-172.65.32.247 and 172.65.32.249-172.xxx.255.255 from being accessible.
[hard to tell what is actually going on without seeing the actual route tables and/or firewall rules involved]

But I'm glad you got your cert! :slight_smile:

4 Likes
23

Thanks Team LE. The feedback has been very helpful..

2 Likes
24

Just to close the loop here - we practically never put up blocks that would impact routing. It has happened, but we have such blocks in place for very very short periods of time.

All the blocks in the last 12+ months have been at our load balancers, so that clients get an informative message about the nature of the block, and instructions on how to request the block be lifted.

I'm not going to say we won't ever block upstream of our load balancers again, but it's exceedingly rare.

10 Likes