Wrong DNS records received from akamai CDN for r3.o.lencr.org letsencrypt server

remizyaka · February 10, 2021, 7:33am

Hi,
I receive network unreachable on one of the servers trying to connect to letsencrypt. This server receives different DNS records than other servers.

dig r3.o.lencr.org


;; QUESTION SECTION:
;r3.o.lencr.org.			IN	A

;; ANSWER SECTION:
r3.o.lencr.org.		81	IN	CNAME	o.lencr.edgesuite.net.
o.lencr.edgesuite.net.	14080	IN	CNAME	a1887.dscq.akamai.net.
a1887.dscq.akamai.net.	20	IN	A	23.58.223.136

The address received is not reachable

requests.exceptions.ConnectionError: HTTPConnectionPool(host='r3.o.lencr.org', port=80): Max retries exceeded with url: / (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f293d5c24e0>: Failed to establish a new connection: [Errno 101] Network is unreachable',))

ping 23.58.223.136
PING 23.58.223.136 (23.58.223.136) 56(84) bytes of data.

--- 23.58.223.136 ping statistics ---
4 packets transmitted, 0 received, 100% packet loss, time 3042ms

Here are DNS records received by working server

dig r3.o.lencr.org

;; QUESTION SECTION:
;r3.o.lencr.org.			IN	A

;; ANSWER SECTION:
r3.o.lencr.org.		103	IN	CNAME	o.lencr.edgesuite.net.
o.lencr.edgesuite.net.	8115	IN	CNAME	a1887.dscq.akamai.net.
a1887.dscq.akamai.net.	14	IN	A	86.124.128.136
a1887.dscq.akamai.net.	14	IN	A	86.124.128.137

Please help with fixing it

rg305 · February 10, 2021, 8:55am

I think they serve different IPs to different networks on the Internet.

Try these addressee; just to test if they react any different:

Name:      a1887.dscq.akamai.net
Addresses: 2600:1402:b800:3::172f:cc09
           2600:1402:b800:3::172f:cc1f
           23.54.169.192
Aliases:   r3.o.lencr.org
           o.lencr.edgesuite.net

remizyaka · February 10, 2021, 9:07am

Yeah these adresses are available, but I don't think it's good Idea if I hardcode DNS records locally.
The problem is that these records returned for my location are not working.
|a1887.dscq.akamai.net.|20|IN|A|23.58.223.136|
|a1887.dscq.akamai.net.|20|IN|A|23.58.223.139|
|a1887.dscq.akamai.net.|20|IN|A|23.58.223.129|

rg305 · February 10, 2021, 9:09am

Understood.
I did not, nor would I ever, ask for anyone to make such a permanent change.
It was just to see if the problem is NOT on your side.

remizyaka · February 15, 2021, 8:53am

Could it be that my location is somehow banned?

test from failed location
wget 23.58.223.130
--2021-02-15 08:47:03-- http://23.58.223.130/
Connecting to 23.58.223.130:80... failed: Connection timed out.

test from another server
wget 23.58.223.130
--2021-02-15 08:47:09-- http://23.58.223.130/
Connecting to 23.58.223.130:80... connected.

JuergenAuer · February 15, 2021, 9:06am

Hi @remizyaka

looks more that your system blocks that ip address.

What says traceroute / tracert?

rg305 · February 15, 2021, 10:00am

Try connecting to other IPs in that same network:
curl -Iki 23.58.223.10
curl -Iki 23.58.223.20
curl -Iki 23.58.223.30
curl -Iki 23.58.223.40

remizyaka · February 18, 2021, 9:07am

Hi

working server traceroute

traceroute 23.58.223.130
traceroute to 23.58.223.130 (23.58.223.130), 30 hops max, 60 byte packets
 1  2.58.126.1 (2.58.126.1)  6.594 ms  6.537 ms  6.479 ms
 2  10.32.35.13 (10.32.35.13)  0.709 ms  0.315 ms  0.645 ms
 3  10.20.0.251 (10.20.0.251)  1.442 ms 10.20.0.250 (10.20.0.250)  1.421 ms 10.20.0.251 (10.20.0.251)  1.753 ms
 4  10.34.35.1 (10.34.35.1)  14.800 ms  15.094 ms *
 5  * * 10.34.35.1 (10.34.35.1)  15.341 ms
 6  a23-58-223-130.deploy.static.akamaitechnologies.com (23.58.223.130)  68.379 ms  68.375 ms  68.317 ms

Not working server traceroute

traceroute 23.58.223.130
traceroute to 23.58.223.130 (23.58.223.130), 30 hops max, 60 byte packets
 1  2.58.126.1 (2.58.126.1)  1.387 ms  1.334 ms  1.342 ms
 2  10.32.35.13 (10.32.35.13)  0.298 ms  0.305 ms  0.283 ms
 3  10.20.0.250 (10.20.0.250)  1.036 ms  1.189 ms 10.20.0.251 (10.20.0.251)  0.987 ms
 4  * 10.34.35.1 (10.34.35.1)  14.720 ms 10.34.35.5 (10.34.35.5)  14.608 ms
 5  10.34.35.1 (10.34.35.1)  14.956 ms  15.176 ms *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Tried to switch off firewalls but without any results

JuergenAuer · February 18, 2021, 9:23am

Looks like that server can't talk with Letsencrypt.

Fix it.

How? No idea, looks like a network problem.

remizyaka · February 18, 2021, 9:39am

It's weird, firewall turned off, as you can see trace is the same as on another server where everything works fine. Also server that have problems connecting to 23.58.223.130 can sucessfully connect to other letsencrypt node 104.76.220.209

 traceroute 104.76.220.209
traceroute to 104.76.220.209 (104.76.220.209), 30 hops max, 60 byte packets
 1  2.58.126.1 (2.58.126.1)  0.285 ms  0.314 ms  0.300 ms
 2  10.32.35.13 (10.32.35.13)  0.272 ms  0.312 ms  0.290 ms
 3  10.20.0.250 (10.20.0.250)  0.986 ms 10.20.0.251 (10.20.0.251)  0.888 ms  1.174 ms
 4  * 10.34.35.5 (10.34.35.5)  14.683 ms 10.34.35.1 (10.34.35.1)  14.717 ms
 5  te0-4-1-3.rcr21.ist01.atlas.cogentco.com (149.14.44.1)  67.332 ms  67.397 ms  67.514 ms
 6  te0-4-1-3.rcr21.ist01.atlas.cogentco.com (149.14.44.1)  67.494 ms  67.295 ms  67.285 ms
 7  be3421.ccr51.beg03.atlas.cogentco.com (130.117.0.94)  73.950 ms  73.931 ms be2549.ccr31.sof02.atlas.cogentco.com (154.54.36.137)  68.632 ms
 8  be3421.ccr51.beg03.atlas.cogentco.com (130.117.0.94)  73.711 ms be3420.ccr51.vie01.atlas.cogentco.com (130.117.0.69)  71.968 ms  71.548 ms
 9  130.117.14.182 (130.117.14.182)  71.510 ms  71.469 ms  70.866 ms
10  130.117.14.182 (130.117.14.182)  70.912 ms  71.362 ms  70.833 ms
11  win-bb3-link.ip.twelve99.net (62.115.114.184)  71.234 ms  71.136 ms *
12  akamai-ic-326165-buca-b1.c.telia.net (213.248.79.66)  94.694 ms buca-b2-link.ip.twelve99.net (62.115.118.51)  86.537 ms buca-b2-link.ip.twelve99.net (62.115.118.49)  87.459 ms
13  a104-76-220-209.deploy.static.akamaitechnologies.com (104.76.220.209)  86.826 ms  86.168 ms  86.155 ms

rg305 · February 18, 2021, 1:24pm

@remizyaka, in your traceroute, lines 2-5 are all private IPs (10/8), so that means they are all within the same ISP.
The problem occurs at line 6 [on the immediate exit from your ISP to the Internet].
On the trace that works, the trace goes no further and line 6 hits the destination [router at line 5 has a direct connection to the network for 6].
But when it fails, both are showing the same exit point (line 5 IP).
So the problem exists between line 5 router [within your ISP] and their immediate next hop address.
I would show these results to your ISP and have them confirm/fix the route [first, before proceeding with any other test/changes].

JuergenAuer · February 18, 2021, 1:31pm

Why is there 10.34.35.1

used, but there

is the 10.34.35.1 with a second ip 10.34.35.5.

I'm not so firm with these outputs. Looks like 10.34.35.1 is a basic address and 10.34.35.5 a virtual network created on that basic ip address (switch with additional virtual configurations).

The virtual network is able to talk with Letsencrypt, the basic ip address isn't.

Looks that the source machine has a wrong routing table.

rg305 · February 18, 2021, 1:39pm

The source machine is 4 hops away from that system (or systems).
Line 5 is operated by the ISP (not the user).
The ISP may be using a load balanced routing cluster.

remizyaka · February 18, 2021, 1:59pm

Thanks, will contact ISP than

system · March 20, 2021, 2:00pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.