Unable to request cert due to IPv6 issue

Kami · April 6, 2021, 2:38pm

Hello,

We're having some issues on multiple servers today that cause us to be unable to request new Let's Encrypt certificates. When we request new certs we see connections from the Let's Encrypt IP range be stuck in the SYN-RECV state.
In a tcpdump, we see a SYN, we send the SYN-ACK, but we never get an ACK from the remote server.

When we do a ping6 we get the error "Hop Limit Exceeded". Running an mtr we get:

Host                                                        Loss%   Snt   Last   Avg  Best  Wrst StDev  
1. 2a0b:3100:100:0:213:136:2:21                              0.0%   651    0.3   0.3   0.2   7.7   0.4  
2. lo0.leaf-sw6.bit-2a.network.bit.nl                        0.0%   651    0.3   0.3   0.3   8.4   0.4  
3. xe-1-3-1.jun1.bit-2a.network.bit.nl                       0.0%   651    0.3   1.0   0.3  32.0   2.6  
4. xe-1-0-1.jun1.bit-1.network.bit.nl                        0.0%   651    1.6   0.9   0.3  18.4   1.8  
5. 2001:418:0:5000::1401                                     0.2%   651    9.0   3.3   1.6  42.1   4.3  
6. 2001:728:0:4000::7a                                      80.9%   650    2.0   3.3   2.0  45.6   5.5  
7. adm-bb4-v6.ip.twelve99.net                                0.0%   650   85.8  85.7  85.4  97.0   0.7  
8. prs-bb2-v6.ip.twelve99.net                                0.0%   650   92.8  92.8  92.4  97.0   0.2  
9. rest-bb1-v6.ip.twelve99.net                               0.0%   650   85.5  85.5  85.2  98.9   0.7 
10. ash-b2-v6.ip.twelve99.net                               77.0%   650   85.8  86.1  85.4 114.5   3.0 
11. viawest-svc073699-ic361683.ip.twelve99-cust.net         52.2%   650   90.2  90.1  89.8  95.0   0.4 
12. 2600:3000:0:2::5f2                                      55.3%   650   85.9  86.0  85.6  96.3   0.7 
13. 2600:3000:0:2::3e4                                      43.8%   650  105.1 104.9 104.7 107.5   0.2 
14. 2600:3000:0:2::4a5                                      40.8%   650  106.5 106.5 106.2 109.8   0.2 
15. 2600:3000:0:2::3a                                       38.3%   650  153.6 153.5 153.2 154.6   0.1 
16. 2600:3000:0:2::aa                                       38.5%   650  150.7 150.6 150.3 155.3   0.3 
17. 2600:3000:0:2::8d                                       37.0%   650  153.5 153.4 153.1 156.1   0.1 
18. 2600:3000:1:230::2                                      34.5%   650  146.7 146.5 146.3 152.5   0.3 
19. 2600:3000:0:2::107                                      28.9%   650  146.6 146.6 146.3 149.0   0.1 
20. 2600:3000:0:2::4ad                                      22.3%   650  146.8 146.8 146.4 152.5   0.3 
21. 2600:3000:0:2::453                                      88.0%   650  143.6 143.7 143.4 150.5   0.8 
22. be55.edrt02.clt01.flexential.net                         0.2%   650  146.6 146.4 146.3 150.6   0.3 
23. 2600:3000:0:2::2b0                                      90.3%   650  150.1 150.2 149.9 150.7   0.0 
24. be55.edrt02.clt01.flexential.net                         0.5%   650  147.0 146.5 146.3 182.7   1.5 
25. 2600:3000:0:2::2b0                                      91.7%   650  150.3 150.4 150.0 157.7   1.0 
26. be55.edrt02.clt01.flexential.net                         0.6%   650  147.0 146.4 146.3 149.7   0.2 
27. 2600:3000:0:2::2b0                                      91.4%   649  150.1 150.2 150.0 150.5   0.0 
28. be55.edrt02.clt01.flexential.net                         0.8%   649  146.5 146.4 146.3 150.8   0.3 
29. 2600:3000:0:2::2b0                                      90.0%   648  150.2 150.2 150.0 151.2   0.0 
30. be55.edrt02.clt01.flexential.net                         0.5%   648  146.7 146.4 146.3 151.6   0.4

It appears that something on the remote end is stuck in a loop and causing issues.
Could you assist with this?

Thanks,
Niels

JuergenAuer · April 6, 2021, 2:50pm

Hi @Kami

your complete command and output is required. May be the destination has a wrong ip.

If I try to connect the Letsencrypt API, I don't hit be55.edrt02.clt01.flexential.net.

Looks like an internal configuration problem of that system.

Kami · April 6, 2021, 2:58pm

Hi Juergen,

Thanks for your response.
Sorry, I will elaborate. The commando is: mtr 2600:3000:2710:300::1f

We get the following error when we request a certificate: Timeout during connect (likely firewall problem)
This is using http validation.

When we ran a tcpdump when the validation is pending, this is the range we see the requests coming from. It actually seems to come from multiple IPs in the 2600:3000:2710::/48 range but this is one of the specific IPs.

petercooperjr · April 6, 2021, 3:01pm

I'm not sure if you're saying you're having trouble hitting the Let's Encrypt API over IPv6, or you're seeing problems validating challenges when Let's Encrypt's servers try hitting your server.

There was an IPv6 routing-or-something issue that started yesterday, but I think their current status says they're working around it.

https://letsencrypt.status.io/pages/incident/55957a99e800baa4470002da/606b5c00a8b4db052d1ba2e1

JuergenAuer · April 6, 2021, 3:03pm

Your domain name with that problem is required if you want help.

Supermathie · April 6, 2021, 3:20pm

Agreed: this looks like an IPv6 routing problem.

My validation requests yesterday (after the workaround) came from 2600:3000:1511:200::1f and …::20. Looks like 2600:3000:2710:300::1f is trying to send you the validation but can't establish a TCP connection as the routing back to that network is broken.

I get the same result as you from multiple locations (a routing loop between 2600:3000:0:2::2b0 and 2600:3000:0:2::2b1:

$ mtr -c 5 -n -r 2600:3000:2710:300::1f
Start: Tue Apr  6 15:17:49 2021
HOST: mdd                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
  2.|-- fd00:0:b::271              0.0%     5    0.6   7.1   0.4  33.7  14.9
  3.|-- 2604:a880:ffff:a::211      0.0%     5    0.5   3.9   0.3  17.8   7.8
  4.|-- 2001:2000:3080:63f::1      0.0%     5    0.5   1.0   0.5   2.3   0.5
  5.|-- 2001:2034:0:127::1         0.0%     5   10.9  10.9  10.8  11.3   0.0
  6.|-- 2001:2034:0:12a::1         0.0%     5   10.7  10.9  10.7  11.5   0.0
  7.|-- 2001:2034:0:135::1         0.0%     5   10.9  11.1  10.9  11.3   0.0
  8.|-- 2001:2000:3080:1ddc::2     0.0%     5   11.2  11.3  11.1  11.5   0.0
  9.|-- 2600:3000:0:2::5f4         0.0%     5   11.4  11.4  11.2  11.8   0.0
 10.|-- 2600:3000:0:2::3a          0.0%     5   33.0  33.0  32.8  33.2   0.0
 11.|-- 2600:3000:0:2::aa          0.0%     5   32.6  32.8  32.6  33.2   0.0
 12.|-- 2600:3000:0:2::8d          0.0%     5   32.9  32.9  32.8  33.0   0.0
 13.|-- 2600:3000:1:230::2         0.0%     5   43.1  43.2  43.1  43.6   0.0
 14.|-- 2600:3000:0:2::107         0.0%     5   43.0  43.2  43.0  43.5   0.0
 15.|-- 2600:3000:0:2::4ad         0.0%     5   43.2  43.5  43.2  43.9   0.0
 16.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 17.|-- 2600:3000:0:2::2b1         0.0%     5   42.9  43.0  42.9  43.3   0.0
 18.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 19.|-- 2600:3000:0:2::2b1         0.0%     5   43.0  42.9  42.8  43.1   0.0
 20.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 21.|-- 2600:3000:0:2::2b1         0.0%     5   42.9  43.0  42.9  43.1   0.0
 22.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 23.|-- 2600:3000:0:2::2b1         0.0%     5   43.0  43.1  42.8  43.9   0.0
 24.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 25.|-- 2600:3000:0:2::2b1         0.0%     5   42.9  42.9  42.9  42.9   0.0
 26.|-- ???                        0.0%     0    0.0   0.0   0.0   0.0   0.0

$ mtr -c 5 -n -r 2600:3000:2710:300::1f
Start: 2021-04-06T15:18:30+0000
HOST: tieinterceptor1a            Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2001:470:1:3a8::1          0.0%     5  279.6 196.1   0.4 623.7 263.6
  2.|-- 2001:470:0:1e7::1          0.0%     5    0.5   0.5   0.3   0.6   0.1
  3.|-- 2001:470:0:eb::2           0.0%     5    2.5   1.3   0.9   2.5   0.7
  4.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
  5.|-- 2001:2034:0:133::1         0.0%     5   13.2  13.3  13.2  13.4   0.1
  6.|-- 2001:2034:0:16f::1         0.0%     5    7.7   7.8   7.7   8.0   0.1
  7.|-- 2001:2034:0:16e::1         0.0%     5    7.6   7.7   7.6   7.8   0.1
  8.|-- 2001:2034:0:bf::1          0.0%     5   10.8  11.2  10.6  12.7   0.8
  9.|-- 2001:2000:3080:1e41::2     0.0%     5    8.1   7.9   7.8   8.1   0.2
 10.|-- 2600:3000:0:2::5fe         0.0%     5    8.4   8.4   8.3   8.5   0.1
 11.|-- 2600:3000:0:2::478         0.0%     5   16.4  16.5  16.4  16.6   0.1
 12.|-- 2600:3000:1:220::1         0.0%     5   31.3  31.4  31.3  31.5   0.1
 13.|-- 2600:3000:0:2::108         0.0%     5   31.1  31.3  31.1  31.4   0.2
 14.|-- 2600:3000:0:2::41a         0.0%     5   31.4  31.5  31.4  31.6   0.1
 15.|-- 2600:3000:0:2::4ad         0.0%     5   31.6  31.7  31.5  31.8   0.1
 16.|-- 2600:3000:0:2::453        80.0%     5   31.5  31.5  31.5  31.5   0.0
 17.|-- 2600:3000:0:2::2b1         0.0%     5   32.9  33.0  32.7  33.3   0.2
 18.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 19.|-- 2600:3000:0:2::2b1         0.0%     5   33.1  33.0  32.7  33.2   0.2
 20.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 21.|-- 2600:3000:0:2::2b1         0.0%     5   33.0  33.0  32.9  33.1   0.1
 22.|-- 2600:3000:0:2::2b0        20.0%     5   33.3  33.3  32.8  33.4   0.3
 23.|-- 2600:3000:0:2::2b1         0.0%     5   33.1  33.1  32.9  33.2   0.1
 24.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 25.|-- 2600:3000:0:2::2b1         0.0%     5   32.9  33.0  32.9  33.2   0.2
 26.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0
 27.|-- 2600:3000:0:2::2b1         0.0%     5   33.1  33.0  32.8  33.1   0.1
 28.|-- 2600:3000:0:2::2b0        80.0%     5   33.3  33.3  33.3  33.3   0.0
 29.|-- 2600:3000:0:2::2b1         0.0%     5   32.8  32.9  32.8  33.1   0.1
 30.|-- ???                       100.0     5    0.0   0.0   0.0   0.0   0.0

Thaillie · April 6, 2021, 3:31pm

Heya, Kami's collegue here.
I have just reproduced the issue from one of our test servers, as we are unable to share the customer's domain etc.

The domain I just reproduced the issue with is 'gillendekaketoe.nl'.
This domain lives on a server with the IP's 134.122.49.32 and 2a03:b0c0:2:f0::3b0:d001.

petercooperjr · April 6, 2021, 3:35pm

Hmm. May be time to wave the @lestaff banner to see if they can help. This may be a recurrence of whatever happened yesterday with their IPv6 routing.

Phil · April 6, 2021, 3:43pm

We have an open issue with our upstream network provider about this. We were made aware of ISP maintenance after-the-fact and the IPv6 breakage was unforeseen.

Kami · April 6, 2021, 3:46pm

Hi Phil,

Thanks for confirming this. Would it perhaps be worth mentioning this on https://letsencrypt.status.io/ ?
Currently the latest update on Let's Encrypt Status suggests the issue is mitigated but it seems that it is not fully mitigated yet.

Thanks!
Niels

Phil · April 6, 2021, 4:09pm

As of ~1hr ago we pointed all API and OCSP traffic to an operational datacenter and internal metrics look good. The status.io was resolved, however we're still working with our upstream ISP to fix IPv6 routes for the other datacenter.

Edit: The problem has been totally resolved.

Kami · April 6, 2021, 7:38pm

Hi Phil,

Cheers, I see our queued certificate was also successfully issued now. Looking good, thanks!

system · May 6, 2021, 7:38pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.