Letsencrypt says it times out (timeout)

rickjanssen · July 1, 2019, 2:11pm

My domain is: kcurv.nl

I ran this command: https://letsdebug.net/kcurv.nl/46963

It produced this output: timeout during connect

My web server is (include version): Apache 2.4

The operating system my web server runs on is (include version): CentOS 6

My hosting provider, if applicable, is: Vimexx

I can login to a root shell on my machine (yes or no, or I don’t know): Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): LetsDebug.net

I can’t figure out why letsencrypt is the only one not being able to contact us.

JuergenAuer · July 1, 2019, 2:27pm

Hi @rickjanssen

that’s curious. There is a check of your domain - https://check-your-website.server-daten.de/?q=kcurv.nl

There it’s able to connect your server via http and https. That’s the same result letsdebug reports - letsdebug can connect your domain.

But Letsencrypt can’t.

Do you have a special, blocking firewall? Looks so.

rickjanssen · July 1, 2019, 2:39pm

I’ve disabled the BGP announcements to JointTransit in our network and suddenly it works again. Might be a problem with Let’s Encrypt, might be a problem with us. We are looking into it.

JuergenAuer · July 1, 2019, 2:51pm

Thanks, good to know.

Looks like that has blocked Letsencrypt.

rickjanssen · July 4, 2019, 10:03am

It’s still happening with Joint Transit enabled, can someone at Let’s Encrypt give us a traceroute from 2600:3000:2710:200::1d?

From us:

traceroute to 2600:3000:2710:200::1d (2600:3000:2710:200::1d), 30 hops max, 80 byte packets
1 2a06:2ec0:1::500 (2a06:2ec0:1::500) 0.314 ms 0.216 ms 0.369 ms
2 2a02:10:0:1::f7:3 (2a02:10:0:1::f7:3) 2.956 ms 1.421 ms 2.901 ms
3 * * *
4 * * *
5 100ge8-1.core1.dub1.he.net (2001:470:0:410::2) 24.536 ms 24.515 ms 24.450 ms
6 100ge5-2.core1.nyc5.he.net (2001:470:0:440::1) 74.218 ms 74.214 ms *
7 100ge4-2.core1.nyc4.he.net (2001:470:0:20a::1) 74.180 ms 74.168 ms 74.349 ms
8 2001:428:601:e::1 (2001:428:601:e::1) 74.887 ms 74.929 ms 74.917 ms
9 * * *
10 * * *
11 * * *
12 * * *
13 * * *
14 * * *
15 * * *

From others:
traceroute to 2600:3000:2710:200::1d (2600:3000:2710:200::1d), 30 hops max, 80 byte packets
1 2a03:4f00:20::1 (2a03:4f00:20::1) 4.051 ms 4.124 ms 4.288 ms
2 te0-0-24.asr1-arn01.breedbandnederland.nl (2a00:c080:0:2a::1) 4.078 ms 4.182 ms 4.287 ms
3 * * *
4 e2-2.rtr2-apl01.breedbandnederland.nl (2a00:c080:0:2b::2) 4.512 ms 4.545 ms 4.649 ms
5 te2-1.rtr1-apl01.breedbandnederland.nl (2a00:c080:0:26::1) 4.306 ms 4.530 ms 4.641 ms
6 te2-2.rtr1-hil01.breedbandnederland.nl (2a00:c080:0:25::1) 4.407 ms 4.236 ms 4.222 ms
7 te0-0-2-3.rtr1-ams01.breedbandnederland.nl (2a00:c080:0:15::2) 5.210 ms 5.321 ms 5.427 ms
8 * * *
9 * * *
10 100ge8-1.core1.dub1.he.net (2001:470:0:410::2) 17.567 ms 17.571 ms 17.604 ms
11 100ge5-2.core1.nyc5.he.net (2001:470:0:440::1) 77.342 ms 77.209 ms 77.290 ms
12 100ge4-2.core1.nyc4.he.net (2001:470:0:20a::1) 77.419 ms 77.462 ms 77.528 ms
13 2001:428:601:e::1 (2001:428:601:e::1) 77.627 ms 77.929 ms 77.982 ms
14 2001:428::205:171:200:219 (2001:428::205:171:200:219) 117.544 ms 117.283 ms 117.253 ms
15 2001:428:3801:208::2 (2001:428:3801:208::2) 118.638 ms 118.397 ms 118.563 ms
16 2600:3000:2:300::1 (2600:3000:2:300::1) 116.168 ms 115.945 ms 116.133 ms
17 2600:3000:1:230::2 (2600:3000:1:230::2) 127.470 ms 127.778 ms 127.743 ms
18 2600:3000:0:2::416 (2600:3000:0:2::416) 130.362 ms 130.469 ms 130.562 ms
19 2600:3000:3:720::2 (2600:3000:3:720::2) 127.475 ms 127.634 ms 127.368 ms
20 2600:3000:2700:1073::4 (2600:3000:2700:1073::4) 126.989 ms 127.052 ms 126.913 ms
21 * * *
22 * * *
23 * * *

schoen · July 5, 2019, 4:48am

@lestaff, could someone please look into this network connectivity question?

rickjanssen · July 9, 2019, 1:05pm

Can someone send us this please?

edit: Can you let me know when this traceroute is being done? Traffic over joint transit is disabled since we can’t requests certificates when its on. When you want to make the traceroute I will have to enable Joint Transit first.

Phil_LE · July 9, 2019, 2:27pm

@rickjanssen and @schoen

Thanks for pinging us, apologies that we didn’t pick up the ball sooner.

@rickjanssen
You can enable your Joint Transit route now. I’ll keep this window open so I get popup notifications upon your response.

rickjanssen · July 9, 2019, 6:34pm

Hi Phil,

I’m so sorry, I thought I had notifications enabled… I’ve enabled it now!

Phil_LE · July 9, 2019, 7:01pm

@rickjanssen
Here’s some ipv6 traceroutes from our firewalls to kcurv.nl.

tracert6 to kcurv.nl (2a06:2ec0:1::97), 30 hops max, 40/8 byte payload/paddata
  1  2600:3000:2700:1073::1 (2600:3000:2700:1073::1)  0.545 ms  0.721 ms  0.329 ms
  2  2600:3000:3:720::1 (2600:3000:3:720::1)  0.946 ms  0.774 ms  0.655 ms
  3  2600:3000:0:2::415 (2600:3000:0:2::415)  0.944 ms  0.687 ms  0.569 ms
  4  2600:3000:1:230::1 (2600:3000:1:230::1)  12.771 ms  12.594 ms  12.404 ms
  5  2600:3000:2:328::2 (2600:3000:2:328::2)  11.973 ms  11.978 ms  11.991 ms
  6  2001:1900:2100::3739 (lag-194.ear2.Denver1.Level3.net)  12.484 ms  12.559 ms  12.573 ms
  7  2001:1900::3:45 (lo-22-v6.edge4.Chicago3.Level3.net)  40.760 ms  37.090 ms  37.189 ms
  8  2001:1900:2100::2ab2 (IPTRIPLEPLA.edge4.Chicago3.Level3.net)  34.201 ms  34.183 ms  34.159 ms
  9  2a03:9d40:fe00:5::1 (2a03:9d40:fe00:5::1)  60.129 ms  60.078 ms  59.998 ms
 10  2a03:9d40:fe00:6::1 (2a03:9d40:fe00:6::1)  50.672 ms  57.718 ms  50.677 ms
 11  2a03:9d40:fe00:10::1 (2a03:9d40:fe00:10::1)  68.726 ms  68.760 ms  68.709 ms
 12  2a03:9d40:fe00:11::1 (2a03:9d40:fe00:11::1)  82.490 ms  82.571 ms  82.493 ms
 13  * * *
 14  * * *
 15  * * *

and

tracert6 to kcurv.nl (2a06:2ec0:1::97), 30 hops max, 40/8 byte payload/paddata
  1  2600:3000:1500:10f3::1 (2600:3000:1500:10f3::1)  0.998 ms  0.898 ms  0.500 ms
  2  2600:3000:2:520::1 (2600:3000:2:520::1)  0.923 ms  0.630 ms  0.473 ms
  3  2600:3000:0:2::ad (2600:3000:0:2::ad)  1.115 ms  0.801 ms  0.664 ms
  4  2600:3000:0:2::aa (2600:3000:0:2::aa)  1.399 ms  0.908 ms  1.087 ms
  5  2600:3000:0:2::8d (2600:3000:0:2::8d)  1.283 ms  1.293 ms  1.112 ms
  6  2600:3000:2:328::2 (2600:3000:2:328::2)  0.496 ms  0.457 ms  0.459 ms
  7  2001:1900:2100::3739 (lag-194.ear2.Denver1.Level3.net)  1.004 ms  1.033 ms  0.879 ms
  8  2001:1900::3:45 (lo-22-v6.edge4.Chicago3.Level3.net)  25.500 ms  25.714 ms  25.560 ms
  9  2001:1900:2100::2ab2 (IPTRIPLEPLA.edge4.Chicago3.Level3.net)  22.648 ms  22.656 ms  22.602 ms
 10  2a03:9d40:fe00:5::1 (2a03:9d40:fe00:5::1)  48.502 ms  48.497 ms  48.456 ms
 11  2a03:9d40:fe00:6::1 (2a03:9d40:fe00:6::1)  39.132 ms  39.189 ms  39.078 ms
 12  2a03:9d40:fe00:10::1 (2a03:9d40:fe00:10::1)  57.168 ms  57.129 ms  57.090 ms
 13  2a03:9d40:fe00:11::1 (2a03:9d40:fe00:11::1)  70.980 ms  70.875 ms  70.937 ms
 14  * * *
 15  * * *
 16  * * *
...
 29  * * 2a06:2ec0:1::97 (web0097.zxcs.nl)  114.710 ms

rickjanssen · July 9, 2019, 7:00pm

Phil_LE:

tracert6 to kcurv.nl (2a06:2ec0:1::97), 30 hops max, 40/8 byte payload/paddata 1 2600:3000:2700:1073::1 (2600:3000:2700:1073::1) 0.545 ms 0.721 ms 0.329 ms 2 2600:3000:3:720::1 (2600:3000:3:720::1) 0.946 ms 0.774 ms 0.655 ms 3 2600:3000:0:2::415 (2600:3000:0:2::415) 0.944 ms 0.687 ms 0.569 ms 4 2600:3000:1:230::1 (2600:3000:1:230::1) 12.771 ms 12.594 ms 12.404 ms 5 2600:3000:2:328::2 (2600:3000:2:328::2) 11.973 ms 11.978 ms 11.991 ms 6 2001:1900:2100::3739 (lag-194.ear2.Denver1.Level3.net) 12.484 ms 12.559 ms 12.573 ms 7 2001:1900::3:45 (lo-22-v6.edge4.Chicago3.Level3.net) 40.760 ms 37.090 ms 37.189 ms 8 2001:1900:2100::2ab2 (IPTRIPLEPLA.edge4.Chicago3.Level3.net) 34.201 ms 34.183 ms 34.159 ms 9 2a03:9d40:fe00:5::1 (2a03:9d40:fe00:5::1) 60.129 ms 60.078 ms 59.998 ms 10 2a03:9d40:fe00:6::1 (2a03:9d40:fe00:6::1) 50.672 ms 57.718 ms 50.677 ms 11 2a03:9d40:fe00:10::1 (2a03:9d40:fe00:10::1) 68.726 ms 68.760 ms 68.709 ms 12 2a03:9d40:fe00:11::1 (2a03:9d40:fe00:11::1) 82.490 ms 82.571 ms 82.493 ms 13 * * * 14 * * * 15 * * *

Thank you, can you run them again? I’ve disabled JT again.

Phil_LE · July 9, 2019, 7:01pm

tracert6 to kcurv.nl (2a06:2ec0:1::97), 30 hops max, 40/8 byte payload/paddata
  1  2600:3000:2700:1073::1 (2600:3000:2700:1073::1)  139.406 ms  6.996 ms  1.298 ms
  2  2600:3000:3:720::1 (2600:3000:3:720::1)  1.180 ms  1.081 ms  1.006 ms
  3  2600:3000:0:2::415 (2600:3000:0:2::415)  1.195 ms  0.902 ms  0.840 ms
  4  2600:3000:1:230::1 (2600:3000:1:230::1)  12.832 ms  12.494 ms  12.750 ms
  5  2600:3000:0:2::8e (2600:3000:0:2::8e)  12.671 ms  12.567 ms  12.564 ms
  6  2600:3000:2:c0::2 (2600:3000:2:c0::2)  13.105 ms  13.079 ms  12.919 ms
  7  2600:3000:2:b60::2 (2600:3000:2:b60::2)  12.551 ms  12.529 ms  12.511 ms
  8  2001:1900:2100::3b99 (lag-101.ear3.Denver1.Level3.net)  13.664 ms  13.750 ms  13.629 ms
  9  2001:1900::3:12e (lo-0-v6.ear1.Dallas1.Level3.net)  26.791 ms  26.752 ms  26.770 ms
 10  2001:1900:4:3::41e (NTT-level3-200G.Dallas1.Level3.net)  27.625 ms  27.406 ms 2600:3002:2:160::2 (2600:3002:2:160::2)  32.613 ms
 11  2001:428:2001:210:0:30:0:1 (2001:428:2001:210:0:30:0:1)  31.398 ms  31.436 ms  31.466 ms
 12  2001:428:0:1:205:171:8:64 (2001:428:0:1:205:171:8:64)  31.127 ms  31.141 ms  36.464 ms
 13  2001:418:0:4000::85 (ae-14.r11.dllstx09.us.bb.gin.ntt.net)  29.772 ms  29.807 ms  29.753 ms
 14  2001:418:0:2000::13a (ae-0.r22.dllstx09.us.bb.gin.ntt.net)  30.461 ms  29.551 ms  29.495 ms
 15  2001:418:0:2000::12 (ae-1.r22.asbnva02.us.bb.gin.ntt.net)  62.104 ms  62.177 ms  62.160 ms
 16  2001:418:0:2000::3b2 (ae-0.r23.asbnva02.us.bb.gin.ntt.net)  62.168 ms  62.185 ms  62.167 ms
 17  2001:418:0:6000::10e (ae-2.r25.amstnl02.nl.bb.gin.ntt.net)  139.241 ms  139.096 ms  139.306 ms
 18  2001:728:0:2000::3a (ae-8.r03.amstnl02.nl.bb.gin.ntt.net)  139.684 ms  139.446 ms  139.674 ms
 19  2001:728:1800:5000::2 (ge-100-0-0-30.r03.amstnl02.nl.ce.gin.ntt.net)  144.601 ms  140.380 ms  140.363 ms
 20  2001:7b8:3:156::6 (2001:7b8:3:156::6)  136.858 ms  145.741 ms  136.811 ms
 21  2a06:2ec0:1::97 (kcurv.nl)  136.624 ms  136.635 ms  136.688 ms

and

tracert6 to kcurv.nl (2a06:2ec0:1::97), 30 hops max, 40/8 byte payload/paddata
  1  2600:3000:1500:10f3::1 (2600:3000:1500:10f3::1)  1.405 ms  0.490 ms  0.729 ms
  2  2600:3000:2:520::1 (2600:3000:2:520::1)  0.719 ms  0.643 ms  0.620 ms
  3  2600:3000:0:2::ad (2600:3000:0:2::ad)  1.140 ms  0.780 ms  0.656 ms
  4  2600:3000:0:2::be (2600:3000:0:2::be)  0.908 ms  0.791 ms  0.532 ms
  5  2600:3000:0:2::75 (2600:3000:0:2::75)  21.229 ms  21.249 ms  21.151 ms
  6  2600:3000:0:2::46 (2600:3000:0:2::46)  21.167 ms  20.981 ms  20.983 ms
  7  2600:3002:2:160::2 (2600:3002:2:160::2)  20.636 ms  20.612 ms  20.621 ms
  8  2001:428:2001:210:0:30:0:1 (2001:428:2001:210:0:30:0:1)  19.387 ms  19.946 ms  19.438 ms
  9  2001:428:0:1:205:171:8:64 (2001:428:0:1:205:171:8:64)  19.482 ms  19.431 ms  19.422 ms
 10  2001:418:0:4000::85 (ae-14.r11.dllstx09.us.bb.gin.ntt.net)  18.048 ms  17.935 ms  17.915 ms
 11  2001:418:0:2000::13a (ae-0.r22.dllstx09.us.bb.gin.ntt.net)  17.745 ms  18.393 ms  17.774 ms
 12  2001:418:0:2000::12 (ae-1.r22.asbnva02.us.bb.gin.ntt.net)  50.098 ms  50.118 ms  50.106 ms
 13  2001:418:0:2000::3b2 (ae-0.r23.asbnva02.us.bb.gin.ntt.net)  53.621 ms  50.390 ms  66.018 ms
 14  2001:418:0:6000::10e (ae-2.r25.amstnl02.nl.bb.gin.ntt.net)  133.693 ms  134.896 ms  133.675 ms
 15  2001:728:0:2000::3a (ae-8.r03.amstnl02.nl.bb.gin.ntt.net)  134.031 ms  134.038 ms  134.058 ms
 16  2001:728:1800:5000::2 (ge-100-0-0-30.r03.amstnl02.nl.ce.gin.ntt.net)  135.818 ms  136.049 ms  135.029 ms
 17  2001:7b8:3:156::6 (2001:7b8:3:156::6)  131.186 ms  139.068 ms  131.097 ms
 18  2a06:2ec0:1::97 (web0097.zxcs.nl)  131.244 ms  131.439 ms  131.182 ms

rickjanssen · July 9, 2019, 7:02pm

Thank you very much Phil!

Phil_LE · July 9, 2019, 7:03pm

Glad to be of assistance.

rickjanssen · July 12, 2019, 1:13pm

This is solved now. I suspect CenturyLink has fixed their problems without responding.