"Timeout during connect" but I do get the challenge request

Help
1

Hello everyone.

My domain is: docs.keda.re

I ran this command: certbot --apache

It produced this output:

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: docs.keda.re
  Type:   connection
  Detail: 2001:67c:da8:1004:b474:eff:fe14:9ad3: Fetching http://docs.keda.re/.well-known/acme-challenge/2yqI3pKM4bueGcBFGXvOBJPM6BviosNblA2CceWkqBw: Timeout during connect (likely firewall problem)

My web server is (include version): Apache 2.4.57

The operating system my web server runs on is (include version): Almalinux 9.3

My hosting provider, if applicable, is: self hosted (IPv6 only)

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.6.0

I don't get why I get this error, I do see the validation request coming and passing fine

2600:1f16:269:da00:2215:8b19:3d8d:468c - - [08/Mar/2024:23:25:43 +0100] "GET /.well-known/acme-challenge/2yqI3pKM4bueGcBFGXvOBJPM6BviosNblA2CceWkqBw HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

I tried on another VM on the same network and get the exact same issue (on a certbot nginx)

Thanks.

2

You should see 5 (!) requests. See Let's Encrypt is adding two new remote perspectives for domain validation for more info.

2 Likes
3

Are those compatible with IPv6 only too ?
I don't have any explicit blocking and my subnet is globally reachable so I don't really understand why I don't see them coming.
I could test reachability from various points (like AWS) and did not have any issues.

4

Yes. 

2 Likes
5

Do you have information about their IP or something that could help troubleshoot ? (Or any place I could get more logs from let's encrypt?)

I don't see any other request getting in the network so far. The DNS is visible from everywhere from my checks.

6

The DNS may be but the AAAA address is not reachable by various Let's Encrypt server farms

The increase to 5 vantage points (from 3) just went into Staging a couple days ago but not yet in production according to the post @Osiris linked. So, you should be seeing 3 or at least 2 right now in production.

That said, a test using Let's Debug (link here) uses staging and also gets timeout

I see your DNS CNAMEs. Which is fine. Can you contact the people that run that IPv6 address and check their firewall logs? You could run Let's Debug test and have them check the logs for that time to see how many (should see 5 incoming requests)

nslookup docs.keda.re

docs.keda.re    canonical name = nc1.mlg1.es.net.keda.re.
nc1.mlg1.es.net.keda.re canonical name = ens18.nc1.mlg1.es.v6.net.keda.re.
Name:   ens18.nc1.mlg1.es.v6.net.keda.re
Address: 2001:67c:da8:1004:b474:eff:fe14:9ad3
4 Likes
7

The topic title says "I do get the challenge request".
How exactly do you "get" it? And via IPv6 or IPv4?

[without more context, that statement seems ambiguous]

1 Like
8

By getting the challenge I mean I do receive the challenge http request (over IPv6 as expected, as I don't have IPv4 on this part)

I tested with let's debug, I do see 3 requests coming fine in my http logs (but 404 as expected) but the systems still says there is an issue : Let's Debug

2a01:4f9:c012:ccd0::1 - - [09/Mar/2024:09:50:39 +0100] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 404 4766 "-" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"
2600:1f16:13c:c400:e11d:d6ac:41fc:ed73 - - [09/Mar/2024:09:50:39 +0100] "GET /.well-known/acme-challenge/JaBesHMa_8HGCLommgfv2BhOJZizNg40WH2OgwXg524 HTTP/1.1" 404 4766 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2a05:d016:dcc:9102:4af9:7c77:aaff:5979 - - [09/Mar/2024:09:50:42 +0100] "GET /.well-known/acme-challenge/JaBesHMa_8HGCLommgfv2BhOJZizNg40WH2OgwXg524 HTTP/1.1" 404 4766 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
1 Like
9

Ok it looks like some of the let's encrypt servers cannot be reached from my ASN, this is strange (I see the traffic coming and leaving my network but looks like it's dropped/blocked somewhere in transit on the return packets for some of the let's encrypt server).
I will investigate with my transit provider.

PCAP from my link to my transit
acme.pcap (30.3 KB)

1 Like
10

I agree with your deduction, looking at the SYN,ACK replies from your server, but the repetitive SYN retransmissions from the Let's Encrypt servers, which includes the primary validation location from within the US at Flexential.

2 Likes
11

So we detected that Swisscom is indeed dropping packets from our network, we are working with them to fix that (likely a stale filter on their side)

3 Likes
12

Thanks for update.

Note the increased number of vantage points from Staging is still rolling out.

As this rolls out a successful challenge from staging should show 3 to 5 challenges in your logs. Production right now should show 2-3 but will change to 3 to 5 when that begins to roll out the expanded vantage points (schedule tbd).

When the increased vantage points are fully rolled out you should see 4-5 for a successful challenge

3 Likes
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.