Connection refused from LE perspective, not from elsewhere

Help
1

My domain is:

gwp.linuxhotel.de
desktop.lab.linuxhotel.de
pad.lab.linuxhotel.de
pad.linuxhotel.de

I ran this command:

systemctl start dehydrated.service
( calls dehydrated -c and then reloads apache config )

It produced this output:
Times are Europe/Berlin timezone

Feb 27 12:40:06 hostname dehydrated[13893]: ERROR: Challenge is invalid! (returned: invalid) (result: ["type"] "http-01"
Feb 27 12:40:06 hostname dehydrated[13893]: ["url"] "https://acme-v02.api.letsencrypt.org/acme/chall/545759006/482241701675/Vv8qCw"
Feb 27 12:40:06 hostname dehydrated[13893]: ["status"] "invalid"
Feb 27 12:40:06 hostname dehydrated[13893]: ["validated"] "2025-02-27T11:40:02Z"
Feb 27 12:40:06 hostname dehydrated[13893]: ["error","type"] "urn:ietf:params:acme:error:connection"
Feb 27 12:40:06 hostname dehydrated[13893]: ["error","detail"] "185.232.103.116: Fetching https://desktop.lab.linuxhotel.de/.well-known/acme-challenge/Ecp0rh24GiRng6xVzl4utmDveQb5PRUqsoS
4wEfcwCU: Connection refused"
Feb 27 12:40:06 hostname dehydrated[13893]: ["error","status"] 400
Feb 27 12:40:06 hostname dehydrated[13893]: ["error"] {"type":"urn:ietf:params:acme:error:connection","detail":"185.232.103.116: Fetching https://desktop.lab.linuxhotel.de/.well-known/ac
me-challenge/Ecp0rh24GiRng6xVzl4utmDveQb5PRUqsoS4wEfcwCU: Connection refused","status":400}

...

My web server is (include version):
Latest Apache on Debian Bookworm

The operating system my web server runs on is (include version):
Debian

My hosting provider, if applicable, is:
Self

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

Dehydrated version: 0.7.0

What I did:

  • Check if the $host/.well-known/acme-challenge/whatever is served properly from multiple network perspectives = :white_check_mark:
    (Other own IP, Deutsche Telekom DSL, Vodafone DSL, Mullvad, Hostsharing, Hetzner) Existing files get served, nonexisting get 404.

  • Check that files created by the acme script are accessible by the webserver :white_check_mark:

  • Check that the firewall does not block incoming TCP 443,80 :white_check_mark:

  • Check that nothing specialcases the LE src ips trying to request :white_check_mark:

  • Check that upstream network does not block LE from reaching us :x:
    (But Email and other stuff works)

  • Check that LE does not block us :x: This is this post.

Our relevant IPs are:
Curl egress 185.232.103.115
A 185.232.103.116
AAAA 2a0f:6480:1::2/64

2

Your IPv6 is not working. If I curl using the -4 option, I get the expected 404 file not found. But with the -6 option, your host at 2a0f:6480:1::2 returns a "Connection refused" ICMP error (due to the TCP handshake returning a RST flag immediately after the initial SYN).

The fact you don't see the IPv6 IP address in the error message is due to a bug in Boulder where it somehow "forgets" which IP family to log in the error message when there is an HTTP to HTTPS redirect. But Let's Encrypt prefers IPv6 and I'm 99.9999 % confident that it's the IPv6 connectivity problem that's causing this issue.

You need to make sure your webserver is also listening on that IPv6 address.

5 Likes
3

Yes, this is the problem. Boulder tries to work around broken IPv6 when it can safely do so, but I was able to reproduce an edge case - which I think this was - where its workaround isn't enough, and could be made more clever. I've opened an issue to see if we can improve this.

3 Likes
4

@JamesLE Interesting! That Boulder behaviour is generally known here on the Community. I would have thought it was also known with the developers and/or an issue was already filed in the past :thinking:

1 Like