Network unreachable for address that works

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mt.r8z.us

I ran this command:

Installed cert-manager via microk8s configured dual stack, using traefik ingress. Service has publicly accessible IPV6 LB IP.

It produced this output:

cert manager pod in namespace default (testing with simple web reflector pod) logs:

I1130 16:00:31.034534 1 solver.go:87] cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="mt.r8z.us" "path"="/.well-known/acme-challenge/(token removed)" "token"="(token removed)”

cert-manager pod in cert-manager namespace logs:

E1130 16:14:42.763988 1 sync.go:186] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perfor
m self check GET request 'http://mt.r8z.us/.well-known/acme-challenge/(token removed)': Get "htt
p://mt.r8z.us/.well-known/acme-challenge/(token removed)": dial tcp [2601:300:4500:202::5:201]:80: connect: network is unreachable" "dnsName"="mt.r8z.us" "resource_kind"="Challenge" "resource_name"="mt.r8z.us-t9gll-598920
55-257349850" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"

NOTE: I can go to that url from outside my network and resolve the challenge token, no problem. There’s no v4 address but the log says it can reach the IPv6 space for some reason, which path I know is good.

My web server is (include version):

Traefik ingress

The operating system my web server runs on is (include version):

microk8s v1.25

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes)

I'm using a control panel to manage my site (no)

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): (“latest” image)

2

Hi @jstevewhite, and welcome to the LE community forum :slight_smile:

Possibly not so good from the entire Internet.
I can also reach it:

curl -Ii http://mt.r8z.us/
HTTP/1.1 200 OK
Accept-Ranges: bytes
Content-Length: 123
Content-Type: text/html
Date: Wed, 30 Nov 2022 16:24:58 GMT
Etag: "6386d8a3-7b"
Last-Modified: Wed, 30 Nov 2022 04:14:27 GMT
Server: nginx/1.20.2

But there may be some Geo-Location blocking that prevents access from other countries.

3 Likes
3

Thanks for the welcome and the response :smiley:

I've tried it from several vpn endpoints and it seems to work.

Something occurred to me, though - does the cert-bot pod itself need to reach the IP? Because pods cannot hairpin back into the traefik ingress metallb-exposed IP.

1 Like
4

I don't think that is the case.

3 Likes
5

OK, I read that wrong.
The failed access is from within your own system/network.
Not sure, but there may be a way to turn off that check.
OR
Update the /etc/hosts to point to the local IP address [and avoid hairpinning].

4 Likes
6

Thanks for the feedback. I've got some sort of odd networking complexity here, I think.

I can curl the metallb exposed (routable) v6 ip from any host on the network, EXCEPT the k8s nodes that are not hosting the address. So my router, my workstation, the internet, all can reach the address, but my k8s nodes ( three nodes, ub1/ub2/ub3; ub2 is hosting the metallb address ) ub1 and ub3 cannot curl the address. Super strange. I can't think of any reason my workstation knows where to send curl -6 http://mt.r8z.us but my k8s nodes (on the same network) cannot figure it out. Firewall at the edge only.

1 Like
7

Do they resolve the name?

3 Likes
8

Yeah, resolve the name no sweat. But I tried the IP. The IP6tables rules were sending the traffic to that address to the traefik load balancer which was not working. I'm trying to roll back to nginx ingress for testing right now, hoping it doesn't break my dual-stack config :smiley: Will report back.

1 Like
9

Update:

It appears the problem is that the Calico deployment in microk8s does not use BGP, but vxlan. Thus even after 'enabling' dual stack, inter-node routing fails. I'll have to go back to vanilla, but that's what broke the self-check.

1 Like
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.