IPv6 not working on the staging us-west-2 VA

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: myip.mn0.us, myipv4.mn0.us, myipv6.mn0.us and other domains

I ran this command: sudo -H certbot certonly --dry-run --webroot -w /srv/www/certbot -d domain

It produced this output: :no_mouth:

My web server is (include version): Nginx 1.17.6

The operating system my web server runs on is (include version): Ubuntu 16.04

My hosting provider, if applicable, is: DigitalOcean and Linode

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot-auto 1.0.0

At the moment, if I try to do HTTP validation for a dual-stack hostname, the us-west-2 staging VA on 34.211.60.134 sends the request over IPv4. For example, my HTTP logs show:

2600:1f16:13c:c401:997b:eeb6:8474:2eb1 myip.mn0.us - [01/Jan/2020:20:52:44 +0000] "GET /.well-known/acme-challenge/46jLT7CqLUmbtib3C6MmJa3EWZqwKbuBljuW2bHhzCs HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2600:3000:2710:300::1e myip.mn0.us - [01/Jan/2020:20:52:45 +0000] "GET /.well-known/acme-challenge/46jLT7CqLUmbtib3C6MmJa3EWZqwKbuBljuW2bHhzCs HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2a05:d014:531:8601:650c:f4b0:fea1:6f1d myip.mn0.us - [01/Jan/2020:20:52:45 +0000] "GET /.well-known/acme-challenge/46jLT7CqLUmbtib3C6MmJa3EWZqwKbuBljuW2bHhzCs HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

34.211.60.134 myip.mn0.us - [01/Jan/2020:20:52:45 +0000] "GET /.well-known/acme-challenge/46jLT7CqLUmbtib3C6MmJa3EWZqwKbuBljuW2bHhzCs HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

(There are two servers and HTTP validation inevitably fails for that domain. The point is what’s attempted. I’ve also checked other domains where it passes.)

An IPv4-only domain looks fine, of course:

18.224.20.83 myipv4.mn0.us - [01/Jan/2020:20:45:33 +0000] "GET /.well-known/acme-challenge/AjKTApMnXOmsYnoZRFgDoeOtPQzIxMEp8XAiYEYGywk HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
34.211.60.134 myipv4.mn0.us - [01/Jan/2020:20:45:33 +0000] "GET /.well-known/acme-challenge/AjKTApMnXOmsYnoZRFgDoeOtPQzIxMEp8XAiYEYGywk HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
52.58.118.98 myipv4.mn0.us - [01/Jan/2020:20:45:33 +0000] "GET /.well-known/acme-challenge/AjKTApMnXOmsYnoZRFgDoeOtPQzIxMEp8XAiYEYGywk HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
66.133.109.36 myipv4.mn0.us - [01/Jan/2020:20:45:33 +0000] "GET /.well-known/acme-challenge/AjKTApMnXOmsYnoZRFgDoeOtPQzIxMEp8XAiYEYGywk HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

An IPv6-only domain logs 3 requests instead of 4:

2600:3000:2710:300::1d myipv6.mn0.us - [01/Jan/2020:20:46:07 +0000] "GET /.well-known/acme-challenge/aN9fJ4FhjOKyP0iY7BJGAJBkx0dJmjUYxLQhp3vIEL0 HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
2a05:d014:531:8601:650c:f4b0:fea1:6f1d myipv6.mn0.us - [01/Jan/2020:20:46:07 +0000] "GET /.well-known/acme-challenge/aN9fJ4FhjOKyP0iY7BJGAJBkx0dJmjUYxLQhp3vIEL0 HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

2600:1f16:13c:c401:997b:eeb6:8474:2eb1 myipv6.mn0.us - [01/Jan/2020:20:46:07 +0000] "GET /.well-known/acme-challenge/aN9fJ4FhjOKyP0iY7BJGAJBkx0dJmjUYxLQhp3vIEL0 HTTP/1.1" 404 209 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Has IPv6 been disabled on that VA for testing or something? Is there an outage?

Edit: I believe the production VA is working.

1 Like
2

Thanks for the report @mnordhoff,

There isn't any intentional change to IPv6 connectivity in the staging environment. I'll pass this thread to the folks with the required access to investigate.

Thanks,

3 Likes
3

@mnordhoff,

The problem was due to cloud-init not enabling IPV6_AUTOCONF by default. We were unaware of this because servers are automatically rebooted at deploy time for kernel/package upgrades and the issue would “resolve” itself. If there are no kernel/package upgrades, the server wouldn’t be rebooted. There are definitely improvements we can make to this process.

Moving forward, the cloud-init issue was fixed by Canonical back in November, but we’ve not yet received an updated package. As a workaround, we deployed a cloud-init fix allowing RVA project servers to automatically reconfigure their networking interfaces and properly establish a default IPv6 gateway.

On the staging us-west-2 server I can see tcp6 connections being established now.

3 Likes
4

Yuck, what a pain. Thank you for investigating it and deploying a workaround. :smile:

Verified fixed – I’m seeing requests from 2600:1f14:a8b:502:ab14:8938:11c5:b75e now.

2 Likes
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.