IPv6 on the CT logs

[mnordhoff]

The lovely new CT logs currently only support IPv4:

oak.ct.letsencrypt.org.        59  A  3.14.209.140
oak.ct.letsencrypt.org.        59  A  3.18.34.253
oak.ct.letsencrypt.org.        59  A  18.218.204.248

testflume.ct.letsencrypt.org.  59  A  34.212.84.189
testflume.ct.letsencrypt.org.  59  A  35.163.216.244
testflume.ct.letsencrypt.org.  59  A  52.43.227.29

Can you add IPv6 support?

I’m only asking because I always promote IPv6, and everything else you run is dual-stack. I don’t have any sort of specific problem.

Assuming they’re running on ELBs, it should be as simple as adding AAAA ALIAS records, and/or switching to the dualstack ELB hostnames. (I’m kind of guessing, since I’ve never done it.) Enabling IPv6 should reduce your Route 53 bills, since ALIAS responses are free but NODATA responses are charged.

Of course, enabling things is simpler than solving other issues like monitoring or software support.

[Phil]

HI @mnordhoff,

I’ve filed an internal ticket to track progress on this. I’ll let you know as soon as we get IPv6 deployed to Testflume and Oak. Thanks!

[Phil]

@mnordhoff,
Edit The records have been removed. Please see the decision below.

[mnordhoff]

@Phil

Wait a sec, I’m getting “Connection refused” from the IPv6 addresses.

Something else needs to get kicked.

[JuergenAuer]

@Phil

2620:108:700f::22d4:54bd

Nice

ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it [2620:108:700f::22d4:54bd]:80

port 80 and port 443.

[Phil]

@JuergenAuer Thanks, I noticed that too and should have held off announcing this . Taking a look into it now.

[Phil]

There’s several things going on here.

1. I thought I had all of the IPv6 configuration in our AWS VPCs sussed out. Turns out that I missed assigning IPv6 routes to the routing table. That’s been fixed in our terraform config.
2. A change was made to an ELB security group to allow IPv6 communication. Kubernetes should have done this automatically, but didn’t. I found the following upstream issue that might address this in the future. https://github.com/kubernetes/kubernetes/issues/77745
3. The nginx-ingress controller in use will bind on IPv6 according to https://github.com/kubernetes/ingress-nginx/blob/master/internal/net/net.go#L42-L61. Investigating the container shows that an IPv6 interface doesn’t exist which is why we can’t make successful connections yet.
4. This led me to find that docker by default doesn’t bind on ipv6. I’ll rebuild the AMI tomorrow to enable that feature. Docs: https://docs.docker.com/v17.09/engine/userguide/networking/default_network/ipv6/

[mnordhoff]

I’m sorry I started this thread.

[Osiris]

I’m not sorry you started this thread

IPv6 isn’t “optional” IMHO. Very good this sort of trouble with enabling IPv6 arises, so it can be improved.

[mnordhoff]

Better now than after the world’s relying on it, but I always feel terrible about sparking difficulties for people.

[tdelmas]

The important thing is it doesn’t seams to have impacted the uptime https://ct.grahamedgecombe.com/ so it shouldn’t impact the inclusion in Chrome: https://bugs.chromium.org/p/chromium/issues/detail?id=963693

[Phil]

I appreciate the sentiments. We’ll get this all sorted out and have a better service as a result.

[Phil]

Hi everybody,

After much trial and error with our environment, we have decided to postpone deploying IPv6 to the Testflume and Oak logs. Our plan is to wait for the Kubernetes and AWS ecosystem to mature. Thank you for your patience with me on this one.

For others who may undertake this adventure on their own AWS based CT Logs hopefully this research will help.

Type: ELB (classic Elastic Load Balancer)	
Protocol support: IPv4, IPv6 only in EC2-Classic. We use EC2-VPC which means no IPv6 support on the ELB
OSI: Layer 4	
TLS Termination: Offered, but not mandatory. Termination can be done at the webserver.	
Viability: No	
Docs: https://docs.aws.amazon.com/elasticloadbalancing/latest/classic/elb-internet-facing-load-balancers.html
    
Type: ALB (Application Load Balancer)	
Protocol support: IPv4, IPv6
OSI: Layer 7	
TLS Termination: Mandatory. Can use either the Amazon Certificate Manager or upload a certificate to the Amazon Identity Store. No existing ingress supports uploading arbitrary certificates to the Identity Store meaning we would have to build our own solution or use an Amazon issued certificates on our logs.	
Viability: Maybe	
Docs: https://docs.aws.amazon.com/elasticloadbalancing/latest/application/application-load-balancers.html#ip-address-type

Type: NLB (Network Load Balancer)	
Protocol support: IPv4	
OSI: Layer 4	
TLS Termination: Offered, but not mandatory. Termination can be done at the webserver.	
Viability: No	
Docs: https://docs.aws.amazon.com/elasticloadbalancing/latest/network/create-network-load-balancer.html