We're having some issues on multiple servers today that cause us to be unable to request new Let's Encrypt certificates. When we request new certs we see connections from the Let's Encrypt IP range be stuck in the SYN-RECV state.
In a tcpdump, we see a SYN, we send the SYN-ACK, but we never get an ACK from the remote server.
When we do a ping6 we get the error "Hop Limit Exceeded". Running an mtr we get:
Thanks for your response.
Sorry, I will elaborate. The commando is: mtr 2600:3000:2710:300::1f
We get the following error when we request a certificate: Timeout during connect (likely firewall problem)
This is using http validation.
When we ran a tcpdump when the validation is pending, this is the range we see the requests coming from. It actually seems to come from multiple IPs in the 2600:3000:2710::/48 range but this is one of the specific IPs.
I'm not sure if you're saying you're having trouble hitting the Let's Encrypt API over IPv6, or you're seeing problems validating challenges when Let's Encrypt's servers try hitting your server.
There was an IPv6 routing-or-something issue that started yesterday, but I think their current status says they're working around it.
My validation requests yesterday (after the workaround) came from 2600:3000:1511:200::1f and …::20. Looks like 2600:3000:2710:300::1f is trying to send you the validation but can't establish a TCP connection as the routing back to that network is broken.
I get the same result as you from multiple locations (a routing loop between 2600:3000:0:2::2b0 and 2600:3000:0:2::2b1:
Heya, Kami's collegue here.
I have just reproduced the issue from one of our test servers, as we are unable to share the customer's domain etc.
The domain I just reproduced the issue with is 'gillendekaketoe.nl'.
This domain lives on a server with the IP's 134.122.49.32 and 2a03:b0c0:2:f0::3b0:d001.
We have an open issue with our upstream network provider about this. We were made aware of ISP maintenance after-the-fact and the IPv6 breakage was unforeseen.
Thanks for confirming this. Would it perhaps be worth mentioning this on https://letsencrypt.status.io/ ?
Currently the latest update on Let's Encrypt Status suggests the issue is mitigated but it seems that it is not fully mitigated yet.
As of ~1hr ago we pointed all API and OCSP traffic to an operational datacenter and internal metrics look good. The status.io was resolved, however we're still working with our upstream ISP to fix IPv6 routes for the other datacenter.