IPv6 works but Letsencrypt fails to renew via IPv6

It looks like different servers are serving IPv4 and IPv6. Or, is there a reverseproxy in front of one or both IPv4, IPv6?

1 Like

No @bruncsak. The same server is serving both the IPs.

1 Like

So there must be a reverseproxy in front of at least one.

1 Like

No, all requests are served directly. No reverse proxy inbetween.

1 Like

OK, so there must be some explanation of the difference of HTTPS options for IPv4-IPv6 relation. Is there any IP version specific web configuration option which may explain any difference?

1 Like

The problem is that there is an IPv6 connectivity problem between DigitalOcean BLR1 and Let’s Encrypt validation servers, isn’t it?

(It still seems to be broken; I just checked.)

1 Like

@mnordhoff, is the timeout confirmed other means from the ACME boulder server than the boulder log itself? For example, with curl? Is the name dev.motrparts.com resolves to 2400:6180:100:d0::19c3:f001 on the boulder server, right?

1 Like

@mnordhoff You’re right. There’s some IPv6 issue on DigitalOcean.

I used this configuration on Nginx and it doesn’t work with IPv6-only domain name.

server {
    listen       80; # This is enabled but the domain name doesn't have any A records
    listen       [::]:80;
    server_name  blrtest01.motrparts.com;
    root         /usr/share/nginx/html;

    location / {
    }
} 

How can I tell DigitalOcean support staff that there’s an issue with their IPv6 path to LetsEncrypt validation servers?

Do you have any sample command with an expected output (like the mtr command above) so that I can show yours and mine output? So that they can get an idea what expected output should be and what’s wrong with their IPv6 connectivity.

As you said above, the output for mtr --report -n -c 10 2600:3000:2710:200::1e on my broken server is

$ mtr --report -n -c 10 2600:3000:2710:200::1e
Start: 2020-01-18T22:47:55+0530
HOST: dev.motrparts.com           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- fd00:0:2::3                0.0%    10    0.7   1.6   0.5   4.3   1.5
  3.|-- 2604:a880:ffff:b::11       0.0%    10    0.5   0.8   0.5   2.4   0.6
  4.|-- 2404:a800:3a00::b5         0.0%    10    1.2   2.3   1.1  10.9   3.0
  5.|-- 2404:a800::104             0.0%    10  207.4 208.2 207.2 216.9   3.1
  6.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  7.|-- 2404:a800::93              0.0%    10  208.1 208.2 207.4 210.1   0.9
  8.|-- 2404:a800::212             0.0%    10  207.2 207.8 206.9 214.4   2.3
  9.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

What output do you get? Will this be enough for them to show that there’s some problem with their IPv6 and LetsEncrypt validation servers?

@hronak, I feel it is important to get answer to that. I am suspicious that there are more than one server running on your site. Development and production mismatch. Please check all IP addresses (both IPv4 and IPv6) in all the systems (ip address show scope global | grep inet), NAT rules for IPv4 address if any, and DNS names cross verify (internal as well if any and external).

1 Like

@bruncsak Only one server is running. There’s no production or development server mismatch. It’s all fine.

I’ve verified it myself after @mnordhoff said that there’s some issue with IPv6.

Renewals and certificate generation are not working over IPv6 on DigitalOcean BLR region.

OK, then my suspicion is just a dead-end. It was triggered by the difference of HTTPS options offered by nginx server based on the underlying carrier protocol IPv4/IPv6. There may be other explanation for that, IP version specific nginx configuration option.

For a comparison, here’s an mtr to that IP from DigitalOcean’s NYC1 location:

$ mtr -brwz 2600:3000:2710:200::1e
Start: Sat Jan 18 17:47:05 2020
HOST: fin                                                                 Loss%   Snt   Last   Avg  Best  Wrst StDev
  1. AS14061 2604:a880:400:d0:ffff:ffff:ffff:fff1                          0.0%    10    2.3   1.2   0.4   3.7   1.0
  2. AS???   2604:a880:ffff:2:1::320                                       0.0%    10    0.8   1.9   0.8   9.8   2.7
  3. AS???   2604:a880:ffff:2:1::318                                       0.0%    10    0.6   3.1   0.5  25.4   7.8
  4. AS3356  lag-105.ear4.Newark1.Level3.net (2001:1900:2100::2a31)        0.0%    10    1.8   1.8   1.7   2.0   0.0
  5. AS3356  lo-0-v6.ear2.Denver1.Level3.net (2001:1900::3:19e)            0.0%    10   48.3  50.4  48.3  56.3   2.9
  6. AS3356  VIAWEST-INT.edge3.Denver1.Level3.net (2001:1900:2100::373a)   0.0%    10   47.8  48.2  47.8  49.4   0.3
  7. AS13649 2600:3000:2:328::1                                            0.0%    10   48.8  49.0  48.7  49.2   0.0
  8. AS13649 2600:3000:1:230::2                                            0.0%    10   60.6  60.6  60.5  60.6   0.0
  9. AS13649 2600:3000:0:2::416                                            0.0%    10   60.6  60.6  60.4  60.6   0.0
 10. AS13649 2600:3000:0:2::2a7                                            0.0%    10   61.0  60.8  60.7  61.0   0.0
 11. AS13649 2600:3000:3:720::2                                            0.0%    10   60.2  62.2  60.1  79.8   6.2
 12. AS13649 2600:3000:2700:1073::4                                        0.0%    10   60.1  60.1  60.0  60.3   0.0
 13. AS???   ???                                                          100.0    10    0.0   0.0   0.0   0.0   0.0

It dies at hop 13 because Let’s Encrypt blocks the traffic inside their network, but you can see that it got all the way to Let’s Encrypt’s ISP, ViaWest, whereas your mtr did not make it that far.

1 Like

They’ve verified that there’s an issue and responded back with this. :frowning:

I’m doing a few other checks, because if the issue is with a mid-route transit provider there’s not much either of us can do. Once the traffic leaves DigitalOcean’s environment, we have no ways to influence those packets :frowning:

They’re still investigating it. I’m patiently waiting for them to resolve it.

Maybe if we all raise tickets in our DO accounts they might take it on priority.

3 Likes

Did my best to explain DigitalOcean that there’s an issue with their IPv6. Referred them to this thread but to no avail.

They say that there didn’t find any issues with their IPv6.

Anyone here working in DigitalOcean? I badly need your help. I don’t know how this is gonna get solved. Or maybe I’ve to move to Linode or some other VPS provider.

1 Like

I wouldn’t be happy with this response. Netops should be able to route around issues like this one. To suggest that they are entirely at the mercy of their transit providers is a bit silly.

As @mnordhoff suggested earlier, you can also avoid this by creating an exception the HTTP to HTTPS redirect produced by your webserver.

So, if this was nginx:

server {
  listen 80;
  listen [::]:80;

  server_name dev.motrparts.com;

  location / {
    return 301 https://$host$request_uri;
  }

  location /.well-known/acme-challenge/ {
    root /var/www/letsencrypt;
  }
}

and you should be able to renew the certificate:

mkdir -p /var/www/letsencrypt
certbot renew --cert-name dev.motrparts.com -a webroot -w /var/www/letsencrypt

This comes down to a subtle detail about how Let’s Encrypt’s validation server deals with IPv4/IPv6 addresses, timeouts and redirects.

3 Likes

If the problem is with DigitalOcean’s ISP, or with DigitalOcean’s ISP’s ISP, DigitalOcean can escalate it to their ISP, and their ISP can escalate it to their ISP’s ISP. In the meantime, as _az said, they can try to route around it.

I’ve always had a reasonably good experience with reporting IPv6 routing issues to DigitalOcean, for what it’s worth. (For problems with their network or their ISP’s network, at least.)

3 Likes

I sent them this

On a fresh droplet I’ve created in my client’s account with the below configuration, I’ve only ran a few commands to reproduce the issue.

Configuration: Fedora 31 x64, $5, BLR1, IPv6 & Monitoring, SSH Keys, Hostname=your.domain.com

Domain: AAAA record for testing.motrparts.com points to 2400:6180:100:d0::92d:d001
Note: A record is not added for this domain. It’s important to reproduce the issue.

Commands I ran on the fresh droplet

==========
dnf install certbot vim nginx iptables-services
vim /etc/sysconfig/ip6tables # to open port 80 & 443
systemctl enable ip6tables && systemctl start ip6tables
vim /etc/nginx/nginx.conf # To edit the below line.
#----------
# listen 80 default_server; #IPv4 has to be disabled for this to reproduce the issue
listen [::]:80 default_server;
server_name your.domain.com;
#----------
systemctl start nginx && systemctl enable nginx
certbot certonly --webroot -w /usr/share/nginx/html/ -d your.domain.com
==========

To which they responded

Hey there,

Thanks for sharing that. It’s hard to say exactly what is preventing this endpoint from communicating with your Droplet’s web server. If you have no forms of traffic filtering enforced locally, it’s certainly possible that there’s an issue with the route that traffic is taking from one of Let’s Encrypt’s endpoints to your Droplet. The trouble is that there is not IP address shared in the log files and the Let’s Encrypt project refuses to publish an IP address range that we could at least investigate (Need a list of Let's Encrypt IP addresses) Since these will change depending on region and other factors, we do not have a way of determining where the restriction would occur.

All I can confirm is that we have no factors on our end that would prevent traffic from reaching your Droplet over IPv6. If Let’s Encrypt has a restriction on their end that prevents it from reaching their server, there is regretfully nothing that can be done to offset or counter that. With this in mind, there is the potential for a permanent solution here that can help you get the same accomplished without much struggle through using a separate plugin for authentication such as the following:

certbot run -a dns-digitalocean --dns-digitalocean-credentials config.ini -i nginx -d domain.com

In order to run this plugin, you’ll just need to make sure that you have the DigitalOcean DNS plugin for certbot installed by grabbing it from your package manager: python3-certbot-dns-digitalocean.

More information on running this command is available here:

https://certbot-dns-digitalocean.readthedocs.io/en/stable/
https://certbot.eff.org/docs/using.html#combining-plugins

If you find that you’re more interested in tracing the issue with the networking, we would require the Let’s Encrypt team to communicate what IP address is failing that lookup as well as an MTR report (mtr -6rwbzc100 x.x.x.x) from your Droplet to that endpoint as well as in reverse. Short of that, we have no actionable information to review network connectivity problems.

I’m seriously frustrated right now and I don’t know if I should continue with DigitalOcean further. This is my client’s server. I’m gonna send them a proposal to move to Linode or other VPS provider.

DigitalOcean was my go-to VPS provider. Not anymore. They aren’t even trying to reproduce the issue.

Just 6 commands and they could reproduce the issue. Is that so hard?

2 Likes

Let’s Encrypt doesn’t publish the IP addresses, but they’re not hard to guess, and they don’t change frequently. (Yet.)

They can try investigating the IP address posted in this thread.

It’s possible they have no IPv6 route to AS13649 at all.

Yes, that would be useful. But it’s possible one direction’s information would be enough.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.