IPv6 works but Letsencrypt fails to renew via IPv6

My domain is: dev.motrparts.com

I ran this command: sudo certbot renew

It produced this output:
- The following errors were reported by the server:

   Domain: dev.motrparts.com
   Type:   connection
   Detail: Fetching
   https://dev.motrparts.com/.well-known/acme-challenge/84VcFIHnMTCgZbHaRU0Xn6_5WWxn5NZOaI_xsVy2FRQ:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): Nginx 1.17.7

The operating system my web server runs on is (include version): Fedora 30 x64 Cloud Edition

My hosting provider, if applicable, is: DigitalOcean

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 1.0.0

I’ve checked IPv6 access to the website. IPv6 is configured correctly and it’s working fine. I’ve also created a test file at for you to check.
https://dev.motrparts.com/.well-known/acme-challenge/test.txt

Partial output of /var/log/letsencrypt/letsencrypt.log is
“challenges”: [
{
“type”: “http-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:connection”,
“detail”: “Fetching https://dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE: Timeout during connect (likely firewall problem)”,
“status”: 400
},

Nginx logs the request from LetsEncrypt as successful.

200 [17/Jan/2020:04:39:17 +0530] - dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE - 2a05:d014:3ad:700:b22c:ca2c:7496:bfa - "http://dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 87

200 [17/Jan/2020:04:39:18 +0530] - dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE - 2600:1f16:269:da00:4ec6:1cf7:34d5:6263 - "http://dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 87

200 [17/Jan/2020:04:39:18 +0530] - dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE - 2600:1f14:804:fd00:312d:4aad:dca:87f1 - "http://dev.motrparts.com/.well-known/acme-challenge/pzJ5eteWZySIyawouk7o1Uh_ZepTWnydjyHFvWKDIEE" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 87

Please help. I’m unable to find a solution to this issue.

1 Like

Currently, you should be getting 4 requests (due to multi-viewpoint validation). e.g.

2600:3000:2710:200::1e - - [17/Jan/2020:10:24:42 +1100] "GET /.well-known/acme-challenge/Zxe9GtUcjh7_eelloZMYjBl8SGuGUgO5IZFtiJnUZT8 HTTP/1.1" 404 146 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
2600:1f16:269:da01:4e9f:c9d7:3ffe:6166 - - [17/Jan/2020:10:24:42 +1100] "GET /.well-known/acme-challenge/Zxe9GtUcjh7_eelloZMYjBl8SGuGUgO5IZFtiJnUZT8 HTTP/1.1" 404 146 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
2600:1f14:804:fd00:312d:4aad:dca:87f1 - - [17/Jan/2020:10:24:43 +1100] "GET /.well-known/acme-challenge/Zxe9GtUcjh7_eelloZMYjBl8SGuGUgO5IZFtiJnUZT8 HTTP/1.1" 404 146 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
2a05:d014:3ad:702:b5ae:ea8c:29d5:5f5f - - [17/Jan/2020:10:24:43 +1100] "GET /.well-known/acme-challenge/Zxe9GtUcjh7_eelloZMYjBl8SGuGUgO5IZFtiJnUZT8 HTTP/1.1" 404 146 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"

It looks like you are missing the important one (2600:3000:2710:200::1e/ViaWest) which determines the success/failure (the other three currently do not affect the outcome).

Perhaps check that you are not accidentally blocking it via a firewall device, and that there’s not network connectivity problems between the two hosts.

2 Likes

Hi @hronak

there should be 4 different ip addresses.

Looks like you block the critical ipv6 address.

1 Like

I ran the

certbot renew

command again. In nginx log files, I still see only 3 IPs

200 [17/Jan/2020:04:59:40 +0530] - dev.motrparts.com/.well-known/acme-challenge/LldT5HLmlAsUEp-DjQXUHHH5rDLa9dGdqs-5yzyDJF0 - 2a05:d014:3ad:700:b22c:ca2c:7496:bfa - "http://dev.motrparts.com/.well-known/acme-challenge/LldT5HLmlAsUEp-DjQXUHHH5rDLa9dGdqs-5yzyDJF0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 87
200 [17/Jan/2020:04:59:41 +0530] - dev.motrparts.com/.well-known/acme-challenge/LldT5HLmlAsUEp-DjQXUHHH5rDLa9dGdqs-5yzyDJF0 - 2600:1f16:269:da00:4ec6:1cf7:34d5:6263 - "http://dev.motrparts.com/.well-known/acme-challenge/LldT5HLmlAsUEp-DjQXUHHH5rDLa9dGdqs-5yzyDJF0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 87
200 [17/Jan/2020:04:59:41 +0530] - dev.motrparts.com/.well-known/acme-challenge/LldT5HLmlAsUEp-DjQXUHHH5rDLa9dGdqs-5yzyDJF0 - 2600:1f14:804:fd02:1be3:bfea:ffcc:a21f - "http://dev.motrparts.com/.well-known/acme-challenge/LldT5HLmlAsUEp-DjQXUHHH5rDLa9dGdqs-5yzyDJF0" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 87

I’ve checked my firewall. No IPs are blocked. Anything I can do to check why the 4th IP is not reaching my server?

1 Like

Maybe a place to start is a reverse trace?

mtr --report -n -c 10 2600:3000:2710:200::1e
1 Like

This IPv6 address seems down. Even Ping test from KeyCDN fails from all 14 locations.

https://tools.keycdn.com/ipv6-ping

It doesn’t respond to ICMP Ping. That’s completely normal - it’s not down. I wanted to see mtr to see if it’s breaking down somewhere in the middle.

You could also try doing a packet capture to see if any traffic makes it at all during validation:

tcpdump -i eth0 net 2600:3000::/32
1 Like

Here’s the output for mtr --report -n -c 10 2600:3000:2710:200::1e

HOST: dev.motrparts.com           Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  2.|-- fd00:0:2::3                0.0%    10    0.9   1.9   0.6   8.3   2.4
  3.|-- 2604:a880:ffff:b::11       0.0%    10    0.7   1.1   0.6   2.6   0.6
  4.|-- 2404:a800:3a00::b5         0.0%    10    8.5   2.1   1.2   8.5   2.2
  5.|-- 2404:a800::104             0.0%    10  207.5 209.2 207.2 225.3   5.7
  6.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0
  7.|-- 2404:a800::93              0.0%    10  209.3 208.1 207.5 209.3   0.6
  8.|-- 2404:a800::212             0.0%    10  207.3 211.5 207.2 236.3   9.1
  9.|-- ???                       100.0    10    0.0   0.0   0.0   0.0   0.0

The output for tcpdump -i eth0 net 2600:3000::/32 is

05:17:19.383418 IP6 2600:3000:1511:200::1e.59098 > dev.motrparts.com.http: Flags [S], seq 1734051879, win 28800, options [mss 1440,sackOK,TS val 1136233328 ecr 0,nop,wscale 7], length 0
05:17:19.383503 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494915605 ecr 1136233328,nop,wscale 7], length 0
05:17:20.384922 IP6 2600:3000:1511:200::1e.59098 > dev.motrparts.com.http: Flags [S], seq 1734051879, win 28800, options [mss 1440,sackOK,TS val 1136234330 ecr 0,nop,wscale 7], length 0
05:17:20.384978 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494916607 ecr 1136233328,nop,wscale 7], length 0
05:17:21.393738 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494917616 ecr 1136233328,nop,wscale 7], length 0
05:17:22.390869 IP6 2600:3000:1511:200::1e.59098 > dev.motrparts.com.http: Flags [S], seq 1734051879, win 28800, options [mss 1440,sackOK,TS val 1136236336 ecr 0,nop,wscale 7], length 0
05:17:22.390921 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494918613 ecr 1136233328,nop,wscale 7], length 0
05:17:24.401768 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494920624 ecr 1136233328,nop,wscale 7], length 0
05:17:26.398970 IP6 2600:3000:1511:200::1e.59098 > dev.motrparts.com.http: Flags [S], seq 1734051879, win 28800, options [mss 1440,sackOK,TS val 1136240344 ecr 0,nop,wscale 7], length 0
05:17:26.399034 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494922621 ecr 1136233328,nop,wscale 7], length 0
05:17:29.924105 IP6 2600:3000:1511:200::1e.36802 > dev.motrparts.com.https: Flags [S], seq 363942667, win 28800, options [mss 1440,sackOK,TS val 1136243857 ecr 0,nop,wscale 7], length 0
05:17:29.924200 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494926146 ecr 1136243857,nop,wscale 7], length 0
05:17:30.417728 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494926640 ecr 1136233328,nop,wscale 7], length 0
05:17:30.925908 IP6 2600:3000:1511:200::1e.36802 > dev.motrparts.com.https: Flags [S], seq 363942667, win 28800, options [mss 1440,sackOK,TS val 1136244860 ecr 0,nop,wscale 7], length 0
05:17:30.925972 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494927148 ecr 1136243857,nop,wscale 7], length 0
05:17:31.953738 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494928176 ecr 1136243857,nop,wscale 7], length 0
05:17:32.929867 IP6 2600:3000:1511:200::1e.36802 > dev.motrparts.com.https: Flags [S], seq 363942667, win 28800, options [mss 1440,sackOK,TS val 1136246864 ecr 0,nop,wscale 7], length 0
05:17:32.929941 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494929152 ecr 1136243857,nop,wscale 7], length 0
05:17:34.961800 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494931184 ecr 1136243857,nop,wscale 7], length 0
05:17:36.937883 IP6 2600:3000:1511:200::1e.36802 > dev.motrparts.com.https: Flags [S], seq 363942667, win 28800, options [mss 1440,sackOK,TS val 1136250872 ecr 0,nop,wscale 7], length 0
05:17:36.937951 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494933160 ecr 1136243857,nop,wscale 7], length 0
05:17:38.609791 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494934832 ecr 1136233328,nop,wscale 7], length 0
05:17:40.977816 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494937200 ecr 1136243857,nop,wscale 7], length 0
05:17:43.793372 IP6 2600:3000:1511:200::1e.60304 > dev.motrparts.com.http: Flags [S], seq 226420427, win 28800, options [mss 1440,sackOK,TS val 1136257729 ecr 0,nop,wscale 7], length 0
05:17:43.793451 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494940015 ecr 1136257729,nop,wscale 7], length 0
05:17:44.069023 IP6 2600:3000:1511:200::1e.60312 > dev.motrparts.com.http: Flags [S], seq 858176478, win 28800, options [mss 1440,sackOK,TS val 1136258005 ecr 0,nop,wscale 7], length 0
05:17:44.069110 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494940291 ecr 1136258005,nop,wscale 7], length 0
05:17:44.795284 IP6 2600:3000:1511:200::1e.60304 > dev.motrparts.com.http: Flags [S], seq 226420427, win 28800, options [mss 1440,sackOK,TS val 1136258732 ecr 0,nop,wscale 7], length 0
05:17:44.795344 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494941017 ecr 1136257729,nop,wscale 7], length 0
05:17:45.070981 IP6 2600:3000:1511:200::1e.60312 > dev.motrparts.com.http: Flags [S], seq 858176478, win 28800, options [mss 1440,sackOK,TS val 1136259008 ecr 0,nop,wscale 7], length 0
05:17:45.071027 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494941293 ecr 1136258005,nop,wscale 7], length 0
05:17:45.841765 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494942064 ecr 1136257729,nop,wscale 7], length 0
05:17:46.097805 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494942320 ecr 1136258005,nop,wscale 7], length 0
05:17:46.799324 IP6 2600:3000:1511:200::1e.60304 > dev.motrparts.com.http: Flags [S], seq 226420427, win 28800, options [mss 1440,sackOK,TS val 1136260736 ecr 0,nop,wscale 7], length 0
05:17:46.799387 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494943021 ecr 1136257729,nop,wscale 7], length 0
05:17:47.075006 IP6 2600:3000:1511:200::1e.60312 > dev.motrparts.com.http: Flags [S], seq 858176478, win 28800, options [mss 1440,sackOK,TS val 1136261012 ecr 0,nop,wscale 7], length 0
05:17:47.075078 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494943297 ecr 1136258005,nop,wscale 7], length 0
05:17:48.849757 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494945072 ecr 1136257729,nop,wscale 7], length 0
05:17:49.105821 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494945328 ecr 1136258005,nop,wscale 7], length 0
05:17:49.361816 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494945584 ecr 1136243857,nop,wscale 7], length 0
05:17:50.807251 IP6 2600:3000:1511:200::1e.60304 > dev.motrparts.com.http: Flags [S], seq 226420427, win 28800, options [mss 1440,sackOK,TS val 1136264744 ecr 0,nop,wscale 7], length 0
05:17:50.807322 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494947029 ecr 1136257729,nop,wscale 7], length 0
05:17:51.087002 IP6 2600:3000:1511:200::1e.60312 > dev.motrparts.com.http: Flags [S], seq 858176478, win 28800, options [mss 1440,sackOK,TS val 1136265024 ecr 0,nop,wscale 7], length 0
05:17:51.087062 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494947309 ecr 1136258005,nop,wscale 7], length 0
05:17:54.362114 IP6 2600:3000:1511:200::1e.38026 > dev.motrparts.com.https: Flags [S], seq 716044708, win 28800, options [mss 1440,sackOK,TS val 1136268294 ecr 0,nop,wscale 7], length 0
05:17:54.362180 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494950584 ecr 1136268294,nop,wscale 7], length 0
05:17:54.564916 IP6 2600:3000:1511:200::1e.38036 > dev.motrparts.com.https: Flags [S], seq 3695965105, win 28800, options [mss 1440,sackOK,TS val 1136268510 ecr 0,nop,wscale 7], length 0
05:17:54.564987 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494950787 ecr 1136268510,nop,wscale 7], length 0
05:17:54.865767 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494951088 ecr 1136257729,nop,wscale 7], length 0
05:17:54.993717 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.59098: Flags [S.], seq 834148236, ack 1734051880, win 64260, options [mss 1440,sackOK,TS val 2494951216 ecr 1136233328,nop,wscale 7], length 0
05:17:55.121768 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494951344 ecr 1136258005,nop,wscale 7], length 0
05:17:55.364016 IP6 2600:3000:1511:200::1e.38026 > dev.motrparts.com.https: Flags [S], seq 716044708, win 28800, options [mss 1440,sackOK,TS val 1136269296 ecr 0,nop,wscale 7], length 0
05:17:55.364101 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494951586 ecr 1136268294,nop,wscale 7], length 0
05:17:55.566802 IP6 2600:3000:1511:200::1e.38036 > dev.motrparts.com.https: Flags [S], seq 3695965105, win 28800, options [mss 1440,sackOK,TS val 1136269512 ecr 0,nop,wscale 7], length 0
05:17:55.566865 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494951789 ecr 1136268510,nop,wscale 7], length 0
05:17:56.401786 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494952624 ecr 1136268294,nop,wscale 7], length 0
05:17:56.593771 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494952816 ecr 1136268510,nop,wscale 7], length 0
05:17:57.368073 IP6 2600:3000:1511:200::1e.38026 > dev.motrparts.com.https: Flags [S], seq 716044708, win 28800, options [mss 1440,sackOK,TS val 1136271300 ecr 0,nop,wscale 7], length 0
05:17:57.368136 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494953590 ecr 1136268294,nop,wscale 7], length 0
05:17:57.570851 IP6 2600:3000:1511:200::1e.38036 > dev.motrparts.com.https: Flags [S], seq 3695965105, win 28800, options [mss 1440,sackOK,TS val 1136271516 ecr 0,nop,wscale 7], length 0
05:17:57.570913 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494953793 ecr 1136268510,nop,wscale 7], length 0
05:17:59.409779 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494955632 ecr 1136268294,nop,wscale 7], length 0
05:17:59.601795 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494955824 ecr 1136268510,nop,wscale 7], length 0
05:18:01.380097 IP6 2600:3000:1511:200::1e.38026 > dev.motrparts.com.https: Flags [S], seq 716044708, win 28800, options [mss 1440,sackOK,TS val 1136275312 ecr 0,nop,wscale 7], length 0
05:18:01.380163 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494957602 ecr 1136268294,nop,wscale 7], length 0
05:18:01.574838 IP6 2600:3000:1511:200::1e.38036 > dev.motrparts.com.https: Flags [S], seq 3695965105, win 28800, options [mss 1440,sackOK,TS val 1136275520 ecr 0,nop,wscale 7], length 0
05:18:01.574907 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494957797 ecr 1136268510,nop,wscale 7], length 0
05:18:03.185778 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494959408 ecr 1136258005,nop,wscale 7], length 0
05:18:03.185828 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494959408 ecr 1136257729,nop,wscale 7], length 0
05:18:05.425772 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494961648 ecr 1136268294,nop,wscale 7], length 0
05:18:05.617763 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494961840 ecr 1136268510,nop,wscale 7], length 0
05:18:05.745768 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.36802: Flags [S.], seq 18532105, ack 363942668, win 64260, options [mss 1440,sackOK,TS val 2494961968 ecr 1136243857,nop,wscale 7], length 0
05:18:13.937819 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38036: Flags [S.], seq 1376637546, ack 3695965106, win 64260, options [mss 1440,sackOK,TS val 2494970160 ecr 1136268510,nop,wscale 7], length 0
05:18:13.937884 IP6 dev.motrparts.com.https > 2600:3000:1511:200::1e.38026: Flags [S.], seq 2758189233, ack 716044709, win 64260, options [mss 1440,sackOK,TS val 2494970160 ecr 1136268294,nop,wscale 7], length 0
05:18:19.569813 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60304: Flags [S.], seq 2799387671, ack 226420428, win 64260, options [mss 1440,sackOK,TS val 2494975792 ecr 1136257729,nop,wscale 7], length 0
05:18:19.569858 IP6 dev.motrparts.com.http > 2600:3000:1511:200::1e.60312: Flags [S.], seq 1670216494, ack 858176479, win 64260, options [mss 1440,sackOK,TS val 2494975792 ecr 1136258005,nop,wscale 7], length 0
^C
76 packets captured
78 packets received by filter
0 packets dropped by kernel

My /etc/sysconfig/ip6tables is as follows.

*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [0:0]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 25 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 587 -j ACCEPT
-A INPUT -p tcp -m state --state NEW --dport 993 -j ACCEPT
-A INPUT -d fe80::/64 -p udp --dport 546 -m state --state NEW -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT

What about iptables-save (outputs the full current state of your iptables, rather than the configuration on disk).

To my eyes, it looks like traffic is successfully arriving from Let’s Encrypt.

However, the traffic in the other direction (from your server to Let’s Encrypt) is not arriving to the other side.

From that packet capture, it doesn’t look like any TCP handshakes were completed.

The mtr is a little suspicious because it times out at some Indian ISP - what region is your droplet in?

1 Like

My droplet is in Bangalore, India. Any parameter that I can use to force the use of IPv4?

Output of ip6tables-save

# Generated by ip6tables-save v1.8.2 on Fri Jan 17 05:39:14 2020
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [123:72969]
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT
-A INPUT -p ipv6-icmp -j ACCEPT
-A INPUT -i lo -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 80 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 443 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 25 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 587 -j ACCEPT
-A INPUT -p tcp -m state --state NEW -m tcp --dport 993 -j ACCEPT
-A INPUT -d fe80::/64 -p udp -m udp --dport 546 -m state --state NEW -j ACCEPT
-A INPUT -j REJECT --reject-with icmp6-adm-prohibited
-A FORWARD -j REJECT --reject-with icmp6-adm-prohibited
COMMIT
# Completed on Fri Jan 17 05:39:14 2020
1 Like

Trying to issue a certificate for DigitalOcean’s speed test server, it falls back to IPv4. That’s some further evidence of a routing issue.

$ sudo certbot certonly --dry-run --webroot -w /srv/www/certbot -d speedtest-blr1.digitalocean.com
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for speedtest-blr1.digitalocean.com
Using the webroot path /srv/www/certbot for all unmatched domains.
Waiting for verification...
Challenge failed for domain speedtest-blr1.digitalocean.com
http-01 challenge for speedtest-blr1.digitalocean.com
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: speedtest-blr1.digitalocean.com
   Type:   unauthorized
   Detail: Invalid response from
   http://speedtest-blr1.digitalocean.com/.well-known/acme-challenge/Q8Gr3GttVAEJt_NDIu7ZE2_5GNjF926sPRKRsqMc7s4
   [139.59.80.215]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404
   Not Found</h1></center>\r\n<hr><center>"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
1 Like

So, is it possible to solve the issue from my end or do I have to raise a ticket with DigitalOcean to check & fix the routing issue (if possible)?

1 Like

I created a droplet in Bangalore in the same IPv6 range as you (2400:6180:100:d0::83a:8001) with no IPv4 address assigned to the domain, and tried the same thing.

You should open a ticket with DigitalOcean. I think the network path towards Let’s Encrypt is broken.

Edit: here, I’ve attached a .pcap file you can send to them in the support ticket, filter by ipv6.addr == 2600:3000:2710:200::1e.

http.pcap (17.7 KB)

As a temporary workaround, you can try temporary withdraw your AAAA record, issue the certificate, and restore the record again. If DO fix their routing within 90 days, no problem.

3 Likes

Ah! Thanks. I’ll raise a ticket with DigitalOcean and post the update here.

Any specific API hostname of LetsEncrypt that I should mention to them or should I just mention the above IPv6 address (2600:3000:2710:200::1e) and tell them that network path towards that IPv6 address is broken?

2 Likes

Might be best to link them to this thread so they get the full picture. No point being vague :stuck_out_tongue: .

2 Likes

I’ve raised the ticket on DigitalOcean. Waiting for their response.

2 Likes

So, DigitalOcean was generous enough to apply entire month of credit to the account for me to test for the issue.

I did tests in multiple regions. It was working everwhere, even in the BLR region with the new droplet. I came back to my original droplet and started with minimal configuration with a new test-subdomain.

LetsEncrypt fails with HTTPS enabled configuration.

This nginx config works. LetsEncrypt generates certificates even with IPv6 enabled.

server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  example.com;
    root         /usr/share/nginx/html;
    include      /etc/nginx/default.d/*.conf;

    location / {
    }
}

This nginx config doesn’t work. LetsEncrypt gives same timeout error.

server {
    listen       80;
    listen       [::]:80;
    server_name  example.com;
    root         /usr/share/nginx/html;

    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;

    server_name example.com;
    root /usr/share/nginx/html;

    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:10m;
    ssl_session_tickets off;

    ssl_dhparam dhparam.pem;

    ssl_protocols TLSv1.2 TLSv1.3;
    ssl_ciphers ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384;
    ssl_prefer_server_ciphers off;

    add_header Strict-Transport-Security "max-age=63072000" always;

    resolver 8.8.8.8 8.8.4.4 [2001:4860:4860::8888] [2001:4860:4860::8844] valid=60s ipv6=on;
resolver_timeout 2s;

    ssl_stapling on;
    ssl_stapling_verify on;

    ssl_certificate /etc/letsencrypt/live/example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/example.com/privkey.pem;
    ssl_trusted_certificate /etc/letsencrypt/live/example.com/chain.pem;

    location / {
    }
}

In the second configuration, even if you enable TLSv1 & TLSv1.1 it doesn’t work. Use Mozilla SSL Configuration generator to do this.

Can you please test this on your side and see why the second configuration doesn’t work?

I tested this on Fedora 30 & 31 x64 Cloud Edition. Same error on both.

1 Like

Let’s Encrypt’s CA software has some ability to compensate for IPv6 timeouts and retry over IPv4. However, it doesn’t work for redirect targets.

It’s still a fact that IPv6 was – or is – broken and needs to be fixed, but when you turn off the redirect, Let’s Encrypt can work around it more successfully.

(That’s why the Certbot output I posted earlier showed a successful IPv4 connection, instead of displaying an IPv6 timeout error.)

1 Like

I’m totally confused right now.

server {
    listen       80;
    listen       [::]:80;
    server_name  example.com;
    root         /usr/share/nginx/html;

    return 301 https://$host$request_uri;
}

This is my config for all hosts on my server.

However, problem only happens on IPv6. The same config works with IPv4 on LetsEncrypt. LetsEncrypt is able to generate certificate even with redirect (HTTP → HTTPS) on IPv4. The same thing doesn’t work on IPv6.

Am I missing something here? I’m totally confused right now.

1 Like