Can't access acme-v02.api.letsencrypt.org over IPv6

victor-chew · December 6, 2022, 3:34am

Have tried researching related topics in the forum, but no joy. Please help.

On a VPS that I am maintaining , I am suddenly unable to renew cert.

# curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f53d:5624:85c7:3a2c...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=gzlong.asia
*  start date: Oct 23 21:29:30 2022 GMT
*  expire date: Jan 21 21:29:29 2023 GMT
*  subjectAltName does not match acme-v02.api.letsencrypt.org
* SSL: no alternative certificate subject name matches target host name 'acme-v02.api.letsencrypt.org'
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'acme-v02.api.letsencrypt.org'

# curl -4 https://acme-v02.api.letsencrypt.org/directory
{
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert",
  "tsR3cMGWtGA": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417"

# curl -6 https://acme-v02.api.letsencrypt.org/directory
curl: (51) SSL: no alternative certificate subject name matches target host name 'acme-v02.api.letsencrypt.org'

Seems to indicate that this is only happening with IPv6, whereas IPv4 works.

Any pointers would be might appreciated!

MikeMcQ · December 6, 2022, 3:43am

Hmm. Not sure this will help but that IP is not for the acme-v02 Let's Encrypt endpoint.

When I try that IP with http:// I get a response from a Netlify server. An http request to the acme-v02 servers will fail with "connection reset" because it does not support http (only https://). Thus, proving that IP does not belong to Let's Encrypt.

Also, your -v output shows a cert belonging to gzlong.asia which is not LE

Looks like your DNS resolving for IPv6 is faulty but you'll need better DNS experts than me to give more advice than that.

curl -v http://[2600:1f1c:471:9d00:c3a7:6694:870f:2072]
*   Trying 2600:1f1c:471:9d00:c3a7:6694:870f:2072:80...
* Connected to 2600:1f1c:471:9d00:c3a7:6694:870f:2072 (2600:1f1c:471:9d00:c3a7:6694:870f:2072) port 80 (#0)
> GET / HTTP/1.1
> Host: [2600:1f1c:471:9d00:c3a7:6694:870f:2072]
> User-Agent: curl/7.81.0
> Accept: */*
>
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Server: Netlify
< X-Nf-Request-Id: 01GKJR5BW7EWEBMGENX8GY3WWN
< Date: Tue, 06 Dec 2022 03:37:37 GMT
< Content-Length: 0
<
* Connection #0 to host 2600:1f1c:471:9d00:c3a7:6694:870f:2072 left intact

webprofusion · December 6, 2022, 4:13am

So acme-v02.api.letsencrypt.org does point to prod.api.letsencrypt.org which in turn points to ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com, which is 2606:4700:60:0:f53d:5624:85c7:3a2c

It could be a cached DNS response?

webprofusion · December 6, 2022, 4:22am

Does the name notional.finance mean anything? That's someone who currently has an AAAA record for that IP.

victor-chew · December 6, 2022, 4:33am

Does the name notional.finance mean anything? That's someone who currently has an AAAA record for that IP.

No, doesn't mean anything to me.. I will try to figure out why the DNS resolution is faulty.

In the mean time, if any of you have more tips or ideas, please continue to send them this way.

Thanks!

victor-chew · December 6, 2022, 4:53am

OK, this is embarrassing. In my original post, I mucked up copy/pasting the output. I must have spent too much time banging at this problem now. I have updated the original post, and taken pains to verify that the pasted output is correct.

As you can see, the IPv6 DNS resolution appears to be correct:

root@node1:~# curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f53d:5624:85c7:3a2c...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=gzlong.asia
*  start date: Oct 23 21:29:30 2022 GMT
*  expire date: Jan 21 21:29:29 2023 GMT
*  subjectAltName does not match acme-v02.api.letsencrypt.org
* SSL: no alternative certificate subject name matches target host name 'acme-v02.api.letsencrypt.org'
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'acme-v02.api.letsencrypt.org'

Just to make sure, I tried connecting to "acme-staging-v02.api.letsencrypt.org", and that appears to be correct too.

root@node1:~# curl -v https://acme-staging-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f41b:d4fe:4325:6026...
* TCP_NODELAY set
* Connected to acme-staging-v02.api.letsencrypt.org (2606:4700:60:0:f41b:d4fe:4325:6026) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=gzlong.asia
*  start date: Oct 23 21:29:30 2022 GMT
*  expire date: Jan 21 21:29:29 2023 GMT
*  subjectAltName does not match acme-staging-v02.api.letsencrypt.org
* SSL: no alternative certificate subject name matches target host name 'acme-staging-v02.api.letsencrypt.org'
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'acme-staging-v02.api.letsencrypt.org'

MikeMcQ · December 6, 2022, 5:06am

Does gzlong.asia mean anything to you? You still show that in the curl debug output as the cert for the destination

Could that be some security appliance monitoring your outbound connections on IPV6?

Do IPV6 connections to other domains show that same cert?

rg305 · December 6, 2022, 5:09am

I must agree.
Something is doing a MiTM on your outbound IPv6 connections.

victor-chew · December 6, 2022, 5:16am

Does gzlong.asia mean anything to you?

No. I tried googling for it, and nothing turned up.

Do IPV6 connections to other domains show that same cert?

Yes!

# curl -v https://www.google.com
* Rebuilt URL to: https://www.google.com/
*   Trying 2607:f8b0:4004:c19::63...
* TCP_NODELAY set
* Connected to www.google.com (2607:f8b0:4004:c19::63) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=gzlong.asia
*  start date: Oct 23 21:29:30 2022 GMT
*  expire date: Jan 21 21:29:29 2023 GMT
*  subjectAltName does not match www.google.com
* SSL: no alternative certificate subject name matches target host name 'www.google.com'
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
* TLSv1.2 (OUT), TLS alert, Client hello (1):
curl: (51) SSL: no alternative certificate subject name matches target host name 'www.google.com'

Damn, I am not sure what to do now...

rg305 · December 6, 2022, 5:26am

I'd turn off IPv6 [if you can use IPv4] until you find out what is going on with that path.
If that is not an option, then check that you aren't using a proxy and check any systems along the way for HTTPS inspection.

Please show:
traceroute -6 -n -T -p 443 acme-v02.api.letsencrypt.org

victor-chew · December 6, 2022, 5:29am

# traceroute -6 -n -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c), 30 hops max, 80 byte packets
 1  2606:4700:60:0:f53d:5624:85c7:3a2c  2.578 ms  2.565 ms  2.539 ms
 2  2606:4700:60:0:f53d:5624:85c7:3a2c  2.530 ms  2.487 ms  2.493 ms

victor-chew · December 6, 2022, 5:33am

"ps -axjf" as root shows no unusual processes. No proxy running on system. In fact, this has been a long-running, pretty stable VPS that has not seen any changes for a couple of years now (except the usual apt-update, apt dist-upgrade performed regularly).

ifconfig shows:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 50.115.168.9  netmask 255.255.255.0  broadcast 50.115.168.255
        inet6 fda2:6e17:d3d:0:216:3eff:fedb:93e3  prefixlen 64  scopeid 0x0<global>
        inet6 fdc3:63f5:f8b5:0:216:3eff:fedb:93e3  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::216:3eff:fedb:93e3  prefixlen 64  scopeid 0x20<link>
        inet6 2604:c00:88:33:216:3eff:fedb:93e3  prefixlen 64  scopeid 0x0<global>
        inet6 fd38:8f88:1b28:0:216:3eff:fedb:93e3  prefixlen 64  scopeid 0x0<global>
        inet6 fd5d:652b:f924:0:216:3eff:fedb:93e3  prefixlen 64  scopeid 0x0<global>
        ether 00:16:3e:db:93:e3  txqueuelen 1000  (Ethernet)
        RX packets 9963151546  bytes 1958937633546 (1.7 TiB)
        RX errors 0  dropped 22993683  overruns 0  frame 0
        TX packets 887178560  bytes 1239190390273 (1.1 TiB)
        TX errors 0  dropped 1 overruns 0  carrier 0  collisions 0

eth0:0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 208.89.215.119  netmask 255.255.255.0  broadcast 208.89.215.255
        ether 00:16:3e:db:93:e3  txqueuelen 1000  (Ethernet)

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1  (Local Loopback)
        RX packets 506  bytes 119535 (116.7 KiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 506  bytes 119535 (116.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

webprofusion · December 6, 2022, 5:39am

Does env | grep -i proxy return anything?

rg305 · December 6, 2022, 5:40am

Hop #1 is doing HTTPS inspection.

victor-chew · December 6, 2022, 5:41am

No, nothing.

rg305 · December 6, 2022, 5:42am

The output should look more like:

traceroute -6 -n -T -p 443 acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c), 30 hops max, 80 byte packets
 1  [redacted]  14.618 ms * *
 2  2600:1700:[redacted]  17.297 ms  17.462 ms  17.458 ms
 3  * * *
 4  * * *
 5  2001:1890:ff:ff0a:12:242:116:17  20.467 ms  20.812 ms  27.805 ms
 6  2001:1890:c07:1f65::111e:7693  29.216 ms  16.204 ms  13.899 ms
 7  2400:cb00:363:3::  29.496 ms 2400:cb00:368:3::  15.683 ms 2400:cb00:363:3::  28.581 ms
 8  2606:4700:60:0:f53d:5624:85c7:3a2c  25.180 ms  21.834 ms  21.033 ms

Where only the last line shows 2606:4700:60:0:f53d:5624:85c7:3a2c.

victor-chew · December 6, 2022, 5:46am

Just to confirm, same with google.com

# traceroute -6 -n -T -p 443 google.com
traceroute to google.com (2607:f8b0:4004:c1b::66), 30 hops max, 80 byte packets
 1  2607:f8b0:4004:c1b::66  0.654 ms  0.654 ms  0.469 ms
 2  2607:f8b0:4004:c1b::66  0.671 ms  0.785 ms  0.649 ms

victor-chew · December 6, 2022, 5:53am

I am seriously considering doing this because there's no solution in sight.

AFAIK, it simply involves adding a line to sysctl.conf and rebooting. Would that be a big problem for a web server?

Thanks.

rg305 · December 6, 2022, 6:31am

Not likely.
How many concurrent connections?
How fast can it restart?

victor-chew · December 6, 2022, 8:03am

Small web server running nginx. Max 100 workers. Restarts in ~10s

Topic		Replies	Views
IPv6 broken on acme-v01.api.letsencrypt.org? Help	5	4216	December 10, 2015
Tor exit nodes blocked from accessing LE acmev2 api Help	3	889	April 30, 2021
Can't Renew Certificate Help	10	1301	July 2, 2021
Failed to connect to acme-v02.api.letsencrypt.org Help	5	185	August 16, 2024
Curl: (56) OpenSSL SSL_read: SSL_ERROR_SYSCALL, errno 104 Help	2	818	August 14, 2024

Can't access acme-v02.api.letsencrypt.org over IPv6

Related topics