Connectivity issues to acme-v02.api.letsencrypt.org

Help
1

Whenever I connect to the ACME API from my server, whether through an ACME client or curl, the HTTPS request hangs without giving any response, not even an error. Strangely, pings work just fine, and the same /directory endpoint works fine from my local machine. I’m connecting from a Hetzner machine, patchouli.emily.moe, with IPv4 address 144.76.4.116 and IPv6 addresses 2a01:4f8:190:7700::1/56.

This started happening abruptly while debugging client issues, after I foolishly ran into renewal rate limits and then accidentally registered a couple accounts for bonus points.

emily@patchouli ~> curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f53d:5624:85c7:3a2c:443...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[hangs]^C

emily@patchouli ~> curl -v4 https://acme-v02.api.letsencrypt.org/directory
*   Trying 172.65.32.248:443...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
[hangs]^C

emily@patchouli ~> ping -c4 acme-v02.api.letsencrypt.org
PING acme-v02.api.letsencrypt.org(2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c)) 56 data bytes
64 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=1 ttl=59 time=5.13 ms
64 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=2 ttl=59 time=5.12 ms
64 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=3 ttl=59 time=5.12 ms
64 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=4 ttl=59 time=5.13 ms

--- acme-v02.api.letsencrypt.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 5.119/5.124/5.130/0.004 ms

emily@patchouli ~> traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  static.97.4.76.144.clients.your-server.de (144.76.4.97)  0.448 ms  0.393 ms  0.375 ms
 2  core24.fsn1.hetzner.com (213.239.229.53)  0.582 ms  0.557 ms  0.541 ms
 3  core0.fra.hetzner.com (213.239.252.37)  4.906 ms core4.fra.hetzner.com (213.239.229.73)  4.973 ms core5.fra.hetzner.com (213.239.224.250)  4.974 ms
 4  core9.fra.hetzner.com (213.239.252.18)  7.821 ms core9.fra.hetzner.com (213.239.224.174)  5.087 ms core9.fra.hetzner.com (213.239.224.221)  5.071 ms
 5  162.158.84.254 (162.158.84.254)  5.778 ms  6.054 ms  6.036 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

emily@patchouli ~> uname -a
Linux patchouli 5.5.9-hardened #1-NixOS SMP Thu Mar 12 06:18:49 UTC 2020 x86_64 GNU/Linux
1 Like
2

Try lowering the MTU size.

You could try to determine the largest usable MTU with PING.
Use “don’t fragment” flag and keep increasing/decreasing the packet size.

1 Like
3

If those suggestions don’t help, we can also ask some of the Let’s Encrypt staff to look into possible CDN connectivity problems.

1 Like
4

I’m not entirely sure I have the ping incantations right, but it doesn’t look like an MTU issue:

emily@patchouli ~> ip addr show dev enp4s0 | grep mtu
2: enp4s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000

emily@patchouli ~> ping -M do -s 1452 -c 4 acme-v02.api.letsencrypt.org
PING acme-v02.api.letsencrypt.org(2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c)) 1452 data bytes
1460 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=1 ttl=59 time=5.16 ms
1460 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=2 ttl=59 time=5.16 ms
1460 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=3 ttl=59 time=5.19 ms
1460 bytes from 2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c): icmp_seq=4 ttl=59 time=5.29 ms

--- acme-v02.api.letsencrypt.org ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3004ms
rtt min/avg/max/mdev = 5.159/5.200/5.290/0.053 ms

emily@patchouli ~> ping -M do -s 1453 -c 4 acme-v02.api.letsencrypt.org
PING acme-v02.api.letsencrypt.org(2606:4700:60:0:f53d:5624:85c7:3a2c (2606:4700:60:0:f53d:5624:85c7:3a2c)) 1453 data bytes
ping: local error: message too long, mtu: 1500
1 Like
5

Looks similar to this thread 4 days ago (also Hetzner): My server ip blocked (unblock IP please)

The same hops leading upto the dropped trace probes: core9.fra.hetzner.com followed by 162.158.84.254 (Cloudflare).

Maybe you will be able to get Hetzner’s attention if you point out that two completely different customers on two different subnets are suffering from similar issues talking to Cloudflare’s network … good luck.

3 Likes
6

The problem seems to have vanished as quickly as it appeared, although that could just mean it’s temporarily working again if this is a persistent connectivity issue with Hetzner. I submitted a support ticket to them, though given that it’s working again I’m not sure I’ll get anything useful in reply.

Since I’m not the only person having issues like this, I’d definitely appreciate it if anyone from Let’s Encrypt could look into these connectivity issues and see if anything looks obviously wrong.

2 Likes
7

@lestaff, could someone take a look at this? Two different Hetzner hosting customers seem to have reported similar reachability problems for the ACME endpoint in a fairly short timeframe.

1 Like
8

I am an hetzner client as well, I can run some tests if you tell me what you need.

root@Quake:~# curl -v https://acme-v02.api.letsencrypt.org/directory
*   Trying 2606:4700:60:0:f53d:5624:85c7:3a2c...
* TCP_NODELAY set
* Connected to acme-v02.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
*  subject: CN=acme-v01.api.letsencrypt.org
*  start date: Feb  6 18:06:22 2020 GMT
*  expire date: May  6 18:06:22 2020 GMT
*  subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x562a224f8580)
> GET /directory HTTP/2
> Host: acme-v02.api.letsencrypt.org
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Connection state changed (MAX_CONCURRENT_STREAMS updated)!
< HTTP/2 200 
< server: nginx
< date: Tue, 24 Mar 2020 10:49:00 GMT
< content-type: application/json
< content-length: 658
< cache-control: public, max-age=0, no-cache
< x-frame-options: DENY
< strict-transport-security: max-age=604800
< 
{
  "WAx3Em3iqUI": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-v02.api.letsencrypt.org left intact
}root@Quake:~#
1 Like
9

sigh:

Dear Client,

we have moved the ticket to the network department, in order to help you with this request. Unfortunately we came to the conclusion that this problem is not detected by us.
Packages for our dedicated root servers and virtual servers do not include software support. The packages also do not include support for configuring these servers. For dedicated root and virtual servers, we only provide the hardware, network access and necessary infrastructure; and of course, we support our customers in case there are any failures or disruptions. Since you are a customer of one of our dedicated root servers or virtual servers, you are responsible for the configuration and administration of your server and for the software which runs on it.

Unfortunately I don't think Hetzner are going to be any help with diagnosing the underlying cause of this issue.

1 Like
10

They either don’t understand the problem or just don’t want to look into it.
The software on your system merely detected a problem on the network and was affected by it.
There is no amount of configuration that can be done to your server to correct a network problem (outside your server).
All you can do is “workaround” the network problem and pray that enough people complain so that they realize it is not just you.

1 Like
11

Yeah, I replied more explicitly clarifying that it was a connectivity issue rather than software and they asked for a traceroute from both ends. I don’t have a great amount of faith in Hetzner doing much to fix this issue if it reoccurs, but hopefully Cloudflare might be able to chase things up more proactively?

1 Like
12

We must realize that neither is responsible for the entire Internet.
That said, your traceroutes/PING/etc. tests could point to exactly which IP/device is causing the problem.
From there they can speak to whomever operates that device.
[and also run their own tests and confirm your findings]

1 Like
13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.