"Connection reset by peer"

I’ve been using https://github.com/diafygi/acme-tiny for years on many servers.

Since yesterday, from a very specific subnet, the script constantly fails in random locations (where curl happens), with urlopen error [Errno 104] Connection reset by peer.

It fails in:

The servers I have problem from fall into the 78.108.34.* range.

Sometimes I reproduce this with plain curl as well:

# connection fail:
$ curl https://acme-v02.api.letsencrypt.org/acme/new-order
curl: (35) gnutls_handshake() failed: Error in the pull function.

# connection success (the "malformed" is just an expected API response):
$ curl https://acme-v02.api.letsencrypt.org/acme/new-order
{
  "type": "urn:ietf:params:acme:error:malformed",
  "detail": "Method not allowed",
  "status": 405

Servers which I manage in Hetzner do not have this issue.

Is there a possibility that the problematic range has been blocked or throttled from letsencrypt for some reason?

Have you tried (or can you try) lowering the MTU size?

Sorry in case the pasted block confused you. I’ve edited my question with clarifying that the second curl is successful (it’s just the expected API response).

Can you show to which LE IP you connected to?

Doing curl -vvv https://acme-v02.api.letsencrypt.org/acme/new-acct from any of the locations I manage (both failing, and succeeding) shows:

Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)

I get the same IPv4 address and have no problem with either of those URLs.

Does the problematic system support:
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384

Can you try connecting to something more generic:
curl -vvv https://google.com/

Yes, connectivity via curl works, just not all of the times. The issue is that it now constantly fails via the https://github.com/diafygi/acme-tiny I use.

My curl versions are:

  • curl 7.58.0 (x86_64-pc-linux-gnu) libcurl/7.58.0 OpenSSL/1.1.1 zlib/1.2.11 libidn2/2.0.4 libpsl/0.19.1 (+libidn2/2.0.4) nghttp2/1.30.0 librtmp/2.3
  • curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Hmm…
then I revert back to my original request:

[just as a “test” : not permanently : 1400-1460 should be fine]
[continued success via lower MTU would prove there exist some problematic device(s) along the path]

Thanks for following up so promptly.

I’ve lowered with ifconfig ens3 mtu 1400 and confirmed with ip addr | grep mtu. I tried 1300 and 1400 and, again, it fails with urlopen error [Errno 104] Connection reset by peer in random places (once in https://acme-v02.api.letsencrypt.org/acme/new-nonce and the second time in https://acme-v02.api.letsencrypt.org/directory).

Ok then that’s ruled out…
Not sure what else to try.
You might be right about IPS type blocks on that IP range.

Hi @cherouvim

what says

traceroute acme-v02.api.letsencrypt.org

and ping?

ping -s 1400 -M do acme-v02.api.letsencrypt.org

If that shows errors, reduce the packet size.

1 Like
$ traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  .......... (78.108.....)  0.425 ms  0.396 ms  0.354 ms
 2  78.108..... (78.108.....)  8.120 ms  8.088 ms  8.011 ms
 3  cloudflare.thess.gr-ix.gr (185.1.123.10)  8.995 ms  8.967 ms  8.916 ms
 4  * * *
 5  * * *
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

Ping with 1400 shows no error.

There

you see the problem.

There is a blocking instance.

But from other (working) locations (e.g in Hetzner), I get similar traceroutes:

$ traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  static.......clients.your-server.de (...)  0.134 ms  0.105 ms  0.089 ms
 2  static.......clients.your-server.de (...)  0.328 ms  0.338 ms  0.376 ms
 3  core24.fsn1.hetzner.com (213.239.203.193)  0.439 ms core23.fsn1.hetzner.com (213.239.203.189)  3.789 ms core24.fsn1.hetzner.com (213.239.203.193)  0.462 ms
 4  core1.fra.hetzner.com (213.239.203.153)  16.790 ms core5.fra.hetzner.com (213.239.224.254)  19.280 ms core4.fra.hetzner.com (213.239.203.149)  27.257 ms
 5  core9.fra.hetzner.com (213.239.252.18)  6.710 ms core9.fra.hetzner.com (213.239.224.221)  5.241 ms core9.fra.hetzner.com (213.239.224.174)  5.229 ms
 6  162.158.84.254 (162.158.84.254)  13.266 ms  14.859 ms  14.837 ms
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *

And another traceroute from Belgium (where from https://github.com/diafygi/acme-tiny works). Note the cloudflare at 5:

$ traceroute acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 60 byte packets
 1  ... (...)  0.562 ms  0.554 ms  0.491 ms
 2  ... (...)  0.731 ms  0.714 ms  0.677 ms
 3  100.64.11.165 (100.64.11.165)  0.766 ms  0.741 ms  0.709 ms
 4  100.64.11.54 (100.64.11.54)  1.700 ms  1.319 ms  1.766 ms
 5  cloudflare.bnix.net (194.53.172.66)  1.270 ms  1.575 ms  1.664 ms
 6  * * *
 7  * * *
 8  * * *
 9  * * *
10  * * *
11  * * *
12  * * *
13  * * *
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *

Try:
traceroute -I --mtu acme-v02.api.letsencrypt.org

There are other users with similar problems. Some Hetzner ip addresses don’t work.

Ask Hetzner.

From working locations:

$ sudo traceroute -I --mtu acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 65000 byte packets
 1  ... (...)  0.622 ms F=1500  0.519 ms  0.414 ms
 2  ... (...)  0.631 ms  0.592 ms  0.560 ms
 3  100.64.11.165 (100.64.11.165)  0.703 ms  0.696 ms  0.663 ms
 4  100.64.11.54 (100.64.11.54)  1.328 ms  1.067 ms  1.234 ms
 5  cloudflare.bnix.net (194.53.172.66)  2.184 ms  2.071 ms  2.014 ms
 6  172.65.32.248 (172.65.32.248)  0.861 ms  0.898 ms  1.013 ms


$ sudo traceroute -I --mtu acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 65000 byte packets
 1  static.....clients.your-server.de (...)  0.125 ms F=1500  0.040 ms  0.039 ms
 2  static.....clients.your-server.de (...)  1.598 ms  0.297 ms  0.273 ms
 3  core24.fsn1.hetzner.com (213.239.203.193)  2.918 ms  3.344 ms  5.528 ms
 4  core1.fra.hetzner.com (213.239.229.77)  4.868 ms  4.877 ms  5.308 ms
 5  core9.fra.hetzner.com (213.239.224.174)  17.095 ms  5.139 ms  5.078 ms
 6  162.158.84.254 (162.158.84.254)  13.691 ms  13.105 ms  13.388 ms
 7  172.65.32.248 (172.65.32.248)  5.138 ms  5.107 ms  5.123 ms

From non working location:

$ sudo traceroute -I --mtu acme-v02.api.letsencrypt.org
traceroute to acme-v02.api.letsencrypt.org (172.65.32.248), 30 hops max, 65000 byte packets
 1  ... (...)  0.628 ms F=1500  1.077 ms  0.671 ms
 2  ... (...)  7.558 ms  7.573 ms  8.401 ms
 3  cloudflare.thess.gr-ix.gr (185.1.123.10)  15.576 ms  10.323 ms  13.406 ms
 4  172.65.32.248 (172.65.32.248)  8.345 ms  8.320 ms  8.358 ms

So the UDP trace is a false negative indicator - ICMP gets through just fine with MTU 1500.

Could you paste the full output of curl -vvv 'https://acme-v02.api.letsencrypt.org/directory' when it fails, please? There may be too many connections from the mentioned IP range 78.108.34.0/24 and Cloudflare may rate-limit the (not necessarily letsencrypt related) inbound connenctions. You want to check the general health of that network.

2 Likes

@bruncsak this is what I’ve just concluded myself as well. I’ve managed to reproduce using this monster oneliner from the problematic network:

curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/ ; curl -I https://acme-v02.api.letsencrypt.org/

The failing one (with -vvv) is:

*   Trying 172.65.32.248...
* Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
* found 148 certificates in /etc/ssl/certs/ca-certificates.crt
* found 596 certificates in /etc/ssl/certs
* ALPN, offering http/1.1


* gnutls_handshake() failed: Error in the pull function.
* Closing connection 0
curl: (35) gnutls_handshake() failed: Error in the pull function.