cURL error to /directory endpoint

Hello!

we are experiencing random connection problems to acme-v02.api.letsencrypt.org/directory endpoint on CentoOS 6/7 machines since 24th September (00:00).

Hosts setup are as follows:

CentOS 7.6.1810
dehydrated 0.6.5
curl-7.29.0-54.el7.x86_64 (stock) and curl 7.65.1

Curl trace shows different errors for IPv4 and IPV6 type of connections.
IPv4 - NSS error -5961 (PR_CONNECT_RESET_ERROR)
IPv6 - NSS error -5938 (PR_END_OF_FILE_ERROR)
cURL trace link

I have already tried force cURL resolve IPv4 and IPv6 address only, explicitly set --tlsv1.2 and also newer version than stock. But no luck so far. Error shows pure randomly - several times per day and host. But it is kinda weird that when testing connection in loop outside dehydrated, no problem encountered (so far).

Is it possible that we are hitting some rate limit? I have read that there is 40/r per second for /directory endpoint. Is is per IP? If so, I would assume some message in response, not killing connection.

I appreciate any help. Thank you.

1 Like

FWIW, someone else has just posted a similar thread:

This indeed looks likes a similar problem to what @mnordhoff linked. I’m investigating the problem on that thread and will provide updates there. Thanks for including your cURL trace link. If you have any more error details or logs that seem relevant please include them here.

I agree that it’s not likely a rate limit issue. We return 429s and a message in that case.

1 Like

Can you please provide some timestamps for these errors?

Are they only when accessing acme-v02 or does it also occur on acme-v01?

1 Like

Of course, here are some failed attempts for two hosts in last two days.

Host 178.238.37.215/2a01:430:13::215

2019-10-01 17:00:04 sid-215 LE[205834]: ERROR: Problem connecting to server (get for https://acme-v02.api.letsencrypt.org/directory; curl returned with 35)
2019-10-02 03:00:37 sid-215 LE[192614]: ERROR: Problem connecting to server (get for https://acme-v02.api.letsencrypt.org/directory; curl returned with 35)

Host 178.238.37.212/2a01:430:13::212

2019-10-01 18:20:16 sid-212 LE[13131]: ERROR: Problem connecting to server (get for https://acme-v02.api.letsencrypt.org/directory; curl returned with 35)
2019-10-02 07:10:19 sid-212 LE[22708]: ERROR: Problem connecting to server (get for https://acme-v02.api.letsencrypt.org/directory; curl returned with 35)

Currently we are using acme-v02 api only, as it is default for dehydrated client. I will try switch to previous version on one host and see what happens :slight_smile:

I noticed that errors occur when LE script is checking possible renewals for domains on given host (dehydrated runs get request to /directory endpoint for every domain). When I run cURL in loop outside script, no error encountered. Maybe CDN does not like multiple concurrent requests from same network. :thinking:

1 Like

Are those timestamps in UTC?

2 Likes

@_az Yay, sorry. They are not. It’s CEST -> UTC+2.

3 Likes

@jillian Same errors on acme-v01 endpoint too.

2019-10-02 12:10:22 sid-215 LE[92384]: ERROR: Problem connecting to server (get for https://acme-v01.api.letsencrypt.org/directory; curl returned with 35)

* Trying 2606:4700:60:0:f53d:5624:85c7:3a2c:443...
* TCP_NODELAY set
* Connected to acme-v01.api.letsencrypt.org (2606:4700:60:0:f53d:5624:85c7:3a2c) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
* CAfile: none
CApath: none
* loaded libnssckbi.so
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
* TCP connection reset by peer
* Closing connection 0
2 Likes

Just a quick update. I’ve been reviewing logs and data and I’m getting close to a root cause. Thanks for your patience while we continue investigating and for including the additional requested data.

3 Likes

We implemented some fixes that we expect to resolve a majority of these errors.

There is a full update on this related post:

2 Likes