"curl returned with 35" + anti-replay nonce (seems IPv6-related)


#1

Hi,

I’m using LE with dehydrated (previously letsencrypt.sh). I’m also using ‘dns-01’ as the type, together with a hook for PowerDNS. This worked just fine, but today I upgraded ‘letsencrypt.sh’ (to ‘dehydrated’). I’m now receiving two errors “randomly” (I’m not sure if the errors are related to upgrading to dehydrated, or if it’s something else – I haven’t really paid attention to LE for a while, as it’s just worked automagically via a cronjob).

1) Domains with “a lot” (~20) of subdomains fails with the following;

ERROR: Problem connecting to server (post for https://acme-v01.api.letsencrypt.org/acme/new-authz; curl returned with 35)

It seems to fail consistently, but it’s random how many subdomains that is processed before the failure occurs (but usually after 10-12 subdomains within the same domain). According to my knowledge, error code 35 has something to do with the SSL/TLS handshake, but there are no more output as to what’s happening.

This happens in both production and staging.

2) Randomly it will fail with the following error;

ERROR: Problem connecting to server (head for https://acme-v01.api.letsencrypt.org/directory; curl returned with 35)
+ ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)

Details:
{
  "type": "urn:acme:error:badNonce",
  "detail": "JWS has no anti-replay nonce",
  "status": 400
}

Might be related to 1) (as it fails with the curl error).

Any takers?


#2

So, it seems to be related to IPv6. Reaching LE over IPv6 is really slow.

IPv4:

letsencrypt@web1:~$ date;time curl -s4 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:48 UTC 2016

real	0m0.404s
user	0m0.016s
sys	0m0.000s

IPv6:

letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:16 UTC 2016

real	0m4.161s
user	0m0.016s
sys	0m0.000s

letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:24 UTC 2016

real	0m3.028s
user	0m0.016s
sys	0m0.004s

letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:29 UTC 2016

real	0m5.891s
user	0m0.016s
sys	0m0.000s

letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:58 UTC 2016

real	0m32.604s
user	0m0.016s
sys	0m0.000s

#3

Doesn’t seem to be latency-related (per se). Trying to find out if this is on our side, or somewhere else.


#4

letsencrypt.org’ has a RTT of ~285ms from my box, and the staging-url only has ~40ms. The first works just fine (consistent curl times), while the latter doesn’t (even if it’s got lower latency).


#5

So, using ‘–ipv4’ on dehydrated works like a charm. Consistently.

We don’t have any issues with IPv6 to other locations, and it doesn’t seem to be issues with latency per se either, so not sure where the error might be?


#6

I’ll also mention @jsha and @cpu here to ask them to look at this possible server-side issue (different behavior of IPv4 and IPv6 endpoints?).


#7

Thanks. Following are traceroute outputs from mtr, over IPv4 and IPv6 (for comparison).

mtr over IPv4;

jocke@kek:~$ mtr -n4 --report acme-staging.api.letsencrypt.org
Start: Thu Oct 13 19:02:09 2016
HOST: kek                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- <snip>                     0.0%    10    1.1   2.0   0.8   4.4   1.1
  2.|-- <snip>                     0.0%    10    0.4   0.5   0.4   0.9   0.0
  3.|-- 82.134.66.197              0.0%    10    1.2   1.3   1.1   1.8   0.0
  4.|-- 193.28.236.254             0.0%    10    7.6   7.5   7.3   7.6   0.0
  5.|-- 193.28.236.253             0.0%    10    7.6   7.7   7.5   8.7   0.3
  6.|-- 62.140.27.29               0.0%    10    7.7   7.6   7.5   7.8   0.0
  7.|-- 4.69.206.110              90.0%    10   14.8  14.8  14.8  14.8   0.0
  8.|-- 4.69.206.110              90.0%    10   14.7  14.7  14.7  14.7   0.0
  9.|-- 4.68.63.186                0.0%    10   14.5  14.5  14.4  14.7   0.0
 10.|-- 195.22.214.167             0.0%    10   37.0  36.9  36.8  37.4   0.0
 11.|-- 195.22.214.237             0.0%    10   36.8  36.9  36.7  37.2   0.0
 12.|-- 195.13.60.178              0.0%    10   46.0  46.2  45.8  46.4   0.0
 13.|-- 193.34.48.162             10.0%    10   51.2  51.4  50.0  53.4   0.7
 14.|-- 193.34.48.146              0.0%    10   51.5  52.0  51.0  55.8   1.3
 15.|-- 104.87.208.66             10.0%    10   50.8  51.3  50.8  52.3   0.0

mtr over IPv6;

jocke@kek:~$ mtr -n6 --report acme-staging.api.letsencrypt.org
Start: Thu Oct 13 19:02:09 2016
HOST: kek                         Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- <snip>                     0.0%    10    1.5   3.4   1.0  15.5   4.2
  2.|-- <snip>                     0.0%    10    0.5   0.5   0.4   0.7   0.0
  3.|-- 2a00:14d8:1001:734::1      0.0%    10    1.3   1.4   1.3   1.5   0.0
  4.|-- 2001:67c:324:2::2          0.0%    10    7.8   7.7   7.5   7.9   0.0
  5.|-- 2001:67c:324:2::1          0.0%    10    8.1   8.1   7.6   9.1   0.3
  6.|-- 2001:1900:5:2:2::3ea5      0.0%    10    7.7   7.7   7.5   7.9   0.0
  7.|-- 2001:1900:2::3:5f          0.0%    10   29.0  28.9  28.8  29.1   0.0
  8.|-- 2001:1900:4:3::266         0.0%    10   29.0  29.6  29.0  34.3   1.6
  9.|-- 2a01:3e0:ff40:200::21      0.0%    10   36.4  36.6  36.4  36.7   0.0
 10.|-- 2a01:3e0:ff40:200::16     30.0%    10   35.2  34.9  34.7  35.2   0.0
 11.|-- 2a02:b0:ffff:ffff:ffff:ff 10.0%    10   64.8  45.5  42.8  64.8   7.2
 12.|-- 2a02:b0:ffff:ffff:ffff:ff 20.0%    10   42.8  42.8  42.6  43.2   0.0
 13.|-- 2a02:26f0:d5:295::3d5     20.0%    10   42.8  42.9  42.7  43.3   0.0

#8

Thanks for the detailed report, @jockek! All our sites, including the API, are fronted by Akamai, so the connectivity problems you’re seeing are problems reaching Akamai. I will share this thread with our contact there and see what we can figure out!


#9

@jockek: Could you run these diagnostics and paste the results? Thanks!

    curl http://ipv4.whatismyip.akamai.com/ ; echo
    curl http://ipv6.whatismyip.akamai.com/ ; echo
    dig +short whoami.ipv4.akahelp.net TXT
    dig +short whoami.ipv6.akahelp.net TXT
    dig +short whoami.ds.akahelp.net TXT
    dig +short whoami.ds.akahelp.net TXT
    dig +short whoami.ds.akahelp.net TXT

#10

No problem. Let me know if there is anything I can do on my part. I’ve kinda ruled out that there’s an issue on our side for now, as most of the requests are okay (even though requests over IPv6 fluctuate a bit more than over IPv4, most of them are still sub 5 second).

I just did about 600 measurements over IPv6 (while-loop with ‘sleep 1’), and ~60 of them had 5+ seconds request time. Out of the ~60, most of them was well below 30 seconds, and all of them below 60 seconds.

This also explains why it mostly happens on domains with many subdomains; the others have too few domains in them, so that they are not likely to hit one of the high-latency requests.


#11

@jsha, here you go. Was it intentional to have 3x the same dig-query?

letsencrypt@web1:~$ curl http://ipv4.whatismyip.akamai.com/ ; echo
185.54.30.6
letsencrypt@web1:~$     curl http://ipv6.whatismyip.akamai.com/ ; echo
2a02:4260:30::6
letsencrypt@web1:~$     dig +short whoami.ipv4.akahelp.net TXT
"ns" "185.54.30.2"
letsencrypt@web1:~$     dig +short whoami.ipv6.akahelp.net TXT
"ns" "2a02:4260:30::2"
letsencrypt@web1:~$     dig +short whoami.ds.akahelp.net TXT
"ns" "185.54.30.2"
letsencrypt@web1:~$     dig +short whoami.ds.akahelp.net TXT
"ns" "185.54.30.2"
letsencrypt@web1:~$     dig +short whoami.ds.akahelp.net TXT
"ns" "185.54.30.2"

#12

Here’s an example of it stalling mid-SSL-handshake (for ~15 seconds);

letsencrypt@web1:~$ time curl -vvvv https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
* Hostname was NOT found in DNS cache
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0*   Trying 2a02:26f0:d5:295::3d5...
* Connected to acme-staging.api.letsencrypt.org (2a02:26f0:d5:295::3d5) port 443 (#0)
* successfully set certificate verify locations:
*   CAfile: none
  CApath: /etc/ssl/certs
* SSLv3, TLS handshake, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Server hello (2):
{ [data not shown]
  0     0    0     0    0     0      0      0 --:--:--  0:00:15 --:--:--     0* SSLv3, TLS handshake, CERT (11):
{ [data not shown]
* SSLv3, TLS handshake, Server key exchange (12):
{ [data not shown]
* SSLv3, TLS handshake, Server finished (14):
{ [data not shown]
* SSLv3, TLS handshake, Client key exchange (16):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
} [data not shown]
* SSLv3, TLS handshake, Finished (20):
} [data not shown]
* SSLv3, TLS change cipher, Client hello (1):
{ [data not shown]
* SSLv3, TLS handshake, Finished (20):
{ [data not shown]
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* Server certificate:
* 	 subject: CN=*.api.letsencrypt.org; O=INTERNET SECURITY RESEARCH GROUP; L=Mountain View; ST=California; C=US
* 	 start date: 2015-06-26 17:05:45 GMT
* 	 expire date: 2018-06-25 17:05:45 GMT
* 	 subjectAltName: acme-staging.api.letsencrypt.org matched
* 	 issuer: C=US; O=IdenTrust; OU=TrustID Server; CN=TrustID Server CA A52
* 	 SSL certificate verify ok.
> GET /acme/new-authz HTTP/1.1
> User-Agent: curl/7.38.0
> Host: acme-staging.api.letsencrypt.org
> Accept: */*
> 
< HTTP/1.1 405 Method Not Allowed
* Server nginx is not blacklisted
< Server: nginx
< Content-Type: application/problem+json
< Content-Length: 91
< Allow: POST
< Boulder-Request-Id: oiBj67rx1uYslGyXq9H76xbUSxMOBmfgmFWxanHoe8I
< Replay-Nonce: AlZbX3g3BklNuSa1DryWOjiYxs_QfEj1yB9gkElwhfA
< Expires: Thu, 13 Oct 2016 23:32:13 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Thu, 13 Oct 2016 23:32:13 GMT
< Connection: keep-alive
< 
{ [data not shown]
100    91  100    91    0     0      5      0  0:00:18  0:00:15  0:00:03    19
* Connection #0 to host acme-staging.api.letsencrypt.org left intact

real	0m15.948s
user	0m0.012s
sys	0m0.008s

#13

Diagnostics output on one our servers:

$ curl http://ipv4.whatismyip.akamai.com/ ; echo
8.36.86.67
$ curl http://ipv6.whatismyip.akamai.com/ ; echo
curl: (7) Failed to connect to 2600:1408:10::1703:de8: Network is unreachable
$ dig +short whoami.ipv4.akahelp.net TXT
"ns" "8.0.14.6"
$ dig +short whoami.ipv6.akahelp.net TXT
"ecs" "8.36.86.0/24/0"
"ns" "2607:f8b0:4003:c07::109"
"ip" "8.36.86.225"
$ dig +short whoami.ds.akahelp.net TXT
"ns" "8.0.14.14"
$ dig +short whoami.ds.akahelp.net TXT
"ns" "8.0.14.14"
$ dig +short whoami.ds.akahelp.net TXT
"ns" "8.0.14.14"

Output on a second server:

$ curl http://ipv4.whatismyip.akamai.com/ ; echo
8.36.86.65
$ curl http://ipv6.whatismyip.akamai.com/ ; echo
curl: (7) Failed to connect to 2600:1408:10::1703:de8: Network is unreachable
$ dig +short whoami.ipv4.akahelp.net TXT
"ip" "8.36.86.243"
"ns" "173.194.99.161"
"ecs" "8.36.86.0/24/0"
$ dig +short whoami.ipv6.akahelp.net TXT
;; connection timed out; no servers could be reached
$ dig +short whoami.ds.akahelp.net TXT
"ns" "192.221.143.2"
$ dig +short whoami.ds.akahelp.net TXT
"ns" "8.0.15.5"
$ dig +short whoami.ds.akahelp.net TXT
"ns" "8.0.15.5"

#14

@jsha, the issue seems to be “solved” now (i.e. there are no slow queries at the moment). I just did ~600 measurements again (using the same approach as last time), and none of them was above 5 seconds. There has been no changes on our side, so I’m not sure what has caused this.

I’ve done the diagnostic-commands again, together with the traceroutes (same source and dst on both IPv4 and IPv6).

letsencrypt@web1:~$ curl http://ipv4.whatismyip.akamai.com/ ; echo
185.54.30.6
letsencrypt@web1:~$     curl http://ipv6.whatismyip.akamai.com/ ; echo
2a02:4260:30::6
letsencrypt@web1:~$     dig +short whoami.ipv4.akahelp.net TXT
"ns" "185.54.30.2"
letsencrypt@web1:~$     dig +short whoami.ipv6.akahelp.net TXT
"ns" "2a02:4260:30::2"
letsencrypt@web1:~$     dig +short whoami.ds.akahelp.net TXT
"ns" "2a02:4260:30::2"
letsencrypt@web1:~$     dig +short whoami.ds.akahelp.net TXT
"ns" "2a02:4260:30::2"
letsencrypt@web1:~$     dig +short whoami.ds.akahelp.net TXT
"ns" "2a02:4260:30::2"
letsencrypt@web1:~$ mtr -n4 --report acme-staging.api.letsencrypt.org
Start: Tue Oct 18 18:41:57 2016
HOST: web1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 185.54.30.1                0.0%    10    2.5   5.6   1.3  29.0   8.5
  2.|-- 185.54.30.32               0.0%    10    0.5   0.5   0.4   0.6   0.0
  3.|-- 82.134.66.197              0.0%    10    1.2   1.4   1.1   3.1   0.6
  4.|-- 193.28.236.254             0.0%    10    7.4   7.5   7.4   7.5   0.0
  5.|-- 193.28.236.253             0.0%    10    7.5   7.7   7.4   8.9   0.3
  6.|-- 62.140.27.29               0.0%    10    7.4   8.0   7.4  12.9   1.5
  7.|-- 4.69.203.98               90.0%    10   28.8  28.8  28.8  28.8   0.0
  8.|-- 4.68.63.42                 0.0%    10   28.8  28.9  28.8  29.2   0.0
  9.|-- 80.231.152.25              0.0%    10   36.5  36.5  36.4  36.6   0.0
 10.|-- 80.231.152.142             0.0%    10   34.7  34.9  34.7  35.1   0.0
 11.|-- 193.34.48.162             10.0%    10   42.9  43.0  42.7  45.5   0.8
 12.|-- 193.34.48.146              0.0%    10   42.8  42.7  42.6  43.1   0.0
 13.|-- 104.87.208.66             20.0%    10   42.7  42.8  42.6  43.1   0.0
letsencrypt@web1:~$ mtr -n6 --report acme-staging.api.letsencrypt.org
Start: Tue Oct 18 18:42:11 2016
HOST: web1                        Loss%   Snt   Last   Avg  Best  Wrst StDev
  1.|-- 2a02:4260:30::1            0.0%    10    2.1   6.1   1.2  42.0  12.6
  2.|-- 2a02:4260:1001::           0.0%    10    0.5   0.5   0.4   0.5   0.0
  3.|-- 2a00:14d8:1001:734::1      0.0%    10    1.2   1.4   1.2   2.4   0.0
  4.|-- 2001:67c:324:2::2          0.0%    10    7.6   7.5   7.4   7.6   0.0
  5.|-- 2001:67c:324:2::1          0.0%    10    7.7   8.0   7.6   8.9   0.3
  6.|-- 2001:1900:5:2:2::3ea5      0.0%    10    7.8   7.6   7.4   7.8   0.0
  7.|-- 2001:1900:2::3:5f          0.0%    10   28.7  28.8  28.7  29.0   0.0
  8.|-- 2001:1900:4:3::266         0.0%    10   28.8  28.8  28.7  29.2   0.0
  9.|-- 2a01:3e0:ff40:200::21      0.0%    10   36.5  36.5  36.4  36.6   0.0
 10.|-- 2a01:3e0:ff40:200::16     10.0%    10   34.9  35.0  34.8  35.8   0.0
 11.|-- 2a02:b0:ffff:ffff:ffff:ff 10.0%    10   42.7  43.8  42.7  47.6   1.8
 12.|-- 2a02:b0:ffff:ffff:ffff:ff 20.0%    10   42.9  42.9  42.6  43.5   0.0
 13.|-- 2a02:26f0:d5:297::3d5     30.0%    10   42.6  42.8  42.5  43.1   0.0

#15

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.