I’m using LE with dehydrated (previously letsencrypt.sh). I’m also using ‘dns-01’ as the type, together with a hook for PowerDNS. This worked just fine, but today I upgraded ‘letsencrypt.sh’ (to ‘dehydrated’). I’m now receiving two errors “randomly” (I’m not sure if the errors are related to upgrading to dehydrated, or if it’s something else – I haven’t really paid attention to LE for a while, as it’s just worked automagically via a cronjob).
1) Domains with “a lot” (~20) of subdomains fails with the following;
ERROR: Problem connecting to server (post for https://acme-v01.api.letsencrypt.org/acme/new-authz; curl returned with 35)
It seems to fail consistently, but it’s random how many subdomains that is processed before the failure occurs (but usually after 10-12 subdomains within the same domain). According to my knowledge, error code 35 has something to do with the SSL/TLS handshake, but there are no more output as to what’s happening.
This happens in both production and staging.
2) Randomly it will fail with the following error;
ERROR: Problem connecting to server (head for https://acme-v01.api.letsencrypt.org/directory; curl returned with 35)
+ ERROR: An error occurred while sending post-request to https://acme-v01.api.letsencrypt.org/acme/new-authz (Status 400)
Details:
{
"type": "urn:acme:error:badNonce",
"detail": "JWS has no anti-replay nonce",
"status": 400
}
Might be related to 1) (as it fails with the curl error).
So, it seems to be related to IPv6. Reaching LE over IPv6 is really slow.
IPv4:
letsencrypt@web1:~$ date;time curl -s4 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:48 UTC 2016
real 0m0.404s
user 0m0.016s
sys 0m0.000s
IPv6:
letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:16 UTC 2016
real 0m4.161s
user 0m0.016s
sys 0m0.000s
letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:24 UTC 2016
real 0m3.028s
user 0m0.016s
sys 0m0.004s
letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:29 UTC 2016
real 0m5.891s
user 0m0.016s
sys 0m0.000s
letsencrypt@web1:~$ date;time curl -s6 https://acme-staging.api.letsencrypt.org/acme/new-authz >/dev/null
Wed Oct 12 19:57:58 UTC 2016
real 0m32.604s
user 0m0.016s
sys 0m0.000s
‘letsencrypt.org’ has a RTT of ~285ms from my box, and the staging-url only has ~40ms. The first works just fine (consistent curl times), while the latter doesn’t (even if it’s got lower latency).
So, using ‘–ipv4’ on dehydrated works like a charm. Consistently.
We don’t have any issues with IPv6 to other locations, and it doesn’t seem to be issues with latency per se either, so not sure where the error might be?
Thanks for the detailed report, @jockek! All our sites, including the API, are fronted by Akamai, so the connectivity problems you’re seeing are problems reaching Akamai. I will share this thread with our contact there and see what we can figure out!
No problem. Let me know if there is anything I can do on my part. I’ve kinda ruled out that there’s an issue on our side for now, as most of the requests are okay (even though requests over IPv6 fluctuate a bit more than over IPv4, most of them are still sub 5 second).
I just did about 600 measurements over IPv6 (while-loop with ‘sleep 1’), and ~60 of them had 5+ seconds request time. Out of the ~60, most of them was well below 30 seconds, and all of them below 60 seconds.
This also explains why it mostly happens on domains with many subdomains; the others have too few domains in them, so that they are not likely to hit one of the high-latency requests.
@jsha, the issue seems to be “solved” now (i.e. there are no slow queries at the moment). I just did ~600 measurements again (using the same approach as last time), and none of them was above 5 seconds. There has been no changes on our side, so I’m not sure what has caused this.
I’ve done the diagnostic-commands again, together with the traceroutes (same source and dst on both IPv4 and IPv6).