No response from Let's Encrypt API on particular IP

[edgaras]

Hi there

We are unable to get any result while issuing a cert via cPanel for client, error shows that “TLS handshake timed out”. Upon debugging we have found out that this happens only when we try to reach https://acme-v01.api.letsencrypt.org/directory via 2.22.101.48, other IPs give us a result as expected. We are also able to ping 2.22.101.48 without any packet loss.

What could be the problem here?

[pfg]

When you say the error occurs only with 2.22.101.48, are you referring to other IP addresses that acme-v01.api.letsencrypt.org resolves to, or some other site that you’ve tested connectivity with?

Let’s Encrypt is hosted behind Akamai’s CDN, so these issues are sometimes hard to debug because they are often specific to certain routes/ISPs. In the past, Let’s Encrypt staff have asked users to run a number of diagnostic commands when something like this occurs, though in similar cases the IP was typically not reachable at all. This will be useful for the Ops team and Akamai to further debug this.

Could you also try running the following command from that IP? I’m curious how far the connection gets, i.e. whether it’s a general block on incoming traffic on port 443 or if it happens somewhere in between. You can replace the hostname with 2.22.101.48 in case the first IP it resolves to is currently different - only the -servername needs to remain as is.

openssl s_client -connect acme-v01.api.letsencrypt.org:443 -servername acme-v01.api.letsencrypt.org

[edgaras]

Yes, we are able to get response from other IPs that this hostname resolves to. And additionally, 2.22.101.48 does not respond only on one of our datacenters, which is located in UK, US servers get result from 2.22.101.48 without any issues.

Ok, here are the results:

[root@xxx ~]# openssl s_client -connect 2.22.101.48:443 -servername acme-v01.api.letsencrypt.org
CONNECTED(00000003)

However, we are able to get a proper response running this command on US DC servers.

[Osiris]

MTU issue? http://packetlife.net/blog/2008/aug/18/path-mtu-discovery/

Try such a tracepath as described in the above blog.

[edgaras]

[root@xxx ~]# tracepath -n 2.22.101.48
1?: [LOCALHOST]     pmtu 1500
1:  <local_ip>     0.500ms
1:  <local_ip>     0.337ms
2:  no reply
3:  46.33.86.165     24.739ms asymm 12
4:  89.149.135.154   26.479ms asymm 11
5:  77.67.123.222    39.502ms asymm 11
6:  87.245.233.238   57.027ms asymm 13
7:  87.245.245.29    40.439ms asymm 15
8:  2.22.101.48      41.783ms reached
 Resume: pmtu 1500 hops 8 back 12

This is all we get when accessing via CURL:

[root@xxx ~]# curl -v https://acme-v01.api.letsencrypt.org/directory
* About to connect() to acme-v01.api.letsencrypt.org port 443 (#0)
*   Trying 2.22.101.48... connected
* Connected to acme-v01.api.letsencrypt.org (2.22.101.48) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none

[ahaw021]

hi @edgaras

try to do a dig (nslookup on windows)

I am getting very different IPs from you however I am in Australia so that could be the reason.

When troubleshooting these kinds of errors

if some of the datacetres work while one doesn’t i would usually look at what configuration is different

A) do you have a hostname enrty for acme-v01.api.letsencrypt.org (check /etc/hosts/)

B) Are you using the same DNS provider
C) I see you have core connectivity but I am a bit dubious about using clients such as CURL and OpenSSL as these sometimes have certs which are not installed in your OS (trusted root store)
D) Can you provide the output of openssl in verbose mode so we can see each of the steps rather than just connected message
E) Can you try opening up https://acme-v01.api.letsencrypt.org in a browser on the machine?
F) Can you provide a more detailed error from the error logs TLS handshake fails is a bit of a wide area to troubleshoot and handshakes may fail for a variety of reasons (such as cipher support, no intermediate certs, etc)

What you will probably find is there is a small difference somewhere in the OSI Stack (i am suspecting that it’s in the TLS) that is causing the issue

Andrei

[edgaras]

Yes, IPs are dependent on geo region, so it’s normal that you see entirely different IP. Here are more IPs:

https://www.whatsmydns.net/#A/acme-v01.api.letsencrypt.org

Answering your questions:

A) We did not had an entry in hosts file before the issue, however, since 2.22.101.48 IP started to ignore us, I’ve added another IP 23.201.74.133 to hosts file, which is working properly:

[root@xxx ~]# curl -v https://acme-v01.api.letsencrypt.org/directory
* About to connect() to acme-v01.api.letsencrypt.org port 443 (#0)
*   Trying 23.201.74.133... connected
* Connected to acme-v01.api.letsencrypt.org (23.201.74.133) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*       subject: C=US,ST=California,L=Mountain View,O=INTERNET SECURITY RESEARCH GROUP,CN=*.api.letsencrypt.org
*       start date: Jun 26 17:05:45 2015 GMT
*       expire date: Jun 25 17:05:45 2018 GMT
*       common name: *.api.letsencrypt.org
*       issuer: CN=TrustID Server CA A52,OU=TrustID Server,O=IdenTrust,C=US
> GET /directory HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.21 Basic ECC zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: acme-v01.api.letsencrypt.org
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 352
< Boulder-Request-Id: hKGtSqX_HLmEpPIBnYaaYb3b9Z6kYEXWeShAztqqSbY
< Replay-Nonce: CR33IA1vHiMatgU6mdOm_nlubM3NlSUwocyr362H5wk
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Mon, 20 Feb 2017 16:01:12 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Mon, 20 Feb 2017 16:01:12 GMT
< Connection: keep-alive
<
{
  "key-change": "https://acme-v01.api.letsencrypt.org/acme/key-change",
  "new-authz": "https://acme-v01.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-v01.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-v01.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-v01.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-v01.api.letsencrypt.org left intact
* Closing connection #0

As I have mentioned, 2.22.101.48 IP is responding on our US servers.

B) Yes, both servers are using Google Public DNS as resolvers.
C) Hmm, but it works as expected on US servers…
D) Just ran s_client with -msg option and did not get any ServerHello in the handshake.
E) Can’t do that, server is accessible only via terminal.
F) Sure:

FATA[0013] Failed to get certificate for map[domain.com:{domain.com domain.com Main /home/user/public_html user}]: Error connecting to service: Get https://acme-v01.api.letsencrypt.org/directory: net/http: TLS handshake timeout, aborting

[edgaras]

We are still facing this issue, can anybody check if our IPs are not blocked on CDN side?

[jillian]

We don’t have any blocked IPs at the CDN and have a ticket open with Akamai to look into the issue. We’ll update when we have more info.