Problem connecting to letsencrypt only from one server

[shadmin]

Hi,

We have found the following issue with one of our servers. When trying to issue certificate, server return following errors:

"There was a problem processing your request

Error issuing certificate

Failed to issue certificate

Error connecting to service: GET failed: Get https://letsencrypt.org/documents/LE-SA-v1.0.1-July-27-2015.pdf: dial tcp 23.7.193.81:443: i/o timeout"

We have ping and traceroute from server to IP address 23.7.193.81.

ping 23.7.193.81

PING 23.7.193.81 (23.7.193.81) 56(84) bytes of data.

64 bytes from 23.7.193.81: icmp_seq=1 ttl=54 time=40.7 ms

64 bytes from 23.7.193.81: icmp_seq=2 ttl=54 time=40.8 ms

64 bytes from 23.7.193.81: icmp_seq=3 ttl=54 time=40.7 ms

64 bytes from 23.7.193.81: icmp_seq=4 ttl=54 time=40.7 ms

64 bytes from 23.7.193.81: icmp_seq=5 ttl=54 time=40.7 ms

64 bytes from 23.7.193.81: icmp_seq=6 ttl=54 time=40.8 ms

ping letsencrypt.org

PING letsencrypt.org (23.7.193.81) 56(84) bytes of data.

64 bytes from a23-7-193-81.deploy.static.akamaitechnologies.com (23.7.193.81): icmp_seq=1 ttl=54 time=40.8 ms

64 bytes from a23-7-193-81.deploy.static.akamaitechnologies.com (23.7.193.81): icmp_seq=2 ttl=54 time=40.7 ms

64 bytes from a23-7-193-81.deploy.static.akamaitechnologies.com (23.7.193.81): icmp_seq=3 ttl=54 time=40.8 ms

64 bytes from a23-7-193-81.deploy.static.akamaitechnologies.com (23.7.193.81): icmp_seq=4 ttl=54 time=41.8 ms

We tried to make telnet check ( from the same IP ) on port 80 and 443 to 23.7.193.81 and this is the results:

telnet 23.7.193.81 443
Trying 23.7.193.81…
telnet: connect to address 23.7.193.81: Connection timed out

telnet 23.7.193.81 80
Trying 23.7.193.81…
telnet: connect to address 23.7.193.81: Connection timed out

We think you have blocked one of our IP because from next IP in the network, the telnet check ( on port 80 and 443 ) is connecting correctly.

Please provide us with information where ( email address or contact form ) should we send the IP address of the server with no connection to letsencrypt so you can check the restriction and remove it ?

Regars

[eva2000]

same here i believe i ran into same issue with @Neilpang acme.sh at https://github.com/Neilpang/acme.sh/issues/203#issuecomment-222682466

curl -4 --dump-header h.txt https://acme-staging.api.letsencrypt.org/directory; cat h.txt 
curl: (7) Failed connect to acme-staging.api.letsencrypt.org:443; No route to host

curl --dump-header h.txt https://acme-staging.api.letsencrypt.org/directory; cat h.txt    
curl: (7) Failed to connect to 2600:141b:5:292::3d5: Network is unreachable

ping -c4 acme-staging.api.letsencrypt.org
PING e981.dscb.akamaiedge.net (23.78.168.127) 56(84) bytes of data.
From domain.com (ip) icmp_seq=1 Destination Host Unreachable
From domain.com (ip) icmp_seq=2 Destination Host Unreachable
From domain.com (ip) icmp_seq=3 Destination Host Unreachable
From domain.com (ip) icmp_seq=4 Destination Host Unreachable

--- e981.dscb.akamaiedge.net ping statistics ---
4 packets transmitted, 0 received, +4 errors, 100% packet loss, time 2999ms
pipe 4

ping -c4 google.com
PING google.com (74.125.21.102) 56(84) bytes of data.
64 bytes from yv-in-f102.1e100.net (74.125.21.102): icmp_seq=1 ttl=48 time=9.86 ms
64 bytes from yv-in-f102.1e100.net (74.125.21.102): icmp_seq=2 ttl=48 time=10.3 ms
64 bytes from yv-in-f102.1e100.net (74.125.21.102): icmp_seq=3 ttl=48 time=10.0 ms
64 bytes from yv-in-f102.1e100.net (74.125.21.102): icmp_seq=4 ttl=48 time=10.0 ms

--- google.com ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3005ms
rtt min/avg/max/mdev = 9.866/10.067/10.351/0.202 ms

[serverco]

That looks like 2 different errors / issues to me.

In the first shadmin can reach / ping the server but is getting an error connecting on port 80.

In the second, eva2000, it looks like a routing issue as you can not ping the server

[eva2000]

it might be a specific server at acme-staging.api.letsencrypt.org as i run it a few times and successful connects are with a different destination ip

seems ip 23.78.168.127 for acme-staging.api.letsencrypt.org is the culprit with both ipv4 and ipv6 issues while acme-staging.api.letsencrypt.org on 104.101.161.90 connects fine over ipv4

curl forced ipv4

curl -v -4 --dump-header h.txt https://acme-staging.api.letsencrypt.org/directory
* About to connect() to acme-staging.api.letsencrypt.org port 443 (#0)
*   Trying 23.78.168.127...
* No route to host
* Failed connect to acme-staging.api.letsencrypt.org:443; No route to host
* Closing connection 0
curl: (7) Failed connect to acme-staging.api.letsencrypt.org:443; No route to host

curl -v -4 --dump-header h.txt https://acme-staging.api.letsencrypt.org/directory
* About to connect() to acme-staging.api.letsencrypt.org port 443 (#0)
*   Trying 104.101.161.90...
* Connected to acme-staging.api.letsencrypt.org (104.101.161.90) port 443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: /etc/pki/tls/certs/ca-bundle.crt
  CApath: none
* SSL connection using TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384
* Server certificate:
*       subject: C=US,ST=California,L=Mountain View,O=INTERNET SECURITY RESEARCH GROUP,CN=*.api.letsencrypt.org
*       start date: Jun 26 17:05:45 2015 GMT
*       expire date: Jun 25 17:05:45 2018 GMT
*       common name: *.api.letsencrypt.org
*       issuer: CN=TrustID Server CA A52,OU=TrustID Server,O=IdenTrust,C=US
> GET /directory HTTP/1.1
> User-Agent: curl/7.29.0
> Host: acme-staging.api.letsencrypt.org
> Accept: */*
> 
< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: application/json
< Content-Length: 296
< Replay-Nonce: g9wheYZCRChenxhIyGGutmZ3ffOqHZFjzDOac7Z4qjc
< X-Frame-Options: DENY
< Strict-Transport-Security: max-age=604800
< Expires: Tue, 31 May 2016 13:45:19 GMT
< Cache-Control: max-age=0, no-cache, no-store
< Pragma: no-cache
< Date: Tue, 31 May 2016 13:45:19 GMT
< Connection: keep-alive
< 
{
  "new-authz": "https://acme-staging.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-staging.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-staging.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-staging.api.letsencrypt.org/acme/revoke-cert"
* Connection #0 to host acme-staging.api.letsencrypt.org left intact
}

[serverco]

Yes, it could be. I don’t know how many different “staging servers” / IP’s there are. Checking what the IP is from half a dozen different servers around the world - I get a different IP from each…

[eva2000]

maybe geodns/geo latency based ?

[serverco]

Could be. Have you tried just defining it to an IP that you know works in your hosts file ? a bit messy, but would confirm that it’s related to a specific LE server and not your end. Alternatively as you’re using a script - can you just define the IP instead of using acme-staging.api.letsencrypt.org as a test.

[eva2000]

ah yes should try that

seems a few ips at letsencrypt aren’t working

 curl -v -4 --dump-header h.txt https://acme-staging.api.letsencrypt.org/directory
* About to connect() to acme-staging.api.letsencrypt.org port 443 (#0)
*   Trying 23.198.115.87...
* No route to host
* Failed connect to acme-staging.api.letsencrypt.org:443; No route to host
* Closing connection 0
curl: (7) Failed connect to acme-staging.api.letsencrypt.org:443; No route to host

curl -v -4 --dump-header h.txt https://acme-staging.api.letsencrypt.org/directory
* About to connect() to acme-staging.api.letsencrypt.org port 443 (#0)
*   Trying 23.78.168.127...
* No route to host
* Failed connect to acme-staging.api.letsencrypt.org:443; No route to host
* Closing connection 0
curl: (7) Failed connect to acme-staging.api.letsencrypt.org:443; No route to host

[eva2000]

webhost fixed some ipv6 issues on my server end, so seems to be combination of issues, maybe some letsencrypt staging servers are ipv6 only so don’t work with ipv4 ?

works now https://github.com/Neilpang/acme.sh/issues/203#issuecomment-222703285

[shadmin]

Hi,

The problem is only when our server tries to connect to 23.7.193.81 . As we can see now the letsencrypt.org is no logner pointed to that IP and everything works now.