OSCP servers are inaccesible from some servers

I noticed that some of my servers (actually, two) are unable to connect to LE servers, it looks like those ip are banned for some reason, and my hosting provider says the same. Is it possible? If yes, then a) what could be the cause? b) how to unban?

It looks like this (and the same for int-x1, x2 and x4):
pys@zabbix:~/ > curl -v -4 ocsp.int-x3.letsencrypt.org

OSCP-stapling is activated in nginx on theese servers, actually thanks to it I have noticed this strangeness. But the traffic on servers is very-very small, so I think it’s unlikely could be some rate limits on LE side.

Please help!

This looks like routing problems. I think the OCSP uses the Akamai-CDN but you don't reach that.
Could you traceroute to another endpoint:

traceroute 104.121.76.43

pys@zabbix:~/ > traceroute 104.121.76.43 11:29
traceroute to 104.121.76.43 (104.121.76.43), 30 hops max, 60 byte packets
1 gw.msk.hoztnode.net (188.120.250.1) 0.420 ms 0.379 ms 0.351 ms
2 edge.webdc.ru (92.63.108.97) 0.343 ms 0.324 ms 0.309 ms
3 ddos-guard.net (185.129.101.76) 1.003 ms 1.605 ms 0.984 ms
4 ae9-358.RT.M9.MSK.RU.retn.net (87.245.255.96) 1.592 ms 1.487 ms 1.651 ms
5 ae4-6.RT.SB.BER.DE.retn.net (87.245.234.126) 31.241 ms 31.193 ms 31.194 ms
6 GW-Akamai.retn.net (87.245.249.219) 31.525 ms 37.184 ms 37.165 ms
7 a104-121-76-43.deploy.static.akamaitechnologies.com (104.121.76.43) 29.109 ms 29.281 ms 29.155 ms

It works.

So the CDN is at least reachable via another route.
Maybe this ddos-guard.net is the culprit or transtelecom. If you are customer of ddos-guard, you may open a ticket there.

Don’t know if the LE-folks might have some insight here, too.

The hosting provider now says that these IP’s are being blocked by ISP on the basis of a court decision:
23.15.14.24 (because of this url: http://www.fun88.com/en-US/Home)
188.43.76.81 (unknown reason)
188.43.76.66 (unknown reason)

The oscp-servers are resolved to these IP’s (on my server). So, when I set up other IP (88.221.132.150) in my hosts file, the curl started to work. But nginx not, I think it doesn’t use hosts file.

The server and ISP are in the Russian jurisdiction. So it would be great if someone told LE, maybe they will stop using those IP’s.

Changes in /etc/hosts are relevant for most programs (that use the system's default resolver). Nevertheless, there might be some caching in the resolver library, try to restart nginx and check again.

Now it looks like this...

In nginx error log:
2017/10/26 14:27:25 [error] 21#21: recv() failed (110: Operation timed out) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
2017/10/26 14:27:25 [error] 21#21: OCSP responder prematurely closed connection while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org

But the server is accessible from nginx's container:
www@ico:~/ico-demo$ docker exec -ti nginx_icodemo sh -c 'curl -v -4 ocsp.int-x3.letsencrypt.org'

GET / HTTP/1.1
Host: ocsp.int-x3.letsencrypt.org
User-Agent: curl/7.56.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Content-Type: text/plain; charset=utf-8
< Content-Length: 0
< Cache-Control: max-age=16557
< Expires: Thu, 26 Oct 2017 19:03:46 GMT
< Date: Thu, 26 Oct 2017 14:27:49 GMT
< Connection: keep-alive
<

Looks like it does not use the system resolver. Do you have configured an external resolver for nginx?
http://nginx.org/en/docs/http/ngx_http_core_module.html#resolver

Nevertheless, I would push your provider to unblock these addresses. Could be that they have been misused in the past but now are part of the LE-CDN.

2017/10/26 14:57:37 [error] 9#9: OCSP_basic_verify() failed (SSL: error:27069076:OCSP routines:OCSP_basic_verify:signer certificate not found) while requesting certificate status, responder: ocsp.int-x3.letsencrypt.org
^C
www@ico:~/ico-demo$ docker exec -ti nginx_icodemo nginx -T | grep resolver
# resolver 8.8.8.8 8.8.4.4;

I’ve disabled this directive, but still.
I think I’ll have to build some hacky workaround for this (like own dummy-always-the-same-ip-resolver) or just disable the stapling…
Thank you for help!

@cpu or someone:

Can you escalate this to Akamai? According to post #5, Russian courts have ordered some Akamai IPs blocked for reasons unknown, (partly) affecting ocsp.

I don’t know if Akamai has a process for that. :confused:

If that’s the reason, Akamai might also not want to cooperate by splitting off other services from the particular service targeted by the order, because that would decrease the collateral effects of the order, and so reduce pressures to reverse it. (Wikipedia, for example, dealt with this situation when different governments tried to block all of Wikipedia because of specific content, which sometimes led to pressure to reverse the blocking because people were upset about the collateral effects.) But I don’t know what Akamai’s policy on this situation is and I agree that it would be good for Akamai to be aware of it if they’re not already aware.

2 Likes

Yeah, you may be right. A policy like that might have exceptions, though.

FWIW, Russia apparently has a unified database of blocked resources. The 24/8 IP is in it (since 2016!) but not the other two IPs. Maybe they’re not blocked now, or the list isn’t as unified as it sounds.

https://eais.rkn.gov.ru/en/

From these three which are inaccessible for me:
23.15.14.24
188.43.76.81
188.43.76.66

Only the first one is present in blacklist at https://eais.rkn.gov.ru/en/

I’ll try to contact with ISP, btw it is transtelecom.ru.

@opiumfor, you could try to use some kind of foreign DNS lookup service (including a web-based one or an ISP looking glass server) to look up ocsp.int-x3.letsencrypt.org from another country, and then set an /etc/hosts record to point to the resulting IP address. As this is a CDN-backed service, the CDN (in this case Akamai) is generally willing to connect you to the appropriate service even if you use an IP address that is not the usual or intended one for your country.

As of right now, some IP addresses in the United States for this host are

184.25.56.154
184.25.56.123
2600:1406:20::1748:2609
2600:1406:20::1748:2618

I believe Akamai will connect you with the appropriate Let’s Encrypt OCSP service if you connect to ocsp.int-x3.letsencrypt.org at any of those IP addresses.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

@opiumfor, Akamai’s tests are showing that transtelecom.ru has now unblocked the affected IPs. Could you please test again, and let us know the details if you’re still having trouble? Thanks for your patience!

2 Likes