http://ocsp.int-x3.letsencrypt.org/ is not working in hongkong.
4 Likes
What DNS resolver are you using? That’s an IP owned by Facebook, not Akamai.
1 Like
Sorry, I do not know what DNS resolver using, I just have a Linux server in the Hongkong in Aliyun。But when I ping ocsp.int-x1.letsencrypt.org, ocsp.int-x2.letsencrypt.org, ocsp.int-x4.letsencrypt.org is working.
1 Like
I’m so sorry, My linux server in China Mainland actually, not in Hong kong, sorry!. When I use renew command, below error occurred.
1 Like
The error in your latest screenshot is not fatal. It occurs when Certbot is checking whether the certificate is revoked, but if it times out, renewal will still proceed anyway.
You can see below the error that “all renewals succeeded”.
As for why OCSP is timing out, could you please run this from your server also:
host ocsp.int-x3.letsencrypt.org
host whoami.akamai.netOk, Thanks for your reply.
I cannot run host command in my server (command not found):
It seems to be caused by DNS pollution, you can set a valid IP for ocsp.int-x3.letsencrypt.org in hosts to fix it.
这似乎是由于中国内地的DNS污染引起的问题，可以在hosts里写对应的正确IP规避一下。
1 Like
Hi, we are getting similar evidence. We’ve run a catchpoint test comparing X1 and X3 availability and X3 OCSP responder availability in China is 10%. See: https://p.catchpoint.com/ui/Entry/PC/V/ARQi-C-D-b0wppjX12Hz5sAA
This is causing issues with some IOT devices that validate. What can be done about this beside switching to a different CA?
1 Like
This is not new btw, this complaint from January echoes the same issue with X3: Cannot establish the TLS connection using some networks in China
In our case, solving with stapling is harder, as these are IOT devices that can’t be updated.
1 Like
I'm preplexed as to how no one seems to care about this issue. OCSP stapling is fine for performance reasons, but the responder need to be globally accessible in any case, no?
Currently X1,X2,X4 work in china, X3 doesn't:
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x4.letsencrypt.org
PING a796.dscq.akamai.net (23.62.226.194) 56(84) bytes of data.
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=1 ttl=52 time=51.0 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=2 ttl=52 time=51.2 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=3 ttl=52 time=51.1 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=4 ttl=52 time=52.3 ms
^C
--- a796.dscq.akamai.net ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 51.086/51.453/52.326/0.578 ms
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x3.letsencrypt.org
PING a771.dscq.akamai.net (66.220.151.20) 56(84) bytes of data.
^C
--- a771.dscq.akamai.net ping statistics ---
8 packets transmitted, 0 received, 100% packet loss, time 7167ms
1 Like
Sounds just like every other fun issue with the GFW?
Since they haven't been tagged into the thread yet, @lestaff
3 Likes
We have an active ticket open with Akamai (our OCSP CDN provider) about this and are working towards a solution. It’s slow going so far.
6 Likes
@Phil, thanks for your answer, any news on this? We’ve been trying to bug Akamai about this as well 
1 Like
@ranrub
Unfortunately no good solution has been created yet. We’re still active on our Akamai ticket.
3 Likes
Further researching this, this looks to be a pure DNS problem. When dig is run out outside china, i get an ip address that is reachable from within china. So, this is specifically a DNS problem from within China:
ran@Rans-MacBook-Pro ~ % dig ocsp.int-x3.letsencrypt.org
; <<>> DiG 9.10.6 <<>> ocsp.int-x3.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16002
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;ocsp.int-x3.letsencrypt.org. IN A
;; ANSWER SECTION:
ocsp.int-x3.letsencrypt.org. 7196 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 21596 IN CNAME a771.dscq.akamai.net.
a771.dscq.akamai.net. 16 IN A 104.99.238.41
a771.dscq.akamai.net. 16 IN A 104.99.238.33
(this from outside china)
Now this IP in china works:
ubuntu@iZbp18ebdg5i10j6m70ybnZ:~$ curl ocsp.int-x3.letsencrypt.org --resolve ocsp.int-x3.letsencrypt.org:80:104.99.238.41 -sv
* Added ocsp.int-x3.letsencrypt.org:80:104.99.238.41 to DNS cache
* Rebuilt URL to: ocsp.int-x3.letsencrypt.org/
* Hostname ocsp.int-x3.letsencrypt.org was found in DNS cache
* Trying 104.99.238.41...
* TCP_NODELAY set
* Connected to ocsp.int-x3.letsencrypt.org (104.99.238.41) port 80 (#0)
> GET / HTTP/1.1
> Host: ocsp.int-x3.letsencrypt.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Length: 0
< Cache-Control: max-age=10559
< Expires: Tue, 05 May 2020 06:59:36 GMT
< Date: Tue, 05 May 2020 04:03:37 GMT
< Connection: keep-alive
<
* Connection #0 to host ocsp.int-x3.letsencrypt.org left intact
1 Like
Our CDN provider shares the conclusion other folks on this thread have reached, that it appears to be a problem caused by the Great Firewall interfering with DNS. We probably cannot work around this issue on our end.
A couple of possible workarounds for clients that you control:
If you configure a different DNS resolver for the affected devices (like 1.1.1.1 or 8.8.8.8) do you get the correct IP address, or do the responses get rewritten / blocked?
If those response are rewritten / blocked, do you have the option of using DNS-over-HTTPS to a DNS resolver server that gets the correct IP address for our OCSP responder?
Thanks,
Jacob
1 Like
@jsha, I’m sorry but I do not control the clients. My customer in this case is an IOT provider of smart devices running a hardened realtime OS, they can’t patch them. Look at the answer section in the call from China:
;; ANSWER SECTION:
ocsp.int-x3.letsencrypt.org. 5580 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 7185 IN CNAME a771.dscq.akamai.net.
a771.dscq.akamai.net. 147 IN A 31.13.73.17
The interference is happening at the last point - 31.13.73.17 is a bogus IP, but a771.dscq.akamai.net is correct. I believe Akamai can fix this, by changing the slot number (a771) to one that is not being interfered with. Did they already do that and that got the new slot blocked as well? This doesn’t seem to be an intentional block by the firewall, as if it was, they would have blocked x1,x2,x4 as well, no? And if the block was intentional, wouldn’t the Great Firewall interfere with ocsp.int-x3.letsencrypt.org? Perhaps the a771 hostname on Akamai was used in the past by an infringing website and the block remained on it?
Our only solution if this case isn’t resolved is to use a different certificate authority (which we already did for this specific customer, thanks Globalsign!) and eventually migrate off Let’s Encrypt completely, if this is not resolved. It seems strange that Let’s Encrypt engineers are OK with leaving 1/4 of their OCSP servers inaccessible from the most populous country in the world.
3 Likes
These receive no traffic to begin with, as they are for retired or backup issuers. No reason to block something that's not used.
But changing the CNAME does sound like a good idea in theory, if they haven't tried it already.
2 Likes