http://ocsp.int-x3.letsencrypt.org/ is not working in hongkong.
http://ocsp.int-x3.letsencrypt.org/ is not working in hongkong.
What DNS resolver are you using? That’s an IP owned by Facebook, not Akamai.
Sorry, I do not know what DNS resolver using, I just have a Linux server in the Hongkong in Aliyun。But when I ping ocsp.int-x1.letsencrypt.org, ocsp.int-x2.letsencrypt.org, ocsp.int-x4.letsencrypt.org is working.
I’m so sorry, My linux server in China Mainland actually, not in Hong kong, sorry!. When I use renew command, below error occurred.
The error in your latest screenshot is not fatal. It occurs when Certbot is checking whether the certificate is revoked, but if it times out, renewal will still proceed anyway.
You can see below the error that “all renewals succeeded”.
As for why OCSP is timing out, could you please run this from your server also:
host ocsp.int-x3.letsencrypt.org host whoami.akamai.net
Ok, Thanks for your reply.
I cannot run host command in my server (command not found):
It seems to be caused by DNS pollution, you can set a valid IP for
ocsp.int-x3.letsencrypt.org in hosts to fix it.
Hi, we are getting similar evidence. We’ve run a catchpoint test comparing X1 and X3 availability and X3 OCSP responder availability in China is 10%. See: https://p.catchpoint.com/ui/Entry/PC/V/ARQi-C-D-b0wppjX12Hz5sAA
This is causing issues with some IOT devices that validate. What can be done about this beside switching to a different CA?
This is not new btw, this complaint from January echoes the same issue with X3: Cannot establish the TLS connection using some networks in China
In our case, solving with stapling is harder, as these are IOT devices that can’t be updated.
I’m preplexed as to how no one seems to care about this issue. OCSP stapling is fine for performance reasons, but the responder need to be globally accessible in any case, no?
Currently X1,X2,X4 work in china, X3 doesn’t:
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x4.letsencrypt.org
PING a796.dscq.akamai.net (188.8.131.52) 56(84) bytes of data.
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (184.108.40.206): icmp_seq=1 ttl=52 time=51.0 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (220.127.116.11): icmp_seq=2 ttl=52 time=51.2 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (18.104.22.168): icmp_seq=3 ttl=52 time=51.1 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (22.214.171.124): icmp_seq=4 ttl=52 time=52.3 ms
— a796.dscq.akamai.net ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 51.086/51.453/52.326/0.578 ms
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x3.letsencrypt.org
PING a771.dscq.akamai.net (126.96.36.199) 56(84) bytes of data.
— a771.dscq.akamai.net ping statistics —
8 packets transmitted, 0 received, 100% packet loss, time 7167ms
Sounds just like every other fun issue with the GFW?
Since they haven’t been tagged into the thread yet, @lestaff
We have an active ticket open with Akamai (our OCSP CDN provider) about this and are working towards a solution. It’s slow going so far.
@Phil, thanks for your answer, any news on this? We’ve been trying to bug Akamai about this as well
Unfortunately no good solution has been created yet. We’re still active on our Akamai ticket.
Further researching this, this looks to be a pure DNS problem. When dig is run out outside china, i get an ip address that is reachable from within china. So, this is specifically a DNS problem from within China:
ran@Rans-MacBook-Pro ~ % dig ocsp.int-x3.letsencrypt.org ; <<>> DiG 9.10.6 <<>> ocsp.int-x3.letsencrypt.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16002 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1452 ;; QUESTION SECTION: ;ocsp.int-x3.letsencrypt.org. IN A ;; ANSWER SECTION: ocsp.int-x3.letsencrypt.org. 7196 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net. ocsp.int-x3.letsencrypt.org.edgesuite.net. 21596 IN CNAME a771.dscq.akamai.net. a771.dscq.akamai.net. 16 IN A 188.8.131.52 a771.dscq.akamai.net. 16 IN A 184.108.40.206
(this from outside china)
Now this IP in china works:
ubuntu@iZbp18ebdg5i10j6m70ybnZ:~$ curl ocsp.int-x3.letsencrypt.org --resolve ocsp.int-x3.letsencrypt.org:80:220.127.116.11 -sv * Added ocsp.int-x3.letsencrypt.org:80:18.104.22.168 to DNS cache * Rebuilt URL to: ocsp.int-x3.letsencrypt.org/ * Hostname ocsp.int-x3.letsencrypt.org was found in DNS cache * Trying 22.214.171.124... * TCP_NODELAY set * Connected to ocsp.int-x3.letsencrypt.org (126.96.36.199) port 80 (#0) > GET / HTTP/1.1 > Host: ocsp.int-x3.letsencrypt.org > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx < Content-Length: 0 < Cache-Control: max-age=10559 < Expires: Tue, 05 May 2020 06:59:36 GMT < Date: Tue, 05 May 2020 04:03:37 GMT < Connection: keep-alive < * Connection #0 to host ocsp.int-x3.letsencrypt.org left intact
@Phil any news on this issue? Thanks!
Our CDN provider shares the conclusion other folks on this thread have reached, that it appears to be a problem caused by the Great Firewall interfering with DNS. We probably cannot work around this issue on our end.
A couple of possible workarounds for clients that you control:
If you configure a different DNS resolver for the affected devices (like 188.8.131.52 or 184.108.40.206) do you get the correct IP address, or do the responses get rewritten / blocked?
If those response are rewritten / blocked, do you have the option of using DNS-over-HTTPS to a DNS resolver server that gets the correct IP address for our OCSP responder?
@jsha, I’m sorry but I do not control the clients. My customer in this case is an IOT provider of smart devices running a hardened realtime OS, they can’t patch them. Look at the answer section in the call from China:
;; ANSWER SECTION: ocsp.int-x3.letsencrypt.org. 5580 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net. ocsp.int-x3.letsencrypt.org.edgesuite.net. 7185 IN CNAME a771.dscq.akamai.net. a771.dscq.akamai.net. 147 IN A 220.127.116.11
The interference is happening at the last point - 18.104.22.168 is a bogus IP, but a771.dscq.akamai.net is correct. I believe Akamai can fix this, by changing the slot number (a771) to one that is not being interfered with. Did they already do that and that got the new slot blocked as well? This doesn’t seem to be an intentional block by the firewall, as if it was, they would have blocked x1,x2,x4 as well, no? And if the block was intentional, wouldn’t the Great Firewall interfere with ocsp.int-x3.letsencrypt.org? Perhaps the a771 hostname on Akamai was used in the past by an infringing website and the block remained on it?
Our only solution if this case isn’t resolved is to use a different certificate authority (which we already did for this specific customer, thanks Globalsign!) and eventually migrate off Let’s Encrypt completely, if this is not resolved. It seems strange that Let’s Encrypt engineers are OK with leaving 1/4 of their OCSP servers inaccessible from the most populous country in the world.
These receive no traffic to begin with, as they are for retired or backup issuers. No reason to block something that’s not used.
But changing the CNAME does sound like a good idea in theory, if they haven’t tried it already.