It seems to be caused by DNS pollution, you can set a valid IP for
ocsp.int-x3.letsencrypt.org in hosts to fix it.
It seems to be caused by DNS pollution, you can set a valid IP for
Hi, we are getting similar evidence. We’ve run a catchpoint test comparing X1 and X3 availability and X3 OCSP responder availability in China is 10%. See: https://p.catchpoint.com/ui/Entry/PC/V/ARQi-C-D-b0wppjX12Hz5sAA
This is causing issues with some IOT devices that validate. What can be done about this beside switching to a different CA?
This is not new btw, this complaint from January echoes the same issue with X3: Cannot establish the TLS connection using some networks in China
In our case, solving with stapling is harder, as these are IOT devices that can’t be updated.
I’m preplexed as to how no one seems to care about this issue. OCSP stapling is fine for performance reasons, but the responder need to be globally accessible in any case, no?
Currently X1,X2,X4 work in china, X3 doesn’t:
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x4.letsencrypt.org
PING a796.dscq.akamai.net (22.214.171.124) 56(84) bytes of data.
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (126.96.36.199): icmp_seq=1 ttl=52 time=51.0 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (188.8.131.52): icmp_seq=2 ttl=52 time=51.2 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (184.108.40.206): icmp_seq=3 ttl=52 time=51.1 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (220.127.116.11): icmp_seq=4 ttl=52 time=52.3 ms
— a796.dscq.akamai.net ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 51.086/51.453/52.326/0.578 ms
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x3.letsencrypt.org
PING a771.dscq.akamai.net (18.104.22.168) 56(84) bytes of data.
— a771.dscq.akamai.net ping statistics —
8 packets transmitted, 0 received, 100% packet loss, time 7167ms
Sounds just like every other fun issue with the GFW?
Since they haven’t been tagged into the thread yet, @lestaff
We have an active ticket open with Akamai (our OCSP CDN provider) about this and are working towards a solution. It’s slow going so far.
@Phil, thanks for your answer, any news on this? We’ve been trying to bug Akamai about this as well
Unfortunately no good solution has been created yet. We’re still active on our Akamai ticket.
Further researching this, this looks to be a pure DNS problem. When dig is run out outside china, i get an ip address that is reachable from within china. So, this is specifically a DNS problem from within China:
ran@Rans-MacBook-Pro ~ % dig ocsp.int-x3.letsencrypt.org ; <<>> DiG 9.10.6 <<>> ocsp.int-x3.letsencrypt.org ;; global options: +cmd ;; Got answer: ;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16002 ;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1 ;; OPT PSEUDOSECTION: ; EDNS: version: 0, flags:; udp: 1452 ;; QUESTION SECTION: ;ocsp.int-x3.letsencrypt.org. IN A ;; ANSWER SECTION: ocsp.int-x3.letsencrypt.org. 7196 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net. ocsp.int-x3.letsencrypt.org.edgesuite.net. 21596 IN CNAME a771.dscq.akamai.net. a771.dscq.akamai.net. 16 IN A 22.214.171.124 a771.dscq.akamai.net. 16 IN A 126.96.36.199
(this from outside china)
Now this IP in china works:
ubuntu@iZbp18ebdg5i10j6m70ybnZ:~$ curl ocsp.int-x3.letsencrypt.org --resolve ocsp.int-x3.letsencrypt.org:80:188.8.131.52 -sv * Added ocsp.int-x3.letsencrypt.org:80:184.108.40.206 to DNS cache * Rebuilt URL to: ocsp.int-x3.letsencrypt.org/ * Hostname ocsp.int-x3.letsencrypt.org was found in DNS cache * Trying 220.127.116.11... * TCP_NODELAY set * Connected to ocsp.int-x3.letsencrypt.org (18.104.22.168) port 80 (#0) > GET / HTTP/1.1 > Host: ocsp.int-x3.letsencrypt.org > User-Agent: curl/7.58.0 > Accept: */* > < HTTP/1.1 200 OK < Server: nginx < Content-Length: 0 < Cache-Control: max-age=10559 < Expires: Tue, 05 May 2020 06:59:36 GMT < Date: Tue, 05 May 2020 04:03:37 GMT < Connection: keep-alive < * Connection #0 to host ocsp.int-x3.letsencrypt.org left intact
@Phil any news on this issue? Thanks!
Our CDN provider shares the conclusion other folks on this thread have reached, that it appears to be a problem caused by the Great Firewall interfering with DNS. We probably cannot work around this issue on our end.
A couple of possible workarounds for clients that you control:
If you configure a different DNS resolver for the affected devices (like 22.214.171.124 or 126.96.36.199) do you get the correct IP address, or do the responses get rewritten / blocked?
If those response are rewritten / blocked, do you have the option of using DNS-over-HTTPS to a DNS resolver server that gets the correct IP address for our OCSP responder?
@jsha, I’m sorry but I do not control the clients. My customer in this case is an IOT provider of smart devices running a hardened realtime OS, they can’t patch them. Look at the answer section in the call from China:
;; ANSWER SECTION: ocsp.int-x3.letsencrypt.org. 5580 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net. ocsp.int-x3.letsencrypt.org.edgesuite.net. 7185 IN CNAME a771.dscq.akamai.net. a771.dscq.akamai.net. 147 IN A 188.8.131.52
The interference is happening at the last point - 184.108.40.206 is a bogus IP, but a771.dscq.akamai.net is correct. I believe Akamai can fix this, by changing the slot number (a771) to one that is not being interfered with. Did they already do that and that got the new slot blocked as well? This doesn’t seem to be an intentional block by the firewall, as if it was, they would have blocked x1,x2,x4 as well, no? And if the block was intentional, wouldn’t the Great Firewall interfere with ocsp.int-x3.letsencrypt.org? Perhaps the a771 hostname on Akamai was used in the past by an infringing website and the block remained on it?
Our only solution if this case isn’t resolved is to use a different certificate authority (which we already did for this specific customer, thanks Globalsign!) and eventually migrate off Let’s Encrypt completely, if this is not resolved. It seems strange that Let’s Encrypt engineers are OK with leaving 1/4 of their OCSP servers inaccessible from the most populous country in the world.
These receive no traffic to begin with, as they are for retired or backup issuers. No reason to block something that’s not used.
But changing the CNAME does sound like a good idea in theory, if they haven’t tried it already.
This is a good idea, we’ll ask about this.
Currently waiting on an update from Akamai.
Tested, and it seems to be a solution. Hope this can get fixed soon.
Since a771 is the CNAME of int-x3, we can test an OCSP request this way (as in the article OpenSSL: Manually verify a certificate against an OCSP):
openssl ocsp -issuer issuer.pem -VAfile issuer.pem -cert cert.pem -url http://a771.dscq.akamai.net/ -text -header HOST=ocsp.int-x3.letsencrypt.org
We get a connection error for this, as expected.
The response of dig to all the OCSP servers is:
ocsp.int-x1.letsencrypt.org.edgesuite.net. 20996 IN CNAME a1824.dscq.akamai.net. ocsp.int-x2.letsencrypt.org.edgesuite.net. 21599 IN CNAME a1796.dscq.akamai.net. ocsp.int-x3.letsencrypt.org.edgesuite.net. 6502 IN CNAME a771.dscq.akamai.net. ocsp.int-x4.letsencrypt.org.edgesuite.net. 21600 IN CNAME a796.dscq.akamai.net.
Considering a771 is contaminated and assuming others are clean, replace it with a796, a1796, a1824 in the openssl command. And we can get valid responses like:
OCSP Response Data: OCSP Response Status: successful (0x0) ... Cert Status: good ... WARNING: no nonce in response Response verify OK cert.pem: good ...
Hope it can be faster, it’s been a month, our ios users are still affected
For whatever it is worth, we are currently trying to get this
<foo>.dscq.akamai.net name changed. We can only expect that this will be a temporary fix, but we investigating ways to manage it going forward.
I am currently working for a Chinese customer. As ranrub suggested, pinging from Germany gives me an IP that’s reachable from China too.
I used this as a workaround: Set the IP you get from outside China into /etc/hosts for ocsp.int-x3.letsencrypt.org and certbot should run without complaint. Of course that can be only a workaround, since when the IP number changes it stops working however a workaround at least.