Ocsp.int-x3.letsencrypt.org is not working in China

It seems to be caused by DNS pollution, you can set a valid IP for ocsp.int-x3.letsencrypt.org in hosts to fix it.
这似乎是由于中国内地的DNS污染引起的问题,可以在hosts里写对应的正确IP规避一下。

1 Like

Hi, we are getting similar evidence. We’ve run a catchpoint test comparing X1 and X3 availability and X3 OCSP responder availability in China is 10%. See: https://p.catchpoint.com/ui/Entry/PC/V/ARQi-C-D-b0wppjX12Hz5sAA

This is causing issues with some IOT devices that validate. What can be done about this beside switching to a different CA?

1 Like

This is not new btw, this complaint from January echoes the same issue with X3: Cannot establish the TLS connection using some networks in China
In our case, solving with stapling is harder, as these are IOT devices that can’t be updated.

1 Like

I’m preplexed as to how no one seems to care about this issue. OCSP stapling is fine for performance reasons, but the responder need to be globally accessible in any case, no?
Currently X1,X2,X4 work in china, X3 doesn’t:

root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x4.letsencrypt.org
PING a796.dscq.akamai.net (23.62.226.194) 56(84) bytes of data.
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=1 ttl=52 time=51.0 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=2 ttl=52 time=51.2 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=3 ttl=52 time=51.1 ms
64 bytes from a23-62-226-194.deploy.static.akamaitechnologies.com (23.62.226.194): icmp_seq=4 ttl=52 time=52.3 ms
^C
a796.dscq.akamai.net ping statistics —
4 packets transmitted, 4 received, 0% packet loss, time 3009ms
rtt min/avg/max/mdev = 51.086/51.453/52.326/0.578 ms
root@iZbp16vbzlu591t1v2hcb7Z:~# ping ocsp.int-x3.letsencrypt.org
PING a771.dscq.akamai.net (66.220.151.20) 56(84) bytes of data.

^C
a771.dscq.akamai.net ping statistics —
8 packets transmitted, 0 received, 100% packet loss, time 7167ms

1 Like

Sounds just like every other fun issue with the GFW?

Since they haven’t been tagged into the thread yet, @lestaff

3 Likes

We have an active ticket open with Akamai (our OCSP CDN provider) about this and are working towards a solution. It’s slow going so far.

6 Likes

@Phil, thanks for your answer, any news on this? We’ve been trying to bug Akamai about this as well :slight_smile:

1 Like

@ranrub
Unfortunately no good solution has been created yet. We’re still active on our Akamai ticket.

3 Likes

Further researching this, this looks to be a pure DNS problem. When dig is run out outside china, i get an ip address that is reachable from within china. So, this is specifically a DNS problem from within China:

ran@Rans-MacBook-Pro ~ % dig ocsp.int-x3.letsencrypt.org

; <<>> DiG 9.10.6 <<>> ocsp.int-x3.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16002
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;ocsp.int-x3.letsencrypt.org. IN A

;; ANSWER SECTION:
ocsp.int-x3.letsencrypt.org. 7196 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 21596 IN CNAME a771.dscq.akamai.net.
a771.dscq.akamai.net. 16 IN A 104.99.238.41
a771.dscq.akamai.net. 16 IN A 104.99.238.33

(this from outside china)

Now this IP in china works:

ubuntu@iZbp18ebdg5i10j6m70ybnZ:~$ curl ocsp.int-x3.letsencrypt.org --resolve ocsp.int-x3.letsencrypt.org:80:104.99.238.41 -sv
* Added ocsp.int-x3.letsencrypt.org:80:104.99.238.41 to DNS cache
* Rebuilt URL to: ocsp.int-x3.letsencrypt.org/
* Hostname ocsp.int-x3.letsencrypt.org was found in DNS cache
* Trying 104.99.238.41...
* TCP_NODELAY set
* Connected to ocsp.int-x3.letsencrypt.org (104.99.238.41) port 80 (#0)
> GET / HTTP/1.1
> Host: ocsp.int-x3.letsencrypt.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Length: 0
< Cache-Control: max-age=10559
< Expires: Tue, 05 May 2020 06:59:36 GMT
< Date: Tue, 05 May 2020 04:03:37 GMT
< Connection: keep-alive
<
* Connection #0 to host ocsp.int-x3.letsencrypt.org left intact
1 Like

@Phil any news on this issue? Thanks!

2 Likes

Our CDN provider shares the conclusion other folks on this thread have reached, that it appears to be a problem caused by the Great Firewall interfering with DNS. We probably cannot work around this issue on our end.

A couple of possible workarounds for clients that you control:

If you configure a different DNS resolver for the affected devices (like 1.1.1.1 or 8.8.8.8) do you get the correct IP address, or do the responses get rewritten / blocked?

If those response are rewritten / blocked, do you have the option of using DNS-over-HTTPS to a DNS resolver server that gets the correct IP address for our OCSP responder?

Thanks,
Jacob

1 Like

@jsha, I’m sorry but I do not control the clients. My customer in this case is an IOT provider of smart devices running a hardened realtime OS, they can’t patch them. Look at the answer section in the call from China:

;; ANSWER SECTION:
ocsp.int-x3.letsencrypt.org. 5580 IN	CNAME	ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 7185	IN CNAME a771.dscq.akamai.net.
a771.dscq.akamai.net.	147	IN	A	31.13.73.17

The interference is happening at the last point - 31.13.73.17 is a bogus IP, but a771.dscq.akamai.net is correct. I believe Akamai can fix this, by changing the slot number (a771) to one that is not being interfered with. Did they already do that and that got the new slot blocked as well? This doesn’t seem to be an intentional block by the firewall, as if it was, they would have blocked x1,x2,x4 as well, no? And if the block was intentional, wouldn’t the Great Firewall interfere with ocsp.int-x3.letsencrypt.org? Perhaps the a771 hostname on Akamai was used in the past by an infringing website and the block remained on it?

Our only solution if this case isn’t resolved is to use a different certificate authority (which we already did for this specific customer, thanks Globalsign!) and eventually migrate off Let’s Encrypt completely, if this is not resolved. It seems strange that Let’s Encrypt engineers are OK with leaving 1/4 of their OCSP servers inaccessible from the most populous country in the world.

3 Likes

These receive no traffic to begin with, as they are for retired or backup issuers. No reason to block something that’s not used.

But changing the CNAME does sound like a good idea in theory, if they haven’t tried it already.

2 Likes

This is a good idea, we’ll ask about this.

3 Likes

@jsha / @_az any news about this?
Thanks!
Ran

2 Likes

Currently waiting on an update from Akamai.

2 Likes

Tested, and it seems to be a solution. Hope this can get fixed soon.

Since a771 is the CNAME of int-x3, we can test an OCSP request this way (as in the article OpenSSL: Manually verify a certificate against an OCSP):

openssl ocsp -issuer issuer.pem -VAfile issuer.pem -cert cert.pem -url http://a771.dscq.akamai.net/ -text -header HOST=ocsp.int-x3.letsencrypt.org

We get a connection error for this, as expected.

The response of dig to all the OCSP servers is:

ocsp.int-x1.letsencrypt.org.edgesuite.net. 20996 IN CNAME a1824.dscq.akamai.net.
ocsp.int-x2.letsencrypt.org.edgesuite.net. 21599 IN CNAME a1796.dscq.akamai.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 6502 IN CNAME a771.dscq.akamai.net.
ocsp.int-x4.letsencrypt.org.edgesuite.net. 21600 IN CNAME a796.dscq.akamai.net.

Considering a771 is contaminated and assuming others are clean, replace it with a796, a1796, a1824 in the openssl command. And we can get valid responses like:

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    ...
    Cert Status: good
    ...
WARNING: no nonce in response
Response verify OK
cert.pem: good
    ...
1 Like

Hope it can be faster, it’s been a month, our ios users are still affected

1 Like

For whatever it is worth, we are currently trying to get this <foo>.dscq.akamai.net name changed. We can only expect that this will be a temporary fix, but we investigating ways to manage it going forward.

3 Likes

I am currently working for a Chinese customer. As ranrub suggested, pinging from Germany gives me an IP that’s reachable from China too.

I used this as a workaround: Set the IP you get from outside China into /etc/hosts for ocsp.int-x3.letsencrypt.org and certbot should run without complaint. Of course that can be only a workaround, since when the IP number changes it stops working however a workaround at least.

2 Likes