Ocsp.int-x3.letsencrypt.org is not working in China

@Phil, thanks for your answer, any news on this? We’ve been trying to bug Akamai about this as well :slight_smile:

1 Like

@ranrub
Unfortunately no good solution has been created yet. We’re still active on our Akamai ticket.

3 Likes

Further researching this, this looks to be a pure DNS problem. When dig is run out outside china, i get an ip address that is reachable from within china. So, this is specifically a DNS problem from within China:

ran@Rans-MacBook-Pro ~ % dig ocsp.int-x3.letsencrypt.org

; <<>> DiG 9.10.6 <<>> ocsp.int-x3.letsencrypt.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 16002
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1452
;; QUESTION SECTION:
;ocsp.int-x3.letsencrypt.org. IN A

;; ANSWER SECTION:
ocsp.int-x3.letsencrypt.org. 7196 IN CNAME ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 21596 IN CNAME a771.dscq.akamai.net.
a771.dscq.akamai.net. 16 IN A 104.99.238.41
a771.dscq.akamai.net. 16 IN A 104.99.238.33

(this from outside china)

Now this IP in china works:

ubuntu@iZbp18ebdg5i10j6m70ybnZ:~$ curl ocsp.int-x3.letsencrypt.org --resolve ocsp.int-x3.letsencrypt.org:80:104.99.238.41 -sv
* Added ocsp.int-x3.letsencrypt.org:80:104.99.238.41 to DNS cache
* Rebuilt URL to: ocsp.int-x3.letsencrypt.org/
* Hostname ocsp.int-x3.letsencrypt.org was found in DNS cache
* Trying 104.99.238.41...
* TCP_NODELAY set
* Connected to ocsp.int-x3.letsencrypt.org (104.99.238.41) port 80 (#0)
> GET / HTTP/1.1
> Host: ocsp.int-x3.letsencrypt.org
> User-Agent: curl/7.58.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Content-Length: 0
< Cache-Control: max-age=10559
< Expires: Tue, 05 May 2020 06:59:36 GMT
< Date: Tue, 05 May 2020 04:03:37 GMT
< Connection: keep-alive
<
* Connection #0 to host ocsp.int-x3.letsencrypt.org left intact
1 Like

@Phil any news on this issue? Thanks!

2 Likes

Our CDN provider shares the conclusion other folks on this thread have reached, that it appears to be a problem caused by the Great Firewall interfering with DNS. We probably cannot work around this issue on our end.

A couple of possible workarounds for clients that you control:

If you configure a different DNS resolver for the affected devices (like 1.1.1.1 or 8.8.8.8) do you get the correct IP address, or do the responses get rewritten / blocked?

If those response are rewritten / blocked, do you have the option of using DNS-over-HTTPS to a DNS resolver server that gets the correct IP address for our OCSP responder?

Thanks,
Jacob

1 Like

@jsha, I’m sorry but I do not control the clients. My customer in this case is an IOT provider of smart devices running a hardened realtime OS, they can’t patch them. Look at the answer section in the call from China:

;; ANSWER SECTION:
ocsp.int-x3.letsencrypt.org. 5580 IN	CNAME	ocsp.int-x3.letsencrypt.org.edgesuite.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 7185	IN CNAME a771.dscq.akamai.net.
a771.dscq.akamai.net.	147	IN	A	31.13.73.17

The interference is happening at the last point - 31.13.73.17 is a bogus IP, but a771.dscq.akamai.net is correct. I believe Akamai can fix this, by changing the slot number (a771) to one that is not being interfered with. Did they already do that and that got the new slot blocked as well? This doesn’t seem to be an intentional block by the firewall, as if it was, they would have blocked x1,x2,x4 as well, no? And if the block was intentional, wouldn’t the Great Firewall interfere with ocsp.int-x3.letsencrypt.org? Perhaps the a771 hostname on Akamai was used in the past by an infringing website and the block remained on it?

Our only solution if this case isn’t resolved is to use a different certificate authority (which we already did for this specific customer, thanks Globalsign!) and eventually migrate off Let’s Encrypt completely, if this is not resolved. It seems strange that Let’s Encrypt engineers are OK with leaving 1/4 of their OCSP servers inaccessible from the most populous country in the world.

3 Likes

These receive no traffic to begin with, as they are for retired or backup issuers. No reason to block something that's not used.

But changing the CNAME does sound like a good idea in theory, if they haven't tried it already.

2 Likes

This is a good idea, we'll ask about this.

3 Likes

@jsha / @_az any news about this?
Thanks!
Ran

1 Like

Currently waiting on an update from Akamai.

2 Likes

Tested, and it seems to be a solution. Hope this can get fixed soon.

Since a771 is the CNAME of int-x3, we can test an OCSP request this way (as in the article OpenSSL: Manually verify a certificate against an OCSP):

openssl ocsp -issuer issuer.pem -VAfile issuer.pem -cert cert.pem -url http://a771.dscq.akamai.net/ -text -header HOST=ocsp.int-x3.letsencrypt.org

We get a connection error for this, as expected.

The response of dig to all the OCSP servers is:

ocsp.int-x1.letsencrypt.org.edgesuite.net. 20996 IN CNAME a1824.dscq.akamai.net.
ocsp.int-x2.letsencrypt.org.edgesuite.net. 21599 IN CNAME a1796.dscq.akamai.net.
ocsp.int-x3.letsencrypt.org.edgesuite.net. 6502 IN CNAME a771.dscq.akamai.net.
ocsp.int-x4.letsencrypt.org.edgesuite.net. 21600 IN CNAME a796.dscq.akamai.net.

Considering a771 is contaminated and assuming others are clean, replace it with a796, a1796, a1824 in the openssl command. And we can get valid responses like:

OCSP Response Data:
    OCSP Response Status: successful (0x0)
    ...
    Cert Status: good
    ...
WARNING: no nonce in response
Response verify OK
cert.pem: good
    ...
1 Like

Hope it can be faster, it’s been a month, our ios users are still affected

1 Like

For whatever it is worth, we are currently trying to get this <foo>.dscq.akamai.net name changed. We can only expect that this will be a temporary fix, but we investigating ways to manage it going forward.

2 Likes

I am currently working for a Chinese customer. As ranrub suggested, pinging from Germany gives me an IP that’s reachable from China too.

I used this as a workaround: Set the IP you get from outside China into /etc/hosts for ocsp.int-x3.letsencrypt.org and certbot should run without complaint. Of course that can be only a workaround, since when the IP number changes it stops working however a workaround at least.

1 Like

@ezekiel any news about this proposed change? Thanks!

1 Like

Sorry, so far we have only been able to rule OUT actions which have the potential to cause increased outages and have not yet identified a good method of working around China’s block.

1 Like

@ezekiel , @_az , any news on this issue? Is it still being actively pursued by LE? We’re still getting blocked in China for OCSP requests.

2 Likes

Hi Thread!

Thank you all for your patience, as I know we are all quite frustrated by the block :dizzy_face: .

Our long-term fix is to change the OCSP URL in newly issued certificates. As one might imagine, this is not a simple nor immediate change. We have plenty of work to do before this will roll out, but in the end we will be using a URL which isn’t affected by the current Akamai map segment which is blocked in China. We plan to make the URL shorter, as well!

Watch for threads in the API Announcements Category for more information as our timeline becomes clearer.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

This thread has automatically closed, but for anyone interested, our current issuance chain involves a new OCSP URI ( with a new, short domain o.lencr.org ) which is also handled within a new CDN "map". So, OCSP for newly-issued certificates should begin working again in China over the 90 days following this post: Beginning Issuance from R3 ( unless the new CDN map or any new URIs are blocked ).

6 Likes