Let's Encrypt IPs are banned in Russia

This issue was previously reported by SagePtr on GitHub.

Summary

This article in Russian describes the problem, in short:
Two IPs, 142.93.108.123 and 167.99.129.42, that host Let’s Encrypt website (letsencrypt.org) are in the Roscomnadzor Registry of forbidden resources. Roscomnadzor did not specifically target Let’s Encrypt, rather it banned a large number of IPs in connection with other websites and Let’s Encrypt just recently moved to these two APIs (after switching hosting from Akamai Technologies, Inc. to Digital Ocean, LLC). Authors note that so far the ban affects only the website, but warns that if Let’s Encrypt certificate issuance APIs move too, they might become unavailable.

I’m not affiliated with Let’s Encrypt in any way, but as far as I know, Let’s Encrypt APIs are hosted in a co-location environment and should not change IPs so are not affected by the Russian Roscomnadzor roulette. Could someone knowledgeable confirm this?

Full article translation

(WIP)
Here is a full article translated to help Let’s Encrypt staff investigate the matter.

4 Likes

The API endpoints’ public IP addresses belong to the CDN providers Let’s Encrypt uses – previously Akamai, and currently Cloudflare. (Of course, that’s subject to change.) For example:

acme-v02.api.letsencrypt.org.                       7042  CNAME  prod.api.letsencrypt.org.
prod.api.letsencrypt.org.                           180   CNAME  ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.  89    A      172.65.32.248
ca80a1adb12a4fbdac5ffcbc944e9a61.pacloudflare.com.  137   AAAA   2606:4700:60:0:f53d:5624:85c7:3a2c

If a government blocks CDN IPs with abandon, they could inadvertently block the API.


https://letsencrypt.org/ and https://www.letsencrypt.org/ are currently outsourced to Netlify, apparently.

5 Likes

I believe Netlify use a mix of AWS, Digital Ocean, etc for their edges. And the Let’s Encrypt marketing site uses Netlify.

If Russia ends up blocking Netlify servers (which is like blocking an entire CDN), I am sure Netlify will work it out with the authorities or just remove the blocked edge servers from their network.

3 Likes

One other way that blanket IP blocking could interfere with issuing certificates, is if traffic FROM our validation services was blocked.

4 Likes

Russia blocked huge amounts of DigitalOcean’s address space because Telegram uses or used it. It’s a problem.

3 Likes

Netlify have responded already apparently (https://community.netlify.com/t/netlify-cdn-or-dns-blocked-in-russia/372/3):

Here is what we can tell you:

  1. We are currently working with the Russian authorities in order to resolve this roadblock, and have contacted them several times over the last few months.
  2. We have been using our corporate “network” of contacts in other companies, and when speaking to others, it seems like that they are taking an unusually long time to respond to our requests, even though we have made them through the correct channels. We will keep you updated as soon as we have information that seems reliable as to how and when this situation is going to be resolved.
  3. We’ve begun exploring working with a Russia-local agency to see if they can help us “from the inside”. They advise that these blocks seem likely to be accidental, and are somehow leftover from the “Great Telegram Blockage of 2018” , though we moved nodes to new addresses and found that they too were blocked shortly afterwards, so there must be more to the story. We do use network providers that were affected by the block.
  4. Finally we are considering trying to set up a CDN Point of Presence within Russia. We’ve received one referral to a hosting company but we’d be interested in introductions to other ones as we have some unique needs and will be shopping around - if you know of any to recommend, please share contact info here or via DM!

But it was quite a few months ago …

5 Likes

To the best of my knowledge, they only ban traffic TO a blocked resource. The author of the article is concerned that certbot might not be able to connect to Let’s Encrypt servers to request a challenge and (after challenge completion) to request a signed certificate.
Could you confirm that this would not be affected? Also, what information passing mechanism does certbot use that is not affected by the block?

As a side note, I wonder if IP bans could affect CT logs and OCSP (including stapling).

2 Likes

Netlify have responded already apparently (https://community.netlify.com/t/netlify-cdn-or-dns-blocked-in-russia/372/3)

Thanks for the link. Unfortunately, there is not much companies can do to unblock their IPs.

these blocks seem likely to be accidental, and are somehow leftover from the “Great Telegram Blockage of 2018”

Yes, this is a collateral damage from Roskomnadzor IP bans. In this case, the IPs are not connected to Telegram blockage; instead, these IPs were blocked for allegedly hosting cryptocurrency casinos, some depressing poetry and something else.

Finally we are considering trying to set up a CDN Point of Presence within Russia.

Considering trying? Does not sound very definitive…

3 Likes

While not ideal, you can access the content available on letsencrypt.org via the Github repository assuming Github is not blocked: https://github.com/letsencrypt/website

The content is in the raw un-processed form in the repo but much of it is Markdown that is fairly legible. In theory one could also install Hugo and host a mirror locally that could be accessed in a browser.

2 Likes