LE seems to depend heavily on Cloudflare, from DNS to proxying API requests to I know not what else. Unfortunately, in the last week I have seen a marked increase in abusive traffic from Cloudflare IP ranges (presumably bad actors hiding themselves behind Cloudflare proxies, but possibly compromised Cloudflare hosts, too). This has triggered automated blocking on my servers, with 2400:cb00::/36 and 172.64.0.0/13 being added to my firewall. The API seems to be using a different IPv6 segment, but that huge IPv4 range appears to cover most/all LE services.
The "Feature Request" aspect that solves the issue can come in many forms. Can you support an alternative (or additional) provider other than Cloudflare? Can you obtain a SLA from them that allows you to publish a whitelist of IPs you're allocated that are part of a "trustworthy" pool? What other suggestions do people have for obtaining essential services from hostile networks?
You should be able to configure your firewall to allow access to banned IP Addresses if you initiate the connection. That would allow your servers to interact with the LetsEncrypt API. You could then use a DNS-01 challenge that resolves outside your network, or take down the firewall for a few seconds to complete a HTTP-01 challenge.
Although incoming traffic to Let's Encrypt's API uses Cloudflare's network, the validation process does not use Cloudflare IP addresses. Allowing outbound traffic to Cloudflare's IP addresses should suffice.
While I am certainly able to do that, I do not consider it security best practices to deliberately connect to networks that are aggressively attacking me.
You could then use a DNS-01 challenge that resolves outside your network
I am indeed using DNS challenges, but that introduction of additional external networking services only makes trying to work around abusive traffic a bigger mess. I'm hoping to find a solution that involves fewer mediators between me and LE.
As noted in my other reply, their entire attacking network is suspect from where I'm sitting. I don't work for Cloudflare, so I can't really say if they're passing through traffic for some customer that has been compromised by a botnet (or whatever), or if they've been compromised themselves as part of a supply chain attack. I'm sure you understand this is especially concerning when they're sitting in the middle of the process of issuing security certificates!
FYI, we do not allow Cloudflare to terminate TLS for us; your connection to the API remains end-to-end encrypted all the way to our own infrastructure. (We’re not using their regular reverse proxy service.)
Traffic from 172.64.0.0/13 and 2400:cb00::/32 should only be proxied traffic destined for hosts behind Cloudflare or replies to such requests. Unsolicited traffic originating from hosts should not be coming from that network. You should probably capture some of the traffic so that you can report the abusive and misconfigured domains to Cloudflare.
Cloudflare operates two main services with different IP ranges:
CDN services, which customers sign up for, and configure their origin connection. The IPs used for both ingress into Cloudflare CDN, and egress to reach origin server are listed in IP Ranges
WARP/Zero Trust, a VPN-like service for protecting corporate end-users (laptops, smartphones, etc). IPs used are not listed in the list above, and they are to be considered untrusted
If you nor any of your customers use CDN services, yet you receive ingress traffic from listed range, you should file abuse report: https://abuse.cloudflare.com/general with details of the perceived attack.
I do not recommend blanket block on firewall. If you must, then I'd recommend only inbound traffic on the affected ports only. Otherwise, given Cloudflare global usage, you risk severely crippling your internet connectivity (e.g. unable to access APIs like Let's Encrypt, or domains you host for the 1.1.1.1 public DNS resolver users).
Is this technically documented somewhere? I didn't see any obvious mention on the LE site for how Cloudflare fits into the overall architecture. I'd be interested to analyze how well "we do not allow" would hold up to a Salt-Typhoon-esque hacker operating inside Cloudflare's network. At the very least, it might point the way towards a good method to only allow connections for the kinds services LE uses. Thanks.
No. I don't work for Cloudflare. Back when I naively spent additional time and effort "reporting" abuse to the abusers, the only action ever taken was them continuing to cash their customer's checks. I'm here hoping to either help improve LE's architecture, or find a minimal-cost way of updating mine to deal with LE choosing to live in a bad network neighborhood.
I do not recommend blanket block on firewall. If you must, then I'd recommend only inbound traffic on the affected ports only. Otherwise, given Cloudflare global usage, you risk severely crippling your internet connectivity (e.g. unable to access APIs like Let's Encrypt, or domains you host for the 1.1.1.1 public DNS resolver users).
That sounds like the "too big to fail" logic that always seems to end in catastrophic failures. My connectivity is improved when I cut off hosts on abusive networks. All ports. In and out. I'd like to think Let's Encrypt can function without being a part of that hostile side of the Internet.
Third party reports indicate over 10% of global internet traffic is routed through Cloudflare, and over 80% of global websites utilize their network.
They are one of the most aggressive actors in mitigating malicious traffic worldwide. Here are some of their stats from last year: Cloudflare 2024 Year in Review
Interpret this information as you will, however to be blunt - your decision and reasons to block Cloudflare seem to be incredibly misguided.
And if for whatever reason you just really hate that Let's Encrypt uses some Cloudflare resources as part of them protecting their own network, feel free to use some other CA. There are a bunch to choose from, many of which are both also free for domain-validation certificates and support ACME.
Interpret this information as you will, however to be blunt - your decision and reasons to block Cloudflare seem to be incredibly misguided.
They are the source of attacks on my servers. No amount of cheerleading for them changes that. If I am "incredibly misguided" for protecting myself from them, we have very different definitions for what constitutes security best practices.
While it is entirely off-topic for this forum, it is incredibly disingenuous to make unsubstantiated claims of attacks on your network while simultaneously refusing to report the alleged attacks to the origin network. It is in even more bad faith to deliberately conflate the network provider with the attacker.
Note that is not a request for proof, as this is not the proper venue for discussing allegations of abusive traffic from Cloudflare. Their abuse form is the right place for that.
I get it. Your network, your rules, but if you are going to wholesale block Cloudflare network traffic, you will need to find a new ACME certificate authority. You will also want to plan for other inconveniences, since
And yet here people are, despite countless historical examples of people and organizations abusing their positions of power and trust, here you are still desperately trying to find ways to punch down at the victims of that abuse. I came here looking for technical solutions, not this.
It is in even more bad faith to deliberately conflate the network provider with the attacker.
That makes no sense. I have zero transparency to any of the infrastructure managed by the owners of the IP ranges that are the source of hostile traffic. The onus is on these cloud providers (Cloudflare is not unique in this respect) to stop shielding abusive customers if they wish to shift the responsibility elsewhere.
Note that is not a request for proof, as it this is not the proper venue for discussing allegations of abusive traffic from Cloudflare. Their abuse form is the right place for that.
Again, no, it is not! There is no "discussion" to be had there, and no expectation that any action will be taken other than burying the incident. Abuse reporting needs to be done in independent, open venues. True, this is probably not the place to discuss it (unless LE wants to talk about reasons to reduce/eliminate their dependence on Cloudflare), but then you need to stop rushing to defend them when there is ample independent evidence they have big abuse problems.
I get it. Your network, your rules, but if you are going to wholesale block Cloudflare network traffic, you will need to find a new ACME certificate authority.
If you don't mind, I'm going to stick around to see if a more reasonable technical solution can be had.