It would be beneficial for ISRG to serve a single text file on the ACME endpoints behind their firewall, such as:
http://acme-v02.api.letsencrypt.org/hello.txt
With the contents simply being something like:
hello, world!
This would simplify troubleshooting of connection issues:
An ability to access the file rules out an IP Blocking issue, and suggests there is an issue with the client's software (OpenSSL Version, etc), or possibly a Firewall issue by the client
An inability to access the file suggests there is either an IP Block by LetsEncrypt or a Firewall issue by the Client.
Alternatively, this could be on a separate server/domain that is also behind ISRG's firewall. The purpose of serving a file on Port 80 is to simplify differentiating a blocked IP from other possible causes.
I assume ISRG is blocking ALL traffic from IPs in the firewall rules, not just 443, so this would have a low cost and no impact. If the firewall rules specify 443, then this is obviously a high impact request and not worth pursing.
To be fair, the API isn't intended for web browsers, and we probably don't want the API to even vaguely be available over HTTP for fear of clients relying on the insecure upgrade -- that's why we've never had plain HTTP for it. It's within reason to have HTTP for just the static stuff, I suppose, but everything in the API would need to 404 and not redirect. Imperfect world.
Having the API available over HTTP would also technically violate RFC 8555 section 6, right?
Communications between an ACME client and an ACME server are done over HTTPS, using JSON Web Signature (JWS) [RFC7515] to provide some additional security properties for messages sent from the client to the server. HTTPS provides server authentication and confidentiality. With some ACME-specific extensions, JWS provides authentication of the client's request payloads, anti-replay protection, and integrity for the HTTPS request URL.
I am definitely not advocating for the API to be available via HTTP, just exposing a single path on port 80. Not serving anything, but returning a 4xx, could provide the same. 421 (midirect) might be the most correct, 418 (teapot) would work too.
I think the real problem here isn't http vs. https, but just that there's no visibility to the Cloudflare blocked ip list. So when someone has trouble accessing the Let's Encrypt API, it's hard to know if it's a Cloudflare-level block or something on the client network's end of things, and the only way to check is to bug an lestaff member (which community members try to avoid if they don't need to). If there were some public interface to check if an IP is on block list (maybe some web page with a Captcha-type thing), or something along those lines, it might be helpful. I think the initial suggestion was assuming that it was an easier way for Let's Encrypt to provide a way that would help test some of the possibilities, but perhaps there's a better approach.
The initial "announcement" of adding blocking at the CDN layer wasn't even posted in the "API Announcements" section because the information was going to be "stale fairly soon", and that was a year and a half ago… Hopefully a better approach is already on the to-do list somewhere? (Though I know that to-do lists tend to get longer rather than shorter.)
We removed basically every entry from the Cloudflare IP block earlier this year; it is pretty unlikely to be a CDN block at this point (There's about 25 IP addresses in the list currently).
That's not really possible, or ideal. Firewalls are designed to drop connections at low level (usually within the kernel), and save the overhead of passing the data on to the higher levels for further processing. Usually they don't even know what sort of data is in the traffic.
Cloudflare and some gateway appliances will sometimes process the whole connection and do as you want, but they're operating from different concerns and don't really care about processing the incoming data. There have been some experimental linux kernel modules to do this sort of stuff too.
When it comes to security concerns or abuse (which is how you'll be tossed on a blocked ip list), you really need to drop that connection immediately and not process any data from it - otherwise you'd wasting bandwidth or computational resources.
Essentially, yes. Serving a HTTP file is a low-cost way (IMHO) for people to debug their connection into the network. I do think it's a bit better than offering an ip-block lookup though, because it can potentially surface other issues in the network hops and is better suited to troubleshoot users who are less technically inclined. Assuming the network blocks exist for all traffic on an IP, being able to access a document via HTTP but not HTTPS strongly suggests an issue with the Client's software or network, while not being able to connect to either suggests a block or network issue.
It is possible, NAT could be used to send those HTTP connections "elsewhere" (and not interfere with the production environment).
And in this case, quite ideal; Any response is much better than no response.
The challenge, as I understand it, is that the reason something is on this block list is that it's sending so much traffic that the regular servers are being overwhelmed. So sending replies back requires more servers and more bandwidth to be able to do so, when the whole problem is that too much server capacity and bandwidth was being used. Offloading it from servers owned by Let's Encrypt to servers owned by a CDN can certainly help, but someone is going to end up needing to pay for the server capacity and bandwidth, and for abusive traffic just closing the connection as soon as you can is a lot more cost-efficient.
The “regular” IP block list does return an error, which includes an email address to request unblocking. So that flow works fairly well.
Cloudflare doesn’t have a TLS certificate they could use to serve an error message. So when we have to block at the Cloudflare level, such as from DDoS type behaviour, there’s not really an option to serve something helpful; the traffic just gets dropped. That’s just because of the product we use. Fortunately we haven’t had much of that lately, hence removing basically all the entries from the cloudflare block list.
