A lot of postings in the Let's Encrypt forum concern Firewall blocking. There was a long posting about one particular case in October by georgep which, as far as I can discover, has not yet been addressed.
I understand why Let's Encrypt needs to spin up new IPs from time to time and why it's impracticable to publish a list of them. But there is a fundamental problem inherent in clients not knowing the IPs used.
Many web server admins block server IP ranges from /24 to /11 and more. In running a web server this is necessary. In general, and with few exceptions such as genuine search engines, servers do not need to access web sites. The plethora of bots, hacks and injections hitting web sites from servers can easily be stemmed by blocking those IP ranges - broadband-based botnets are a different and partly unresolvable problem.
Let's Encrypt's arbitrary IPs often fall within those blocked ranges. Every few weeks I have to extract IP ranges from the Firewall until the probe works and I get a renewed certificate for one or more of my couple of dozen sites. The process is necessarily hit and miss - the probe's IP may be from a range I've recently blocked or from one blocked years ago.
Using a DNS solution is not an option for many people. Not all DNS administrators allow outside agents to write to their servers - and why should they? It's just one more potential weak spot in an already weak system.
May I propose a solution to this? When a probe is refused access, send an email to the address you hold for that domain saying which IP was refused. This would simplify our task of working through Firewalls on an ad hoc basis.
Yes: because it undermines what LE is trying to do with HTTP validation, which you already know--you've apparently read at least some of the threads. You know the arguments, you know LE's position, so what makes you think this flavor of "tell us what IP will be used to validate) will get any more traction?
I don't see that. It's easy anough to find out what the failed IP is if you really want to; it's just a matter of a lot of logs and a lot of time. All I'm suggesting is a way of saving webmasters and others a LOT of time.
Since we do not know which of the many probe IPs are going to be used for the next probe, what difference would a hint make? I'd even settle for a /16 range of IPs.
Either disable the firewall (at least temporarily during certificate updating);
Impracticable; it would have to be disabled for long periods of time, which defeats the point of a firewall.
Alternatively, use dns-01 validation.
As I said in my posting, not all DNS servers have that option and it's yet another weak, attackable point in a weak system.
Whatever Let's Encrypt does, it is not going to kill the bad guys that easily.
...by doing exactly what LE have repeatedly, and explicitly, said they're not going to do.
If you implement IP-based blocking on port 80, don't expect HTTP validation to work for you--the more IPs you block, the greater the risk that it won't work.
If you implement IP-based blocking on port 443, don't expect TLS/ALPN validation to work for you--the more IPs you block, the greater the risk that it won't work.
If you're unable or unwilling to use DNS service that permits automatic updates (even in a very limited case as with acme-dns), DNS validation isn't likely to be viable for you.
If none of the available validation methods will work, perhaps Let's Encrypt isn't the CA for you.
In other words, if there is no DNS option avaliable (which is common), open your firewall and allow criminals to probe your server. Disabling a firewall is against general security advice which, coming from a security-based company, is not good.
I still do not understand why a simple email bearing the rejected IP's /16 can be a security hazard, especially as the IP can be obtained from suitably-enabled logs.
IMHO, there's always a DNS option available. CNAME challenge redirection works great and there are plenty of both free and paid DNS providers (including self-hosted acme-dns) out there that are well supported by a bunch of ACME clients. Delegate a subdomain to one of those providers and go to town. I'd also argue DNS validation is the most secure because your web server doesn't have to be exposed to the internet at all. It doesn't even have to be running yet if you do cert generation from a central location.
Sure, you have the extra step of needing to go through your company's annoying change control process to create the initial CNAME record in the main domain. But that's the cost of your company's security policy. Security is always a tradeoff with convenience.
There are also other free ACME CAs out there that might have more predictable/known validation hosts such that you wouldn't have to change your firewall rules as often.
One of the main reasons for this policy is that Let's Encrypt wants to enable and encourage automation in certificate renewal. If you whitelist specific IP addresses that are used now, you won't have a solution that works automatically when Let's Encrypt changes those addresses in the future. Let's Encrypt does not want to receive pressure from subscribers not to change validation IP addresses because of people having hard-coded old ones in their firewalls.
Secondarily—and I don't know where Let's Encrypt is going with this right now—there is an idea that validation from many different parts of the Internet will help make it harder for attackers to manipulate Internet routing (or DNS) in order to get certificates that they shouldn't be entitled to. This may in principle also involve changing the validation addresses much more frequently or unpredictably in some way, in order to prevent attackers from being able to target particular capabilities that they can be sure would get them a predictable certificate misissuance ability. (Again, I don't know Let's Encrypt's current thinking on this topic, only some discussions from a few years ago.)
If you think letting the general public connect to port 80 of your services is risky, you could try
only letting the general public connect at moments when you're issuing or renewing a certificate
limiting the general public's access to requesting /.well-known/acme-challenge and nothing else
using one of the other challenge methods to prove your control over the domain, including with a CNAME as @rmbolger mentions (so that you don't even need programmatic access to the same DNS zone where your other DNS records are located!)
rmbolger, schoen - thanks for the responses. I will investigate and implement as best I can. I haven't been able to find too much on the DNS solution: is there an FAQ on this site covering it?
My main problem is: the firewall has been in use for several years and some of the blocked ranges may be utilised by LE. It's tedious trying to find the faulty ones, most of which block all ports (the server was originally for mail only).
Last night I logged the iptables for a failing renewal and found a culprit by comparing times with the LE log. As I said, tedious. And then, before I could release the IP, another renewal attempt was made that succeeded, so a change of probe IP was obviously in play.
Again thanks to all who responded here. Much appreciated.
I'm not sure what exists on the main site beyond the description of the dns-01 challenge. But I have a section in the tutorial for my client on using CNAMEs with DNS challenges. Most of it is not specific to my client so it might fill in the gaps even if you're not using it.
Thanks for that. Looks rather complicated compared to the basic linux certbot and needs quite a bit of preparation, but that sort of documentation is useful and I'll bear it in mind should I need it, although I'd need to adapt it for linux.
Could this and other processes/tools be more obviously documented in the LE FAQs?
@schoen Currently, I don't think the certbot DNS plugins can even handle CNAME DNS redirects unfortunately. So I'm not sure there currently is a guide to write. See for example: https://github.com/certbot/certbot/issues/8281
how about a stateless mode with mock webserver? as it doesn't need to talk to even acme client, you can throw on outside firewall and internal domain won't care about public DNS record.
For the paranoid who only trust things when firewalls are up and blocking (I'm one of those):
Using CNAMEs (as mentioned several times above) is the best way to go.
You can CNAME the a zone that points to an IP that you control.
Your firewall allows only DNS to this internal IP all the time.
You only run the DNS server when you need to validate a challenge request (otherwise it is OFF 99.999%).
[one minute out of every 60 days]
The big attraction of Let's Encrypt, apart from being free, is that it's easy to set up. Run certbot and you have certificates. My OP was to do with firewall problems. I suspect a LOT of LE users do not worry too much about firewalls and in many cases may not even have access to them.
What is needed by the majority of users is something quick and easy to set up. If you have to fiddle with DNS, which a lot of web site owners may not have access to or are not comfortable changing, or set up dummy sites and domains, my guess, based in part on my own lack-of-time/laziness, would be to forget it and find something simpler. Or possibly pay someone else to do it, which defeats the "free" aspect.
At least some extra documentation should be aimed at the non-technical and be couched in very simple terms that will help rather than scare.
The majority of users don't go around blocking all kinds of IP ranges. That is a consious decision. And with that, it's your decision to choose between easily setting up a free certificate or a (IMHO) misguided sense of security.
