The problem: at the moment to renew, I have to open port 80 to a wide variety of IPs - I try not to open it to the world, but EFF/Certbot seems to have greatly widened the possible IPs that the authorization check might come from.
The solution: I would like certbot-auto to get a short list of possible IPs that might be used to authorize, feed them to my --pre-hook routine, and then I can open my firewall only to that list, for a short period, say 5 minutes.
I realize this might take a little effort, but it should be possible and the benefits would be huge.
This is not a good practice.
You need to allow HTTP and control it accordingly.
This type of request has been made many times here before.
It has not changed and should not be changed.
Publishing specific source IPs is a very bad thing that can lead to even bigger problems.
The "benefits" are only for a few and specific clients and are far outweighed by the risks associated.
Again, you need to understand how to control HTTP requests.
They need not reach your production (at risk) systems [use separate and secured HTTP only system].
The HTTP system only needs to allow the /.well-known/acme-challenge/ requests to pass.
All other requests can be redirected to HTTPS [with one single line of code].
I can see my request produced a strong reaction, but I would like to suggest the response was in error. My request is highly reasonable, and obviously a lot of people want it. But why is it a bad idea? The objection seems to be to “publishing” the letsencrypt server list. But I don’t want that. I want to be told in my pre- script what IPs are about to come in with requests so I can briefly open my firewall. I don’t need the whole list, and it doesn’t have to be precisely minimal - I don’t care if there are a few extras. I’m about to be notified of those IPs anyway, when they show up in my firewall logs as denied, so at that point (milliseconds or seconds later) they are not going to be “secret”. So the IPs are no more and no less published either way. This is an incredibly reasonable approach.
The other, sadly inevitable, objection is “why do you want to do that”. Ok, I’ll bite - the reasons not to do the DNS approach are several. Without going through every detail, I’ll just cite the letsencrypt documentation itself which points out that this is much harder. It is. It would be easier if the zone involved was a dynamic one, but of the 400+ domains here, that particular one is not dynamic, so making adjustments is a lot more messy and error prone.
Anyway, this is a very reasonable approach and doesn’t change the security landscape or anything else; it just makes it easier for me to anticipate and permit the incoming verification steps.
So if I got this right, you have a set of domains, not accessible from outside (or only accessible to a limited set of IPs), for which you would like to have certificates issued using HTTP verification, but without changing your firewall rules. I would say that it is a rather specific scenario and perhaps you might consider something that might help you without any changes on LE side?
Considering that LE requests are rather specific, and target the files in just one location, you could open port 80 on your firewall to all and configure access on your web-server level to only allow access to LE verification files location for any requests not coming from the set of IPs you trust?
Alternatively, if you do not want your web server to be reachable via port 80 at all, and you are using a split DNS, you could have your external IP for the domains needing verification pointing to a specific box implementing appropriate access control and proxying the requests if needed.
That would require some effort, but it might be more effective than waiting for when something you are requesting gets implemented (or rejected) on LE side
No this is not what I am asking at all. You have it partly right - I have many domains for which the web server is not public, so I have to open all firewalls, including the one on the web server to allow access to the web-server during verification. I have done all that and it is working, but with a slight kludge. I have by experience collected a set of nets from looking at rejections in the firewall logs, so in order to limit exposure I open all the various firewalls only to those nets and not the entire world and I do so for 5 minutes. What I would prefer, and what I view as far less of a hack, is that the the certbot hand my pre hook a few IPs from which set the current about-to-happen verification will come in. I can manage opening what I need to, both in the various routers and the webserver for a window of time.
This would be cleaner and safer than for that window of time than opening things up to the world.
I used to generate a list of IPs myself because in the “old days” I could cycle through the DNS names outbound[123].letsencrypt.org and I had observed that this was where verification would come from. Alas nowadays the tests can come from a variety of outside sources, including any of a large number of IPs in the Amazon cloud. That particular source (Amazon) is where a pretty large amount of the net attacks come from, so it’s become more important than ever to be able to distinguish between letsencrypt traffic and unwanted traffic. I am not just afraid of attempts to compromise various web-servers, but all there is a pretty constant attempt to harvest information that is supposed to be internal. Limiting exposure would be easier if I could distinguish more readily.
A small set of IPs to open firewall(s) to for that one session would be most welcome, and make life easier and safer.
Not public in what sense - are they only accessible internally (apart from the interval of time when you had to open them for the verification) or they are accessible externally as well but only to certain IPs?
Hand it how? Simply put, the IPs from which verification might be requested may change. How do you see certbot getting that information - via some additional API call to LE?
Also, as I mentioned above, if you are running quite a few sites/domains in a complex configuration, you might consider those options I listed - for example by having a specific box in front of your otherwise firewalled sites, which would perform necessary checks for what is requested and, as long as it is a request to a verification file location, forward it further. Have you tried to implement something like that?
Yes, you have the sense of internal website correctly. They are normally accessible only internally and from fairly extensive private networks. Note that some of the certificates are used for other purposes, such as email, but since those servers have an internal website, it has worked out well to do the renewals through that mechanism.
What would be handy would be for the pre-hook to be passed a parameter that lists a few IPs from which the requests would come in. I don’t mind if I’m handed something that tells me how to go get the info myself, but the parameter seems better, and then the certbot-auto program can handle the mechanism for getting this one-time use short list.
I do NOT need a permanent list, or a comprehensive list, or a list I can use going forward for any length of time. The pre-hook is called only when a renewal is needed and just before - so that is perfect.
I understand your suggestion that I redirect outside traffic to a different server and it’s an interesting one. However I feel that this is pretty messy and I can easily imagine that the certbot program local to that redirected server might now or in the future catch this as an attempt to do something undesirable. For instance certbot would notice that I am trying to renew a certificate for a machine that is not this one. This could be really thorny.
I'm confused.
Certbot should only need run on the system that is requesting the cert, the one that will use the cert.
The "middle-man" only needs to filter out those unwanted HTTP requests (99.99%).
Allowing only those which are looking for /.well-known/acme-challnge/ path to pass (via proxy).
So the one single "middle-man" proxy can front-end hundreds, thousands, of back-end systems.
Protecting them all from unwanted HTTP access.
I feel that adding a proxy or proxies for many many servers is an unnecessary level of difficulty and I feel is a really extreme solution. @rg305 I know that you feel strongly opposed to what I am asking but what I am suggesting seems very straightforward to me and by far the most elegant way to solve this rather common problem. certbot-auto already has to go out to letsencrypt servers and declare it’s intent to update certificates and then wait for the new certificate. Having the servers pass back a short list of what IPs might be used in this one upcoming authentication process and passing that list as parameters to my pre-hook is simple and elegant. The proxy infrastructure is a whole later of infrastructure just to meet a very simple and reasonable need. The simplest solutions are the best. This facility should have been built in in the first place. It’s a common need. I stand by my feature request.
You are repeating yourself.
And I believe we all understood it the first time you stated your position.
We can argue the merits until we turn blue in the face.
Your position is a lot more complicated than you make it out to be.
Very few, if any, systems could implement such an instant “turn-on” / “turn-off” IP list.
IMHO, the benefit doesn’t outweigh the complexity.
[and don’t get me started on the security aspect of it]
I could see the ability to know what IP addresses validation will come from before validation is completed being some form of attack vector in the future. Possibly if some core internet routers become compromised through some OS/Firmware vulnerability.
I spend a lot of time with network security and I do not see an attack vector. There are two kinds of vulnerability. One is, can someone do ill with the information about IP numbers? I argue that the short list is being revealed to me anyway. I just want it a few seconds earlier. The second is whether someone could disrupt my certificate procurement and substitute something else secretly. I would argue that we already rely on the connection from certbot to letsencrypt to be secure and immune against MITM attacks - otherwise we are lost anyway. As long as that connection is secure and encrypted and we’ve made sure we’re protected against unwanted “proxies” (i.e. MITM), then that secure TCP connection should be usable for this too.
If you understand why they don’t publish a list, then you can see how this request goes against that.
[You should understand how sending that list via a certbot request is only one step from publishing such a list. A very simple step that any IP on the planet can take would equal a published list]
If you don’t understand why they don’t publish a list, then I can see why you still insist on this debate.
Maybe you should first open a topic to ask that question directly.
[or, better yet, just visit some of the other such topics that have been opened over the years]
Do you really even think there would be a simple global way to implement opening and closing firewall rules to allow any list of IPs temporary HTTP access?
[the potential user base is close to non-existent and the practical actual user count would be just you]
I don't believe this to be the case. Even if the connection between Certbot and the ACME CA was subject to an active MITM (or just straight up plain HTTP), the protocol is designed to be immune to replay/relay/modification attacks (in terms of mis-issuance anyway). This is thanks to the use of JWS and embedding the JWK thumbprint in the key authorizations.
You are certainly right that the obscuring of IP numbers is not a real hurdle for a determined attacker.
I believe this is more about discouraging the practice of whitelisting so that Let's Encrypt does not end up in a situation where there are millions of deployments using whitelisting, leaving Let's Encrypt completely unable to change their validation practices due to the massive incident it would cause.
We already know this happens on a minor scale because there have been a lot of help threads posted here when AWS was introduced into the validation server pool.
That flexibility to change their validation practices has to be protected in order to be able to deal with attacks like this and any other research that will appear in the future.
FWIW, the people who develop Certbot and the people who run Let's Encrypt, are completely different sets of people in different organizations.
The one thing I will say in favor of something resembling your proposal is that having a list of observed addresses would make troubleshooting/helping people much easier. I wrote a program at the end of last year (va-monitor) which is an HTTP server + dnstap sink, creates orders occasionally, and logs what addresses it sees for each validation method and ACME directory. But I hesitate to publish anything because of the risk that people will just copy-paste crap into iptables.
You may wish to publish just the code itself but not a service or result of a run. That way only people with sufficient expertise will be able to use it. Then, those knowledgeable people must understand the pitfalls relying on it.
Both @rg305 and @_az already made some good points regarding why this might be not such a good idea and that its implementation might be not as simple as it may seem. However, looking at that as an interesting engineering exercise, I’d like to reiterate the point that you do not really need LE to do what you want to do.
Just to recap - you have full control over your firewall rules and you know when renewal process starts. Even if you do not want to try setting up some sort of DMZ, you could still do this. In essence, all you need, if we take a simple example of one host, is to configure your web server to allow access to everything for your internal network and allow access to only verification files location for external IPs, while keeping your firewall up (so no external connections to your web server are possible, or only possible from the IPs you trust).
Once you start your renewal script, you change firewall rules to allow external access to port 80 (and if you redirect to HTTPS, then 443 too) of your web server(s) - the configuration of the web server will handle the rest. Once the process is finished, you switch that access off again on your firewall. Makes sense?
Let's start with the obvious.
[I have been known to master that subject]
If I read your request correctly, you would have your firewall trust an ACME client to open and close ports to any IPs it deems required at will.
ACME clients are written by third parties.
Yes they may be open source.
But they also update themselves at will.
That means (to me) that code that is subject to change (which I can't control nor vet) can make changes to my firewall policy anytime it wants.
I find that incredibly risky and prone to exploitation.
If you don't see an attack vector... are you looking?
Or are you blinded by your needs and too proud to see your solution as it really is?
