Hi Rudy - I suspect we have both been around the Internet (BIX? Usenet? Arpanet?) long enough to understand that communication too often turns into unintended miscommunication.
You paraphrased well. LE needs to either (a) put its new AWS servers to use to take some innocuous load for a month or two (eg; take some of the simple website load) until the assigned IP becomes clean or (b) go through the process of contacting all the DNS BL holders saying "Hey, we're the EFF - please delete IP xxx.xxx.xxx.xxx from your list".
(a) is easier than (b) - even if the IP was just left dormant and not doing anything "naughty" for a while it should eventually fall off the blocklists. I understand that EFF is a donation funded organisation and that money is always tight but - step back - the cost of a tiny AWS instance (that issues an IP address) pales against the value of just this one user's time spent on this nonsense and as soon as the IP is clean the AWS instance can be scaled to whatever size preserving the IP and put into production causing nobody harm.
The important thing is to ensure that only servers with squeaky clean IP addresses are put into production as acme-challenge servers.
I don't think my situation/environment is that uncommon. OK - I'm an old white-haired IT guy who has the chops to run a server farm behind a firewall that uses DNS blacklists to minimise the amount of crud that hits the systems.
But one of your early comments was that (my turn to paraphrase) change happens ... which in this example takes the form of a small business (the precise target market I suspect EFF is aiming certbot at) signing up for a virtual or real rack server on-line and ticking the box that asks "Do you want a firewall with that?"
Viewed technically, the only difference then between that environment and the one I operate is a slight difference in scale. Oh, and the fact that I can at least open up my firewall when I need to - where the innocent small business person has absolutely zero control over the (probably functionally so similar as to be considered identical) hosting company firewall.
So, what are their options? Give up on HTTPS altogether (unlikely as if they want any kind of SEO placement let alone e-commerce these days HTTPS is de-rigeur) or go hunting for other ACME solutions -> which probably involves some expensive "professional" help -> by which time this story is already far away from the EFF's original intent to make use of HTTPS simple and cheap.
As you say, the solution is far from mind-troubling. As I say, it needs to be done and ASAP.
I'm happy to write a quick article about the issue and linking back to here - what subject / topic area would you suggest?
- flippant?
I asked "what changed?" as a serious question because I wanted to understand what had altered in the design / implementation / operation of LE and certbot so that I could decide whether to adjust my systems and persist with it or, myself, head off to look at ACME alternatives.
Recall my opening description that I have four servers operating numerous certificates that have been happily running and updating LE almost since Le's inception - and several of the websites (including a couple comprising a few million lines of PHP that I wrote on top of a somewhat (English understatement!) complex database that haven't seen a config change in over a decade - other than the initial implementation of LE - and now none of them will renew nor even when stripped down to basics install afresh certificates using certbot/LE and you may begin to understand my eagerness to understand just what was going on.
The response you gave was - how can I describe it when I am English, you are American, two nations forever separated by a common language? "Jocular"? Jokey"? "Off-hand"? I have many American friends but we all know each other well so converse in a kind of mid-Atlantic jargon and know each other well enough to simply ask for explanation when something is not clear.
Anyway - I was hoping for a serious answer to my serious question that would help me push the penny up the hill.
I got a 'jokey' response that took me nowhere.
Understand that this was against a background of having spent much of the past month tearing certbot and my servers' logs apart ... line by line ... with absolutely zero starting understanding or available documentation covering the way that certbot is designed to work and how it is implemented operationally. Also, as my certificates entered the expiry / renewal period one-by-one and not all at once this problem started as a small camp-fire then turned into a raging forest fire as the certificate on my email server actually expired yesterday causing the loss of all access to email (and a queue of incoming emails that will only arrive over the coming days) and you might understand why "flippant" - cf; https://www.thefreedictionary.com/flippant to read the first definition as "marked by inappropriate levity; frivolous or offhand". No offence is meant or taken from use of "flippant" in English use and, if you read the above explanation, you will I think understand it is an appropriate word.
Let me know where I should best suggest whoever releases LE / certbot into the wild that they pay a bit more intention to both the project's mission statement and think through the impact of decisions on (according to the LE website just now) even 1% of 225 million LE certificate holders and I'll give it my best shot.
Regards from Bordeaux, SW France 