Well I kind of have my suspicions. firewall, so this is easy to fix right - wrong.
I think in this world where cyber attacks and the like have increased significantly that it is time for the letsencrypt initiative to provide a little more transparency on the IP ranges it uses - yes I've read all the arguments - no longer valid these days where we are all striving to secure our borders from bad actors that hide in geographic regions to avoid any kind of prosecution or retribution.
I block a number of countries outright, Russia for one, Seychelles aand a few other nations that host or facilitate those that habitually attempt to hack and probe weaknesses and decline to do anything about it. I don't need a lecture on how I'm reducing access to my systems and only hurting myself, that is my choice, those countries aren't part of our customer base anyhow.
So in order to manage my border security I need to know where things are coming from - and if this process is using dubious locations for whatever reason then that becomes a concern - I realise it may not be within the initiatives power to control but it should be possible to figure out ranges at least.
It is going to take a lot of work to identify exactly which one of our blocks is causing this problem, this has only recently started to happen. Without this range information whitelisting this process will be a challenge as I have so many trying to probe and penetrate it is hard to single out the sources to good vs bad.
It sounds like your only option is to use the DNS-01 challenge, as LetsEncrypt have continually stated they will not publish the IP Addresses of validation servers.
We don’t publish a list because it changes somewhat frequently, but the rough shape of what we use isn’t exactly a secret. At this time, all requests come from three sources:
I appreciate those hints on the IP's, that's a good place to start. I don't mind if they change at all as long as they don't drop into the banned ranges. I will likely move to DNS but that has thrown its own challenges.
I tried DNS on one of the boxes and it worked first time but the second time it threw an error about plugin compatibility, the only way to use DNS seems to be in manual mode, anything else triggers an error, going entirely to a manual mode would be royal pain in the neck so I'm exploring it. I've read a few posts that say the only way to use DNS once you've been using HTTP is to uninstall Certbot totally and start again.
The challenge is that you get a bot notice that a cert is about to expire before you figure something isn't working, then you check the other boxes and they aren't either and this applies a fair bit of pressure to resolve before the expiry occurs. I dropped the firewall into 'in line' mode bypassing rules and the renew worked so for sure I think an IP in our banned ranges is in use. IPV6 has been hit and miss so I really haven't taken much time to set it all up although we do have an IPV6 address I haven't explored how to route/NAT things internally.
I actually haven't seen these IP's in my tests, at least not since you started using Cloudflare Magic transit, though I haven't tested the last few days.
The Cloudflare magic transit IP's I've seen all resolved to somewhere in this range:
Though there are apparently no IPv6 addresses either.
Relying on this is bad practice of course. Allow-listing these is certainly going to break at some point. DNS challenge seems to be the best option indeed.
I shall continue to explore the DNS option as I do have total control over that also and it seems the option most repeatable.
All I need to do is get it to work reliably on Ubuntu 22.04 (it currently does not) ..... when I do I will post the 'how' and any gotcha's if it is of interest of course.