Renewal or new issue behind a firewall fails due to LE operation

We got one, quite a while ago: Let's Encrypt Fails due to ModSec rule In that case, our own on-premise validation servers were listed, so this kind of problem isn't limited to just AWS.

The cause was sinkholing: our validation server connected to a formerly malicious site that had since been taken over by a blocklist operator. Their systems assume that if an IP address is connecting to a sinkhole, it's infected by malware (or is itself doing malicious scanning). That assumption has a pretty low rate of false positives, but it's bad for us, because we validate a lot of sites - and anyone can ask us to validate any site.

The upshot is that any validation server we spin up, previously clean IP or not, has a good chance of mistakenly being added to blocklists. I've tried to follow up with list operators to get exemptions, and have had no success.

Because of this, if you're blocking traffic based on feeds of suspicious IP addresses, I recommend making an exemption for traffic that's destined for your DNS servers. That should be a pretty simple exemption, exposing the narrowest possible attack surface, and would allow you to use dns-01 challenges successfully.

One followup question, @georgep - am I reading your post correctly that you had this problem when requesting certificates with Certbot, but not with other clients? If that's correct, then something else is going on! The validation from our side happens the same way regardless of which ACME client is used.