I have a peculiar situation: it seems that any request to the path /.well-known/acme-challenge/ gets dropped (never arrives at the target web server). But anything to /.well-known/ works OK. This only happens on HTTP/80. Using HTTPS/443 works correctly (obviously, LE requires HTTP/80, that's not part of my question). The ACME client is home-grown, and not relevant; it works fine in other environments.
For the sake of illustrating the issue, the following command works (all commands are performed using PowerShell 7) :
But the following command fails with an error "Invoke-WebRequest: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.."
I first suspected it was the A/V (WebRoot) interfering, but after uninstalling it we still had the problem; and requests to 127.0.0.1 were passed just fine on the machine. So that wasn't the issue.
A proxy could be interfering with inbound requests on HTTP/80, but no one I worked with is aware of any such proxy in this environment. HTTPS/443 could bypass this proxy. We expect port 80 is getting passed straight through to our server from the network firewall and not through a proxy; we can't confirm this because it's managed by a third-party we cannot contact.
My next theory is that it's happening at the perimeter firewall, which we can't access or control. I've never seen a firewall that behaves like that, but I don't know what I don't know.
Yeah, I agree with Rudy and Peter this is most likely a Palo Alto Networks firewall causing a problem. We have seen several different symptoms from their recent (last few months) changes to their default rules.
The first set of symptoms is shown here:
Later variations of this are to see a response but with a certain set of response headers for requests to your server. But, that is not your case. Another variation is to only happen when using a user-agent that matches the Let's Encrypt servers. Your tests show this is not your variation.
In short, you are suffering from the "Classic" Palo Alto block (almost certainly)
Look for an Application Rule about "acme protocol" in your firewall.