/.well-known/acme-challenge/<challenge> does not make it to the web server; /.well-known/<challenge> does - has anyone seen this before?

I have a peculiar situation: it seems that any request to the path /.well-known/acme-challenge/ gets dropped (never arrives at the target web server). But anything to /.well-known/ works OK. This only happens on HTTP/80. Using HTTPS/443 works correctly (obviously, LE requires HTTP/80, that's not part of my question). The ACME client is home-grown, and not relevant; it works fine in other environments.

For the sake of illustrating the issue, the following command works (all commands are performed using PowerShell 7) :

Invoke-WebRequest -Uri http://example.com/.well-known/test.txt

But the following command fails with an error "Invoke-WebRequest: Unable to read data from the transport connection: An existing connection was forcibly closed by the remote host.."

Invoke-WebRequest -Uri http://example.com/.well-known/acme-challenge/test.txt

Switching to HTTPS works fine:

Invoke-WebRequest -Uri https://example.com/.well-known/acme-challenge/test.txt -SkipCertificateCheck

I first suspected it was the A/V (WebRoot) interfering, but after uninstalling it we still had the problem; and requests to 127.0.0.1 were passed just fine on the machine. So that wasn't the issue.

A proxy could be interfering with inbound requests on HTTP/80, but no one I worked with is aware of any such proxy in this environment. HTTPS/443 could bypass this proxy. We expect port 80 is getting passed straight through to our server from the network firewall and not through a proxy; we can't confirm this because it's managed by a third-party we cannot contact.

My next theory is that it's happening at the perimeter firewall, which we can't access or control. I've never seen a firewall that behaves like that, but I don't know what I don't know.

Has anyone ever seen a network behave this way?

Sincerely,
Befuddled in Colorado Springs

It could be a firewall - recently we've seen that Palo Alto has implemented an acme-protocol type and seems to drop those requests (by default).

It could be a poorly written/executed location (or location-match) statement.

It could be other stuff ...

We would need more information to better help you determine where the problem is likely to be found.
Like: An example URL to work with.

6 Likes

This is almost certainly a Palo Alto firewall. Their default settings started blocking these requests in that sort of way a few months ago.

8 Likes

Hard to say for certain; As the "test" URL doesn't show what user-agent was being used.

For all we know, there is another system (in-line) that is handling all the ACME challenge requests.

5 Likes

Yeah, I agree with Rudy and Peter this is most likely a Palo Alto Networks firewall causing a problem. We have seen several different symptoms from their recent (last few months) changes to their default rules.

The first set of symptoms is shown here:

Later variations of this are to see a response but with a certain set of response headers for requests to your server. But, that is not your case. Another variation is to only happen when using a user-agent that matches the Let's Encrypt servers. Your tests show this is not your variation.

In short, you are suffering from the "Classic" Palo Alto block :slight_smile: (almost certainly)

Look for an Application Rule about "acme protocol" in your firewall.

6 Likes

Well, then your next step is to contact whoever can review the firewall logs and update its' policy.

7 Likes

This looks likely. Thank you!

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.