How to diagnose if certbot is being blocked by an application firewall?

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: ctf.sonoma.edu

I ran this command:
# certbot certonly --nginx -d ctf.sonoma.edu --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Simulating a certificate request for ctf.sonoma.edu
Performing the following challenges:
http-01 challenge for ctf.sonoma.edu
Waiting for verification...
Challenge failed for domain ctf.sonoma.edu
http-01 challenge for ctf.sonoma.edu
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: ctf.sonoma.edu
   Type:   unauthorized
   Detail: 130.157.3.120: Invalid response from
   http://ctf.sonoma.edu/.well-known/acme-challenge/ksx8wFC_BqUX_UJMrU9PbLJsntpJvnSi_-oobidZcHg:
   503

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I ran this command:

# systemctl stop nginx
# certbot certonly --standalone -d ctf.sonoma.edu --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org
Simulating a certificate request for ctf.sonoma.edu
Performing the following challenges:
http-01 challenge for ctf.sonoma.edu
Waiting for verification...
----------------------------------------
Exception happened during processing of request from ('::ffff:54.212.17.137', 47978, 0, 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/site-packages/acme/standalone.py", line 208, in __init__
    BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
  File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/lib/python2.7/site-packages/acme/standalone.py", line 217, in handle
    BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
  File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.7/BaseHTTPServer.py", line 310, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
error: [Errno 104] Connection reset by peer
----------------------------------------
----------------------------------------
Exception happened during processing of request from ('::ffff:66.133.109.36', 15125, 0, 0)
Traceback (most recent call last):
  File "/usr/lib64/python2.7/SocketServer.py", line 295, in _handle_request_noblock
    self.process_request(request, client_address)
  File "/usr/lib64/python2.7/SocketServer.py", line 321, in process_request
    self.finish_request(request, client_address)
  File "/usr/lib64/python2.7/SocketServer.py", line 334, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "/usr/lib/python2.7/site-packages/acme/standalone.py", line 208, in __init__
    BaseHTTPServer.BaseHTTPRequestHandler.__init__(self, *args, **kwargs)
  File "/usr/lib64/python2.7/SocketServer.py", line 649, in __init__
    self.handle()
  File "/usr/lib/python2.7/site-packages/acme/standalone.py", line 217, in handle
    BaseHTTPServer.BaseHTTPRequestHandler.handle(self)
  File "/usr/lib64/python2.7/BaseHTTPServer.py", line 340, in handle
    self.handle_one_request()
  File "/usr/lib64/python2.7/BaseHTTPServer.py", line 310, in handle_one_request
    self.raw_requestline = self.rfile.readline(65537)
  File "/usr/lib64/python2.7/socket.py", line 476, in readline
    data = self._sock.recv(self._rbufsize)
error: [Errno 104] Connection reset by peer
----------------------------------------
Challenge failed for domain ctf.sonoma.edu
http-01 challenge for ctf.sonoma.edu
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: ctf.sonoma.edu
   Type:   unauthorized
   Detail: 130.157.3.120: Invalid response from
   http://ctf.sonoma.edu/.well-known/acme-challenge/lcMnBGvGLP0uK9tAlpplWi_YeWbujB7WxWK3dVyfsM4:
   503

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):

# nginx -v
nginx version: nginx/1.20.1

The operating system my web server runs on is (include version):

# uname -a
Linux ctf.sonoma.edu 3.10.0-1160.59.1.el7.x86_64 #1 SMP Wed Feb 23 16:47:03 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

# certbot --version
certbot 1.11.0

Hi @mgondree,

There was a recent change to Palo Alto firewall products that causes many validation requests to fail with an error similar to this one. Can you find out from your network administrators if a Palo Alto firewall is in use on your network? The solution people found for that is to explicitly allow acme-protocol in the firewall.

4 Likes

Thanks, @schoen -- I will follow-up with that question.

1 Like

Edit: to more directly answer your question, I would suggest looking at server logs or especially the output from running a packet sniffer like tcpdump to see whether what your server sends is the same thing Let's Encrypt receives. In the section "The following errors were reported by the server", you can see a summary of what the certificate authority saw when trying to connect to your server; if that doesn't match what your server thinks it transmitted, then a firewall is typically to blame.

3 Likes

Agreed. And, this problem is similar but not identical. With the cases so far the comms failure was "reset by peer" for URL paths including the acme challenge path. As shown in this thread.

This case shows an http 503 failure instead of "reset by peer". Maybe Palo Alto firewall has changed how they block such a request but it could be something else. Still, looks clearly like some sort of firewall or even server config blocking these specific kinds of requests.

(parts removed from responses for brevity)

curl -I ctf.sonoma.edu/.well-known/acme-challenge
HTTP/1.1 301 Moved Permanently
Server: nginx/1.20.1
Location: http://ctf.sonoma.edu/ctf/.well-known/acme-challenge

curl -I ctf.sonoma.edu/.well-known/acme-challenge/
HTTP/1.1 503 Service Unavailable
P3P: CP="CAO PSA OUR"

curl -I ctf.sonoma.edu/.well-known/acme-challenge/ForumTest
HTTP/1.1 503 Service Unavailable
P3P: CP="CAO PSA OUR"
3 Likes

If there is no other HTTP proxy involved, then we may need to have a look at the HTTP server block that is handling the challenge requests.

1 Like

The http response for 503's do not include any 'server: nginx' header. And, the 301 redirect did not include any P3P header. @mgondree also stopped nginx and used --standalone which got same 503.

I think the P3P header is important clue. Should help find what device in front of their nginx server rejected the request.

I found this post over at Palo Alto Networks. Not the exact symptom but it rejected requests with a 503 and has the same P3P response header. As noted this 503 is different than the others we've seen for "reset by peer" but this seems good place to focus attention.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.