Connection reset error getting cert

[jonathan-g]

After other ways of trying to renew my certificate failed, I ran certbot --manual certonly -d alo.ees.vanderbilt.edu to get a certificate. I verified using a web browser from a different domain that I could retrieve the challenge from the URL (I copied and pasted from a terminal SSH session). I did this from home (off-site) and also using a VPN to make it appear I was connecting from another state. I also checked that this works when I put the IP address for the server in the URL instead of the hostname.

My web browser gets the challenge correctly, but certbot reports connection reset. I tried Let's Debug and it reported everything good until it tried to retrieve http://alo.ees.vanderbilt.edu/.well-known/acme-challenge/letsdebug-test, when it got a connection reset. When I try the same URL in my browser (from off-site at home, and also over a VPN so it looks like I'm coming from another state) I get a 404 for letsdebug-test.

I would be grateful for any help in figuring out why I can't renew my certificate with certbot, even though I can retrieve challenges using a web browser (Firefox).

My server log /var/log/nginx/error.log has this error in it:

2022/09/11 00:17:15 [debug] 3433#3433: *87 accept: 23.178.112.203:35304 fd:21 2022/09/11 00:17:15 [debug] 3433#3433: *87 event timer add: 21: 60000:15993779 2022/09/11 00:17:15 [debug] 3433#3433: *87 reusable connection: 1 2022/09/11 00:17:15 [debug] 3433#3433: *87 epoll add event: fd:21 op:1 ev:80002001 2022/09/11 00:17:15 [debug] 3433#3433: *87 http wait request handler 2022/09/11 00:17:15 [debug] 3433#3433: *87 malloc: 000055C565D90660:1024 2022/09/11 00:17:15 [debug] 3433#3433: *87 recv: eof:1, avail:-1 2022/09/11 00:17:15 [debug] 3433#3433: *87 recv: fd:21 -1 of 1024 2022/09/11 00:17:15 [info] 3433#3433: *87 recv() failed (104: Connection reset by peer) while waiting for request, client: 23.178.112.203, server: 0.0.0.0:80 2022/09/11 00:17:15 [debug] 3433#3433: *87 close http connection: 21

My domain is: alo.ees.vanderbilt.edu

I ran this command: sudo certbot --manual certonly -d alo.ees.vanderbilt.edu

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Certificate is due for renewal, auto-renewing...
Renewing an existing certificate for alo.ees.vanderbilt.edu
Performing the following challenges:
http-01 challenge for alo.ees.vanderbilt.edu

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Create a file containing just this data:

MbwqEPNJQCjICP545pdQn8Nhz6_HisIjprjnwhT0h5I.1T99SLifjs8m0ex1mzRuLoOTT_vk7ctVqiM74Fovg0c

And make it available on your web server at this URL:

http://alo.ees.vanderbilt.edu/.well-known/acme-challenge/MbwqEPNJQCjICP545pdQn8Nhz6_HisIjprjnwhT0h5I

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Press Enter to Continue
Waiting for verification...
Challenge failed for domain alo.ees.vanderbilt.edu
http-01 challenge for alo.ees.vanderbilt.edu

Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
  Domain: alo.ees.vanderbilt.edu
  Type:   connection
  Detail: 129.59.80.49: Fetching http://alo.ees.vanderbilt.edu/.well-known/acme-challenge/MbwqEPNJQCjICP545pdQn8Nhz6_HisIjprjnwhT0h5I: Connection reset by peer

Hint: The Certificate Authority failed to verify the manually created challenge files. Ensure that you created these in the correct location.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): nginx 1.18.0

The operating system my web server runs on is (include version): Ubuntu 20.04.5 (LTS)

My hosting provider, if applicable, is: self-hosted

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.30.0

[MikeMcQ]

Welcome to the community @jonathan-g

You are another one affected by a Palo Alto Networks brand firewall. They changed a default setting in their gear earlier this year and we have seen this regularly.

You should talk to your network admins and have them change the Application Rule for "ACME protocol".

As a test, the below request should reply with a 404 Not Found. Instead, it gets the "reset by peer" error. In your case, you must use the -A as shown as that is the user-agent used by Let's Encrypt servers. You can use this sample curl and URL to test the setting was changed properly

curl -I alo.ees.vanderbilt.edu/.well-known/acme-challenge/SampleToken -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
curl: (56) Recv failure: Connection reset by peer

Should get this, even with that -A value, instead

curl -I alo.ees.vanderbilt.edu/.well-known/acme-challenge/SampleToken
HTTP/1.1 404 Not Found
Server: nginx
Date: Sun, 11 Sep 2022 00:50:02 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive

[jonathan-g]

Thanks. When I use the -A value, I get connection reset, and without the -A, I get a 404, so it sounds like your diagnosis is correct. I'll contact my network admins.