Connection reset by peer but URL appears accessible

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
spinnaker.cs.man.ac.uk

I ran this command:
certbot -vvv certonly --webroot --webroot-path /var/www/ --dry-run --debug-challenges

It produced this output:

  Domain: spinnaker.cs.man.ac.uk
  Type:   connection
  Detail: 130.88.193.57: Fetching http://spinnaker.cs.man.ac.uk/.well-known/acme-challenge/PAcqsH4as8Ung2z9TLS-y3OMy_WGd6t4XSfFmsa8xKI: Connection reset by peer

My web server is (include version):
nginx/1.24.0

The operating system my web server runs on is (include version):
Ubuntu 22.04

My hosting provider, if applicable, is:
University of Manchester, UK

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
2.6.0

More Details

When certbot is paused I have tried accessing the URL using curl from home, from Amazon Cloudshell from us east 1 and 2 and us west 1 and 2 and from GitPod (which I think runs on Amazon). All these accesses work without issue.

I did a tcpdump and I can see the reset happening on the server:

07:21:10.142577 IP spinnaker.cs.manchester.ac.uk.http > ec2-18-191-132-112.us-east-2.compute.amazonaws.com.32516: Flags [S.], seq 3116770880, ack 1727408937, win 65160, options [mss 1460,sackOK,TS val 3491115398 ecr 1453352439,nop,wscale 7], length 0
07:21:10.240388 IP ec2-18-191-132-112.us-east-2.compute.amazonaws.com.32516 > spinnaker.cs.manchester.ac.uk.http: Flags [.], ack 1, win 491, options [nop,nop,TS val 1453352537 ecr 3491115398], length 0
07:21:10.240573 IP ec2-18-191-132-112.us-east-2.compute.amazonaws.com.32516 > spinnaker.cs.manchester.ac.uk.http: Flags [R.], seq 1, ack 1, win 491, length 0
07:21:10.596643 IP ec2-35-93-41-167.us-west-2.compute.amazonaws.com.23994 > spinnaker.cs.manchester.ac.uk.http: Flags [S], seq 4013645096, win 62727, options [mss 1460,sackOK,TS val 2515074867 ecr 0,nop,wscale 7], length 0
07:21:10.596691 IP spinnaker.cs.manchester.ac.uk.http > ec2-35-93-41-167.us-west-2.compute.amazonaws.com.23994: Flags [S.], seq 2042638634, ack 4013645097, win 65160, options [mss 1460,sackOK,TS val 1543080663 ecr 2515074867,nop,wscale 7], length 0
07:21:10.663502 IP outbound1h.letsencrypt.org.34066 > spinnaker.cs.manchester.ac.uk.http: Flags [S], seq 1355323246, win 64240, options [mss 1436,sackOK,TS val 3792470763 ecr 0,nop,wscale 7], length 0
07:21:10.663534 IP spinnaker.cs.manchester.ac.uk.http > outbound1h.letsencrypt.org.34066: Flags [S.], seq 1116034860, ack 1355323247, win 65160, options [mss 1460,sackOK,TS val 2541037907 ecr 3792470763,nop,wscale 7], length 0
07:21:10.752885 IP ec2-35-93-41-167.us-west-2.compute.amazonaws.com.23994 > spinnaker.cs.manchester.ac.uk.http: Flags [.], ack 1, win 491, options [nop,nop,TS val 2515075024 ecr 1543080663], length 0
07:21:10.753446 IP ec2-35-93-41-167.us-west-2.compute.amazonaws.com.23994 > spinnaker.cs.manchester.ac.uk.http: Flags [R.], seq 1, ack 1, win 491, length 0
07:21:10.789667 IP outbound1h.letsencrypt.org.34066 > spinnaker.cs.manchester.ac.uk.http: Flags [.], ack 1, win 502, options [nop,nop,TS val 3792470890 ecr 2541037907], length 0
07:21:10.790218 IP outbound1h.letsencrypt.org.34066 > spinnaker.cs.manchester.ac.uk.http: Flags [R.], seq 1, ack 1, win 502, length 0

I am not too sure if the R here means that the reset happens somewhere upstream from the server. A tcpdump of a curl from us-west 2:

07:31:41.394042 IP ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700 > spinnaker.cs.manchester.ac.uk.http: Flags [S], seq 1440220121, win 26883, options [mss 1460,sackOK,TS val 3664079574 ecr 0,nop,wscale 7], length 0
07:31:41.394084 IP spinnaker.cs.manchester.ac.uk.http > ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700: Flags [S.], seq 2893390907, ack 1440220122, win 65160, options [mss 1460,sackOK,TS val 3400581894 ecr 3664079574,nop,wscale 7], length 0
07:31:41.553784 IP ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700 > spinnaker.cs.manchester.ac.uk.http: Flags [.], ack 1, win 211, options [nop,nop,TS val 3664079735 ecr 3400581894], length 0
07:31:41.554203 IP ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700 > spinnaker.cs.manchester.ac.uk.http: Flags [P.], seq 1:123, ack 1, win 211, options [nop,nop,TS val 3664079735 ecr 3400581894], length 122: HTTP: HEAD /.well-known/acme-challenge/test.html HTTP/1.1
07:31:41.554232 IP spinnaker.cs.manchester.ac.uk.http > ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700: Flags [.], ack 123, win 509, options [nop,nop,TS val 3400582054 ecr 3664079735], length 0
07:31:41.554417 IP spinnaker.cs.manchester.ac.uk.http > ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700: Flags [P.], seq 1:237, ack 123, win 509, options [nop,nop,TS val 3400582054 ecr 3664079735], length 236: HTTP: HTTP/1.1 200 OK
07:31:41.713975 IP ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700 > spinnaker.cs.manchester.ac.uk.http: Flags [.], ack 237, win 219, options [nop,nop,TS val 3664079895 ecr 3400582054], length 0
07:31:41.713975 IP ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700 > spinnaker.cs.manchester.ac.uk.http: Flags [F.], seq 123, ack 237, win 219, options [nop,nop,TS val 3664079895 ecr 3400582054], length 0
07:31:41.714049 IP spinnaker.cs.manchester.ac.uk.http > ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700: Flags [F.], seq 237, ack 124, win 509, options [nop,nop,TS val 3400582214 ecr 3664079895], length 0
07:31:41.873844 IP ec2-34-222-120-162.us-west-2.compute.amazonaws.com.58700 > spinnaker.cs.manchester.ac.uk.http: Flags [.], ack 238, win 219, options [nop,nop,TS val 3664080055 ecr 3400582214], length 0

I would be grateful for any advice on helping to work out what is going wrong here!

2

I should add that the server is constantly running and there is a test.html file at:
http://spinnaker.cs.man.ac.uk/.well-known/acme-challenge/test.html

3

Do you have some kind of geographical firewall set up blocking access outside of e.g. Europe?

From The Netherlands I can reach your test file fine, but e.g. Let's Debug (Let's Debug, probably using some location in the VS I recon) also sees the Connection reset by peer error.

1 Like
4

Maybe there's a Palo Alto firewall in front of it? There is definitely user-agent filtering going on.

With Let's Encrypt in the user-agent:

root@letsdebug:~# curl -i -H "User-Agent: Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"  http://spinnaker.cs.man.ac.uk/.well-known/acme-challenge/PAcqsH4as8Ung2z9TLS-y3OMy_WGd6t4XSfFmsa8xKI
curl: (56) Recv failure: Connection reset by peer

Changing it to Let's Not Encrypt it succeeds:

root@letsdebug:~# curl -i -H "User-Agent: Mozilla/5.0 (compatible; Let's Debug emulating Let's Not Encrypt validation server; +https://letsdebug.net)"  http://spinnaker.cs.man.ac.uk/.well-known/acme-challenge/PAcqsH4as8Ung2z9TLS-y3OMy_WGd6t4XSfFmsa8xKI
HTTP/1.1 404 Not Found
Server: nginx/1.24.0
Date: Mon, 24 Jul 2023 09:35:47 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.24.0</center>
</body>
</html>

Changing it back fails again:

root@letsdebug:~# curl -i -H "User-Agent: Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"  http://spinnaker.cs.man.ac.uk/.well-known/acme-challenge/PAcqsH4as8Ung2z9TLS-y3OMy_WGd6t4XSfFmsa8xKI
curl: (56) Recv failure: Connection reset by peer

For more info see Palo Alto firewall users with failing HTTP-01 challenges: enable "acme-protocol".

8 Likes
5

Excellent information - I will pass this on to our IT department!

5 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.