That is sad (especially in light of the amount of fruitless debugging work that it then imposes on other people downstream, like the original poster).
I would suggest that in cases of weird network behavior it can be helpful to run a packet sniffer like Wireshark (especially as here, where a technical organization/department is trying to determine whether the error is in its own configuration or some other organization's configuration). This can make more clear whether what's seen by the outside world matches what the server is doing or not. If it doesn't, then it means that some other device in between is interfering.
A long time ago we were investigating network interference by ISPs and we found it instructive to run a packet sniffer at both ends of the connection, and then compare the traffic as seen by both ends. This can reveal if one end apparently receives something from the other end that the other end doesn't recall having sent—which would then mean that some other device in between actually sent that.
In this case, you could run a packet sniffer and a web client (whether a browser or curl) on a non-university network connection, and also run a packet sniffer on your server. The client will receive a disconnection of some sort from the server, and you can then check whether that corresponds to something that the server believes it sent, or not.
This is a kind of advanced technique, but, again, a useful one when a technically-oriented organization wants to pin down where in the network an error is coming from.
Totally agree @9peppe. That was a clever test (post #61) and is compelling evidence the acme challenges are blocked somewhere other than within poster's area.
It is too bad the site www.wmich.edu uses a cert from GlobalSign otherwise it would also soon be affected.
You mean the tests you made from devices within the univ network? As noted in earlier posts this actually further indicates the firewall impacting you is at the outer edge(s) of the university network connection.
Multiple Let's Encrypt servers will make challenges from various points around the globe and they must all succeed for a successful HTTP challenge. It does not matter that the machines running certbot (the acme client) can connect to that path, it is the LE servers that must be able to connect.
They're also sending an HSTS header with includeSubdomains, and I don't think they know what that implies for any website they do not control directly but under the same domain.
UPDATE: Finally found a buried apache setting that was redirecting http to https. If we run curl on http://wiki.ceas.wmich.edu/.well-known/acme-challenge it not longer gives us a 301, gives us a 404 as we wanted. STILL getting a "connection reset by peer" error
Checked with the university Office of Information Technology and we are not blocking anything with our firewall. We tracked the packets with Wireshark. It seems our server receives the response from LetsEncrypt and sends nothing back. Not sure what to make of this.
So sorry. Forgot to mention I tried this as well. Did not work. Pretty much all our servers have stopped renewing for an unknown reason and some of them use nginx for LetsEncrypt so I feel like it has to be some sort of firewall or some other higher up setting we are not recognizing
Not literally that error, but usually the Connection reset by peer error is caused by a "RST" package from the host to the client. If I run WireShark on my host and do a curl, I'm getting the following result:
That's because only adding the slash on the end results in the Connection reset by peer, without the slash it's NOT giving any error. Which is the whole reason why everybody here suspects a firewall somwehere in between.
The curl output is:
HTTP/1.1 404 Not Found
Date: Mon, 11 Apr 2022 19:21:39 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
I am running this from a windows machine. It IS on the same network as the server. Curling outside the network may change that. I can check later today when I have access to an non-University computer.
If you're getting a 404 not found result (when you indeed include the slash [or more] after acme-challenge) from within the same network as the server, but get a connection reset by peer from outside the network, you'll know where to look: somewhere in between your network and the location where curl gets the connection reset by peer.
