The error log has no updates to it when I attempt to renew the cert or access wiki.ceas.wmich.edu/.well-known/acme-challenge. The access log when I attempt to connect to wiki.ceas.wmich.edu/.well-known/acme-challenge doesn't show any errors, just the GET requests.
I feel like I have run grep on every apache config file I know of for /.well-known and/or acme-challenge and have found nothing, unless of course there is config files I do not know of. Which ones would you suggest I look into to verify I have done everything?
It might even be hidden in .htaccess files with Apache, so that's rather difficult to say specifically where to look..
Can you perhaps share parts of the access_log entries showing the GET requests?
3 Likes
This is all the access log writes when I try to access it:
141.218.149.252 - - [07/Apr/2022:10:10:23 -0700] "GET /.well-known/acme-challenge/test HTTP/1.1" 404 709 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36"
gives me 404 in browser
And I'm getting a connection refused.
Especially if your webserver is on the same network as your own computer you used for this most recent test and that is working, but requests from the internet are not, this looks more and more like there is some kind of firewall in between your webserver and the internet at large. Is that maybe the case?
2 Likes
I have tried completely disabling ufw and that doesn't seem to work. Any other kind of WAF I can disable?
Check with the university IT department. There may be firewalls for all traffic inbound to the univ.
1 Like
As a last resort, you might be able to issue a certificate using the dns-01 plugin, however, it might be difficult to get access to the DNS zone in a university setting..
1 Like
Bruh I work in the university IT department. We got 5 CS majors working on this with no success.
ok, let's go for overkill:
grep -ir acme /etc/apache2
grep -ir Location /etc/apache2
grep -ir well-known /etc/apache2
2 Likes
grep -ir acme /etc/apache2 NO RESULTS
grep -ir Location /etc/apache2:
/etc/apache2/apache2.conf:# ErrorLog: The location of the error log file.
/etc/apache2/conf-available/localized-error-pages.conf:# SetHan dler directive in a context somewhere. Adding
/etc/apache2/conf-available/localized-error-pages.conf:# the fo llowing three lines AFTER the context should
/etc/apache2/conf-available/localized-error-pages.conf:# <Locat ion /error/>
/etc/apache2/conf-available/localized-error-pages.conf:# </Loca tion>
/etc/apache2/mods-available/status.conf: <Location /server-status>
/etc/apache2/mods-available/status.conf:
/etc/apache2/mods-available/actions.conf:# Format: Action media/type /cgi-script /location
/etc/apache2/mods-available/actions.conf:# Format: Action handler-name /cgi-scri pt/location
/etc/apache2/mods-available/ldap.conf:<Location /ldap-status>
/etc/apache2/mods-available/ldap.conf:
/etc/apache2/mods-available/proxy_html.conf:# at top level, but can also be used in a .
/etc/apache2/mods-available/proxy_html.conf:# <Location /my-gateway/>
/etc/apache2/mods-available/proxy_html.conf:#
/etc/apache2/mods-available/info.conf: <Location /server-info>
/etc/apache2/mods-available/info.conf:
/etc/apache2/mods-available/proxy_balancer.conf: # <Location /balancer- manager>
/etc/apache2/mods-available/proxy_balancer.conf: #
/etc/apache2/envvars:# temporary state file location. This might be changed to / run in Wheezy+1
/etc/apache2/apache2.conf.in:# ErrorLog: The location of the error log file.
grep -ir well-known /etc/apache2: NO RESULTS
Is apache listening directly on that IP?
Do you have mod_security installed?
1 Like
I do not have mod security and I am honestly not sure how to check if it is directly listening but I want to say yes. How would I check and what do I need mod security for?
If you had it, might have caused this issue.
1 Like
Then do you have any other networking gear between your server's UFW and the public internet?
You have a long history with Let's Encrypt certs. You were renewing certs every 60 days just like a well-functioning server should.
Then, something changed after the latest cert you got Jan8 2022. Based on prior pattern you would not have tried until early March and that failed so something in that time period.
3 Likes
We had a cronjob set up top run an auto renew for forever and it worked great. Then it suddenly changed. The only thing we changed is updating the server from ubuntu 16.04 to 18.04. That is the only thing we touched. So I tried reverting the server back to 16.04 from a snapshot from February and we still can't get it to renew, automatically or manually. Not sure what changed.
Is there any configurable equipment between your server and the ISP connection? Like a router or firewall appliance from a vendor? If so, was anything swapped out or upgraded?
Have you checked the security settings in them?
2 Likes
Nothing like that at all.
Makes no sense to me. Here is a sequence of requests I just did (2:16pm Central)
curl -I http://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
curl: (56) Recv failure: Connection reset by peer
curl -I http://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
curl: (56) Recv failure: Connection reset by peer
curl -I http://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
curl: (56) Recv failure: Connection reset by peer
(note the MM in the path of the next 2)
curl -I http://wiki.ceas.wmich.edu/.MMwell-known/acme-challenge/ForumTest
HTTP/1.1 301 Moved Permanently
Date: Thu, 07 Apr 2022 19:15:48 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://wiki.ceas.wmich.edu/.MMwell-known/acme-challenge/ForumTest
Content-Type: text/html; charset=iso-8859-1
curl -I http://wiki.ceas.wmich.edu/.MMwell-known/acme-challenge/ForumTest
HTTP/1.1 301 Moved Permanently
Date: Thu, 07 Apr 2022 19:15:51 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://wiki.ceas.wmich.edu/.MMwell-known/acme-challenge/ForumTest
Content-Type: text/html; charset=iso-8859-1
(no more MM)
curl -I http://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
curl: (7) Failed to connect to wiki.ceas.wmich.edu port 80 after 19 ms: Connection refused
(waited 2 minutes and then:)
curl -I http://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
curl: (56) Recv failure: Connection reset by peer
(2:23pm central, oddly https gets 404 as expected, only http gets 'reset by peer')
curl -I https://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
HTTP/1.1 404 Not Found
Date: Thu, 07 Apr 2022 19:23:21 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
You should see all of these in your apache error or access logs. From earlier tests you did not see any request resulting in "reset by peer". If that is still the case then something is blocking those requests before Apache sees them.
In these cases I just follow the request from the ISP connection to your server and look for anything in the way.
2 Likes
Just a note we have been deleting the cert and reverting our system several times. We can't even delete the cert and just get a new one. We ALWAYS get a "reset by peer" warning. We do not have a firewall on or anything between us and the ISP.
I am consistently now getting a 404 from
curl -I https://wiki.ceas.wmich.edu/.well-known/acme-challenge/ForumTest
and my phone gives me NOT FOUND when connecting to it. Phone is on mobile data.
Deleting the cert has no effect on getting another one.
Try HTTP. As I showed just previous HTTP get "reset" failure but HTTPS get 404
You may need to explicitly state http or better is to use curl or similar from outside your university network
Once you start getting 404 from HTTP requests to the .well-known/acme-challenge folder then Let's Encrypt server requests should start working too.
UPDATE:
So, your server is plugged right into a plain modem? And, that modem has no configuration whatsoever?
2 Likes