Renewal is failing with acme challenge 404

rg305 · April 15, 2022, 8:40pm

I think the problem is within Apache
[but I always think that]

Please show the ENTIRE output of:
sudo apachectl -t -D DUMP_VHOSTS

bpattan · April 15, 2022, 8:42pm

root@ip-172-31-44-194:/etc/apache2/sites-available# sudo apachectl -t -D DUMP_VHOSTS
VirtualHost configuration:
*:443 admin.wizpms.com (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)
*:80 ip-172-31-44-194.us-east-2.compute.internal (/etc/apache2/sites-enabled/000-default.conf:1)
root@ip-172-31-44-194:/etc/apache2/sites-available#

rg305 · April 15, 2022, 8:44pm

hmm...

Please show:
ls -l /var/www.html/.well-known/acme-challenge/

bpattan · April 15, 2022, 8:45pm

root@ip-172-31-44-194:/etc/apache2/sites-available# ls -l /var/www.html/.well-known/acme-challenge/
total 4
-rw-r--r-- 1 root root 5 Apr 15 17:47 Test_File-1234
root@ip-172-31-44-194:/etc/apache2/sites-available#

rg305 · April 15, 2022, 8:46pm

OK we scroll it all the way back and...
begin at the begining!

Please show the output of:
curl -4 ifconfig.co

bpattan · April 15, 2022, 8:47pm

root@ip-172-31-44-194:/etc/apache2/sites-available# curl -4 ifconfig.co
52.14.105.212
root@ip-172-31-44-194:/etc/apache2/sites-available#

rg305 · April 15, 2022, 8:50pm

OK, that's a match:

Name:    admin.wizpms.com
Address: 52.14.105.212

now...
I would suggest we handle the challenge requests in HTTP (rather than redirecting them to HTTPS).
For that, please replace file: /etc/apache2/sites-enabled/000-default.conf
with the following:

<VirtualHost *:80>
  ServerAdmin webmaster@localhost
  DocumentRoot /var/www/html
  ErrorLog ${APACHE_LOG_DIR}/error.log
  CustomLog ${APACHE_LOG_DIR}/access.log combined
  #skip challenge requests and
  <LocationMatch "^/(?!\.well-known)">
    #send all other requests to HTTPS
    RewriteEngine On
    RewriteCond %{HTTPS} !=on
    RewriteRule ^/?(.*) https://%{SERVER_NAME}/$1
  </LocationMatch>
</VirtualHost>

Then restart Apache

bpattan · April 15, 2022, 8:55pm

Made changes, and restarted.

rg305 · April 15, 2022, 8:55pm

Please show the tail of the Apache error.log and access.log files.

I'm getting:

curl -Ii http://admin.wizpms.com/.well-known/acme-challenge/Test_File-1234
HTTP/1.1 404 Not Found
Date: Fri, 15 Apr 2022 20:56:06 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1

bpattan · April 15, 2022, 9:04pm

access log

17.121.204.209 - - [15/Apr/2022:20:58:21 +0000] "GET /.well-known/acme-challenge/W6Mw80GXwXMhsBuvAJ85Livxe3HAtqGnSvN2j6agA5w HTTP/1.1" 404 498 "-" "AppleNewsBot"
17.121.204.209 - - [15/Apr/2022:20:58:21 +0000] "GET /.well-known/acme-challenge/g_p6YJuOKhIFFkpeiE9Luypy25Zf2iDBLFZ5l2ly4ro HTTP/1.1" 404 5169 "-" "AppleNewsBot"
17.121.204.209 - - [15/Apr/2022:20:58:21 +0000] "GET /.well-known/acme-challenge/gIewdfL4m6tkbKt7QZY8CCRQcUoV6GxEZfy4TSkPfPI HTTP/1.1" 404 493 "-" "AppleNewsBot"
17.121.204.209 - - [15/Apr/2022:20:58:22 +0000] "GET /.well-known/acme-challenge/aDBI0e-v8Hbv9AzzO90Rtg2QOmilsXhWC-jeJhhZLAM HTTP/1.1" 404 495 "-" "AppleNewsBot"
17.121.204.209 - - [15/Apr/2022:20:58:22 +0000] "GET /.well-known/acme-challenge/nA19Sbz_7a2sr4jG2X_5pBkmQlLBqjND62h9iT6Q608 HTTP/1.1" 404 494 "-" "AppleNewsBot"
17.121.204.209 - - [15/Apr/2022:20:58:22 +0000] "GET /.well-known/acme-challenge/m5gQGgLSPrzV-gizndXORSViinyY16v7inpu5Ta1zBA HTTP/1.1" 404 495 "-" "AppleNewsBot"
17.121.204.209 - - [15/Apr/2022:20:58:22 +0000] "GET /.well-known/acme-challenge/7WJ9ebplKAwM-ZYvLk8hRM9tewGZxiF5wUdVirlVoD8 HTTP/1.1" 404 5166 "-" "AppleNewsBot"
45.155.204.20 - - [15/Apr/2022:20:59:22 +0000] "GET /js/chunk-78a5aa55.e794d1c0.js HTTP/1.1" 200 7180 "-" "osee2unifiedRelease/1466 osee2unifiedReleaseVersion/6.9.1 Mozilla/5.0 (iPhone; CPU iPhone OS 12_4 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Mobile/15E148"
52.14.105.212 - - [15/Apr/2022:20:59:55 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 140 "-" "curl/7.58.0"

Nothing in the error log

rg305 · April 15, 2022, 9:08pm

That is very strange!
The IP shown is the IP of the server.
As if there is some local proxy... to itself.

Or did you try that curl request from the server itself?

bpattan · April 15, 2022, 9:10pm

Yes. I did run the curl from the same server

9peppe · April 15, 2022, 9:10pm

What's this fetching those for?

rg305 · April 15, 2022, 9:10pm

OK that explains it.
Show:
grep Test_File-1234 access.log

rg305 · April 15, 2022, 9:11pm

because it fetches every link posted in discord.

bpattan · April 15, 2022, 9:12pm

root@ip-172-31-44-194:/var/log/apache2# grep Test_File-1234 access.log

70.226.121.235 - - [15/Apr/2022:17:47:37 +0000] "GET /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 5399 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
18.211.147.95 - - [15/Apr/2022:17:59:45 +0000] "GET /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 659 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
35.172.241.34 - - [15/Apr/2022:18:00:07 +0000] "GET /.well-known/acme-challenge/Test_File-1234%60 HTTP/1.1" 301 661 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/92.0.4515.159 Safari/537.36"
38.205.191.197 - - [15/Apr/2022:18:00:10 +0000] "GET /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 603 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
38.205.191.215 - - [15/Apr/2022:18:00:19 +0000] "GET /.well-known/acme-challenge/Test_File-1234%60 HTTP/1.1" 301 605 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.84 Safari/537.36"
185.178.95.254 - - [15/Apr/2022:18:02:05 +0000] "GET /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 659 "-" "Mozilla/5.0 (Linux; Android 11; M2007J20CG) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.127 Mobile Safari/537.36"
75.51.0.154 - - [15/Apr/2022:20:43:51 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 226 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:43:56 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 5059 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:52:06 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 226 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:52:13 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 226 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:52:33 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 226 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:52:40 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 226 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:52:49 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 301 226 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:54:47 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 140 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:56:06 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 140 "-" "curl/7.58.0"
75.51.0.154 - - [15/Apr/2022:20:56:48 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 140 "-" "curl/7.58.0"
52.14.105.212 - - [15/Apr/2022:20:59:55 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 140 "-" "curl/7.58.0"
52.14.105.212 - - [15/Apr/2022:21:02:17 +0000] "HEAD /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 140 "-" "curl/7.58.0"
70.226.121.235 - - [15/Apr/2022:21:03:03 +0000] "GET /.well-known/acme-challenge/Test_File-1234 HTTP/1.1" 404 495 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"
70.226.121.235 - - [15/Apr/2022:21:03:04 +0000] "GET /favicon.ico HTTP/1.1" 302 588 "http://admin.wizpms.com/.well-known/acme-challenge/Test_File-1234" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/99.0.4844.51 Safari/537.36"

9peppe · April 15, 2022, 9:13pm

That's not it. 17.121.204.209 belongs to Apple, Inc.

The Discourse fetcher uses IPs belonging to them explicitly. Maybe it works differently on iOS.

rg305 · April 15, 2022, 9:14pm

What don't you get?

Apple fetches every link posted in discord.

9peppe · April 15, 2022, 9:15pm

I get that Apple is fetching.

I don't get why it does that, or what Discourse has to do with it.

rg305 · April 15, 2022, 9:16pm

Discourse has nothing to do with Apple querying public information from a public site.