Webroot challenge failing all of a sudden (IP issue) // Solution: A and AAAA need to point to the same server!

kghbln · June 2, 2018, 1:51pm

For a about one and a half years the webroot challenge was working for me fine. Now it is broken and I have no clue to get it working. I spent many hours again with several things browsed via the internet.

My setup for https://www.mediawikiwidgets.org/

<VirtualHost *:80>
ServerAdmin mail@example.info
ServerName www.mediawikiwidgets.org
ServerAlias mediawikiwidgets.org

RedirectPermanent / https://www.mediawikiwidgets.org/

ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel warn

CustomLog "|/usr/bin/rotatelogs -l ${APACHE_LOG_DIR}/%Y%m%d.access.log 86400" combined
</VirtualHost>

and

<VirtualHost *:443>
ServerAdmin karsten@hoffmeyer.info
ServerName www.mediawikiwidgets.org
ServerAlias mediawikiwidgets.org

DocumentRoot /path/to/root/

<IfModule mod_rewrite.c>
	RewriteEngine on

	RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-f
	RewriteCond %{DOCUMENT_ROOT}%{REQUEST_URI} !-d
	RewriteRule ^(.*)$ %{DOCUMENT_ROOT}/w/index.php [L]
</IfModule>


ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel warn

CustomLog "|/usr/bin/rotatelogs -l ${APACHE_LOG_DIR}/%Y%m%d.access.log 86400" combined
</VirtualHost>

This has worked fine. Now I was forced to require login - not authentication.

With authentication I would do for http:

<VirtualHost *:80>
ServerAdmin mail@example.info
ServerName www.mediawikiwidgets.org
ServerAlias mediawikiwidgets.org

<Directory /path/to/root>
	<IfModule mod_rewrite.c>
		RewriteEngine on
		RewriteBase /
		RewriteCond %{REQUEST_URI} !^.well-known/acme-challenge
		RewriteRule ^(.*) https://www.mediawikiwidgets.org/$1 [R=301,L]
	</IfModule>
</Directory>

ErrorLog ${APACHE_LOG_DIR}/error.log

LogLevel warn

CustomLog "|/usr/bin/rotatelogs -l ${APACHE_LOG_DIR}/%Y%m%d.access.log 86400" combined
</VirtualHost>

This works fine for authentification but it does not for registration:

With e.g. certbot certonly --webroot -w /path/to/root -d www.mediawikiwidgets.org --dry-run I get:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.mediawikiwidgets.org
Using the webroot path /path/to/root for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.mediawikiwidgets.org (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.mediawikiwidgets.org/.well-known/acme-challenge/PxS4t_GEM9Oj4fElEDL0dwa9NHUa1srhiwDj-PqOZ5w: "<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/D"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.mediawikiwidgets.org
   Type:   unauthorized
   Detail: Invalid response from
   http://www.mediawikiwidgets.org/.well-known/acme-challenge/PxS4t_GEM9Oj4fElEDL0dwa9NHUa1srhiwDj-PqOZ5w:
   "<?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/D"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

Apparently I have no idea how to mitigate. Perhaps there are some people around who may help me with my misery. Moreover I am utterly sorry to bother you with something like this but I personally am out of luck. Sadly the DNS challenge is no option for me.

Osiris · June 2, 2018, 1:56pm

I’m getting a 404 file not found error when I try to reach your challenge file. Which isn’t surprisingly, as certbot already removed it after cleaning up

Could you put a test file in /path/to/root/.well-known/acme-challenge/ ?

Also, your HTTP to HTTPS is completely broken when I removed the whole acme-challenge and well-known part of the URL. I’m guessing it’s because when you’re not having a DocumentRoot in your HTTP VirtualHost, but you are specifying a specific <Directory> directive, the whole Rewrite part isn’t called upon.

kghbln · June 2, 2018, 2:02pm

I added a file: Login required - MediaWiki Widgets

I’m guessing it’s because when you’re not having a DocumentRoot in your HTTP VirtualHost, but you are specifying a specific directive, the whole Rewrite part isn’t called upon.

No I showed the whole config file. Perhaps this is the issue. I am wondering why this is working for other domains like this.

Osiris · June 2, 2018, 2:03pm

Now I'm getting a redirect to HTTPS, which wasn't there before. Did you change anything?

kghbln · June 2, 2018, 2:05pm

No but I would like to add a document root to 80

Osiris · June 2, 2018, 2:07pm

Well, a few minutes ago, when I surfed to the HTTP site http://www.mediawikiwidgets.org/, I got a "Everything works" kind of example page of your webserver in German, not the redirect to HTTPS ór your main site through HTTP. And now I'm getting the redirect. So you did change something

kghbln · June 2, 2018, 2:12pm

No and you can trust me on this one. However when testing different solutions it got this too in arbitrary occasions. What makes me crazy is that the page you are referring to is at /var/www/html/ whereas default.conf and default-ssl.conf specifiy /var/www/htdocs/. This is indeed driving me crazy too.

kghbln · June 2, 2018, 2:23pm

The error certbot provides suggests that it is getting /var/www/html/index.html. I do not know why since Apache is configured to go for /var/www/htdocs/…

kghbln · June 2, 2018, 2:25pm

I just killed this directory since it is unused.

kghbln · June 2, 2018, 2:33pm

No idea what is going on. The setup worked for one and a half years and it is still working for the other websites on the same server. I am clueless - no kidding.

kghbln · June 2, 2018, 2:36pm

Or, let*s ask the other way around: What is the expected output of the challenge? Currently I am getting a wiki page as in https://www.mediawikiwidgets.org/.well-known/acme-challenge/ma49FKu-0WHYgor876loNogOXQI6UtASncNqUQcIISo: Should a file or so be served? So if a file should be served I need something that prevents the redirect to the wiki page.

Osiris · June 2, 2018, 2:42pm

Yes, certbot places a file with a token in it on exactly the location which is requested in the logs.

kghbln · June 2, 2018, 2:46pm

Ok. So what to do? If I place a file it works like https://www.mediawikiwidgets.org/.well-known/acme-challenge/challenge.txt Moreover I wonder why it worked for one and a half years and now starts failing. The only difference is that I require user to log in. In the meantime I believe that this does not make a difference.

Osiris · June 2, 2018, 2:48pm

Try renewing again now that the test file works

Also, in a certbot above you only specified -d www.mediawikiwidgets.org, NOT also -d mediawikiwidgets.org, which is required if you also want the base domain to be accessible through HTTPS.

kghbln · June 2, 2018, 2:49pm

Sadly it does not. This is driving me crazy. I spent many hours on this and now I am also wasting your time.

Osiris · June 2, 2018, 2:51pm

Try the --manual plugin for testing purposes. The webroot plugin removes the challenge file again, which makes debugging sometimes a little bit more difficult.

Also, what was the log from certbot this time?

kghbln · June 2, 2018, 2:51pm

I know. I just added www for demonstration purposes. Currently I do not get further to the actual cert.

jmorahan · June 2, 2018, 2:54pm

It works via IPv4 but not IPv6. Make sure your AAAA record is pointing to the correct IPv6 address.

Osiris · June 2, 2018, 2:57pm

That's very strange, my curl does confirm this, but my Google Chrome gets the file without a hitch..? Going debugging my own workstation now

jmorahan · June 2, 2018, 2:58pm

HTTP returns a 404, but HTTPS doesn’t connect so maybe Chrome is failing back to IPv4? It won’t try HTTP because HSTS.