Webroot challenge failing all of a sudden (IP issue) // Solution: A and AAAA need to point to the same server!

kghbln · June 2, 2018, 3:00pm

A complete disaster:

certbot certonly --manual -w /var/www/htdocs/mw/02120 -d www.mediawikiwidgets.org -d mediawikiwidgets.org --dry-runSaving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for www.mediawikiwidgets.org
http-01 challenge for mediawikiwidgets.org

-------------------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
-------------------------------------------------------------------------------
(Y)es/(N)o: y

-------------------------------------------------------------------------------
Make sure your web server displays the following content at
http://www.mediawikiwidgets.org/.well-known/acme-challenge/Vo-KOGpn5T4tv9pZJMSfORYmQrGSVVlytLcOfRJVA7c before continuing:

Vo-KOGpn5T4tv9pZJMSfORYmQrGSVVlytLcOfRJVA7c.GYoUP2kb9PLDQ5pztmYlfzsbKyc5SQadoXnpAAlLtOs

If you don't have HTTP server configured, you can run the following
command on the target server (as root):

mkdir -p /tmp/certbot/public_html/.well-known/acme-challenge
cd /tmp/certbot/public_html
printf "%s" Vo-KOGpn5T4tv9pZJMSfORYmQrGSVVlytLcOfRJVA7c.GYoUP2kb9PLDQ5pztmYlfzsbKyc5SQadoXnpAAlLtOs > .well-known/acme-challenge/Vo-KOGpn5T4tv9pZJMSfORYmQrGSVVlytLcOfRJVA7c
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
-------------------------------------------------------------------------------
Press Enter to Continue

-------------------------------------------------------------------------------
Make sure your web server displays the following content at
http://mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g before continuing:

BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g.GYoUP2kb9PLDQ5pztmYlfzsbKyc5SQadoXnpAAlLtOs

If you don't have HTTP server configured, you can run the following
command on the target server (as root):

mkdir -p /tmp/certbot/public_html/.well-known/acme-challenge
cd /tmp/certbot/public_html
printf "%s" BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g.GYoUP2kb9PLDQ5pztmYlfzsbKyc5SQadoXnpAAlLtOs > .well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g
# run only once per server:
$(command -v python2 || command -v python2.7 || command -v python2.6) -c \
"import BaseHTTPServer, SimpleHTTPServer; \
s = BaseHTTPServer.HTTPServer(('', 80), SimpleHTTPServer.SimpleHTTPRequestHandler); \
s.serve_forever()" 
-------------------------------------------------------------------------------
Press Enter to Continue
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. mediawikiwidgets.org (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g: "<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/D", www.mediawikiwidgets.org (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.mediawikiwidgets.org/.well-known/acme-challenge/Vo-KOGpn5T4tv9pZJMSfORYmQrGSVVlytLcOfRJVA7c: "<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/D"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: mediawikiwidgets.org
   Type:   unauthorized
   Detail: Invalid response from
   http://mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g:
   "<?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/D"

   Domain: www.mediawikiwidgets.org
   Type:   unauthorized
   Detail: Invalid response from
   http://www.mediawikiwidgets.org/.well-known/acme-challenge/Vo-KOGpn5T4tv9pZJMSfORYmQrGSVVlytLcOfRJVA7c:
   "<?xml version="1.0" encoding="UTF-8"?>
   <!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
     "http://www.w3.org/TR/xhtml1/D"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

I will check. Still I am wondering why it worked for one and a half year and starts to fail now.

Osiris · June 2, 2018, 3:02pm

My Chrome also caches everything, I don't see anything in Wireshark when I go to the HTTP link.. I've got to stop debugging with a browser

kghbln · June 2, 2018, 3:02pm

Indeed the AAAA recored held the wrong IP. I just updated with TTL of 5 minutes. Let's see where we go from there.

Osiris · June 2, 2018, 3:04pm

The problem is IPv6 all right:

osiris@desktop ~ $ curl -Lv6 http://www.mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g
*   Trying 2a01:488:42:1000:50ed:8477:ec:78e...
* Connected to www.mediawikiwidgets.org (2a01:488:42:1000:50ed:8477:ec:78e) port 80 (#0)
> GET /.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g HTTP/1.1
> Host: www.mediawikiwidgets.org
> User-Agent: curl/7.49.0
> Accept: */*
> 
< HTTP/1.1 404 Not Found
< Date: Sat, 02 Jun 2018 15:03:12 GMT
< Content-Type: text/html; charset=utf-8
< Transfer-Encoding: chunked
< Connection: keep-alive
< Server: Apache
< Vary: accept-language,accept-charset
< Accept-Ranges: bytes
< Content-Language: en
< Expires: Sat, 02 Jun 2018 15:03:12 GMT
< 
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Strict//EN"
  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd">
<html xmlns="http://www.w3.org/1999/xhtml" lang="en" xml:lang="en">
<head>
<title>Object not found!</title>
<link rev="made" href="mailto:webmaster@mediawikiwidgets.org" />
<style type="text/css"><!--/*--><![CDATA[/*><!--*/ 
    body { color: #000000; background-color: #FFFFFF; }
    a:link { color: #0000CC; }
    p, address {margin-left: 3em;}
    span {font-size: smaller;}
/*]]>*/--></style>
</head>

<body>
<h1>Object not found!</h1>
<p>


    The requested URL was not found on this server.

  

    If you entered the URL manually please check your
    spelling and try again.

  

</p>
<p>
If you think this is a server error, please contact
the <a href="mailto:webmaster@mediawikiwidgets.org">webmaster</a>.

</p>

<h2>Error 404</h2>
<address>
  <a href="/">www.mediawikiwidgets.org</a><br />
  <span>Apache</span>
</address>
</body>
</html>

* Connection #0 to host www.mediawikiwidgets.org left intact
osiris@desktop ~ $ curl -Lv4 http://www.mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g
*   Trying 217.197.83.171...
* Connected to www.mediawikiwidgets.org (217.197.83.171) port 80 (#0)
> GET /.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g HTTP/1.1
> Host: www.mediawikiwidgets.org
> User-Agent: curl/7.49.0
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Date: Sat, 02 Jun 2018 15:03:15 GMT
< Server: Apache/2.4.25 (Debian)
< Location: https://www.mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g
< Content-Length: 401
< Content-Type: text/html; charset=iso-8859-1
< 
* Ignoring the response-body
* Connection #0 to host www.mediawikiwidgets.org left intact
* Issue another request to this URL: 'https://www.mediawikiwidgets.org/.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g'
*   Trying 217.197.83.171...
* Connected to www.mediawikiwidgets.org (217.197.83.171) port 443 (#1)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*   CAfile: /etc/ssl/certs/ca-certificates.crt
  CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use http/1.1
* Server certificate:
*  subject: CN=www.mediawikiwidgets.org
*  start date: Jun  2 06:57:52 2018 GMT
*  expire date: Aug 31 06:57:52 2018 GMT
*  subjectAltName: host "www.mediawikiwidgets.org" matched cert's "www.mediawikiwidgets.org"
*  issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
*  SSL certificate verify ok.
> GET /.well-known/acme-challenge/BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g HTTP/1.1
> Host: www.mediawikiwidgets.org
> User-Agent: curl/7.49.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sat, 02 Jun 2018 15:03:16 GMT
< Server: Apache/2.4.25 (Debian)
< Strict-Transport-Security: max-age=31536000
< Last-Modified: Sat, 02 Jun 2018 14:56:15 GMT
< Accept-Ranges: bytes
< Content-Length: 88
< Cache-Control: max-age=2592000
< Expires: Mon, 02 Jul 2018 15:03:16 GMT
< 
BEjwOG0KGkkX_SVNa2CsfppzNap1AOwZzk_pu-poO9g.GYoUP2kb9PLDQ5pztmYlfzsbKyc5SQadoXnpAAlLtOs
* Connection #1 to host www.mediawikiwidgets.org left intact
osiris@desktop ~ $

Good one @jmorahan

jmorahan · June 2, 2018, 3:06pm

I think (though I'm not sure and maybe someone can confirm this) if the IPv6 fails to connect at all, the validation falls back to IPv4, so maybe the other server on the IPv6 address only recently started responding on port 80?

kghbln · June 2, 2018, 3:11pm

Heureka!!! After rectifying the AAAA record things are in fluff again and I am no longer grumpy.

@jmorahan You rock

@Osiris You rock too.

So many cool people here. I am amazed.

kghbln · June 2, 2018, 3:19pm

So it first checks if IPv6 is available and tries to connect. If not available it checks via IPv4. This is interesting to know.

jmorahan · June 2, 2018, 3:22pm

Yes, that’s correct. I wasn’t sure before but now I’ve found the thread where I remembered reading it: Improved timeout errors from Boulder

kghbln · June 2, 2018, 3:25pm

Indeed, that explains my situation. I would never have come up with this though I usually make sure that A and AAAA point to the same server. I guess a senior moment made me not do this in this case.

system · July 2, 2018, 3:26pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot 1.21 webroot HTTP 500, 1.20 works Client dev	12	2260	December 11, 2021
Renew with apache plugin, challenges fail Server	9	1374	January 26, 2019
Getting around apache configs with 443 redirects Help	3	983	March 3, 2017
Failed authorization Help	3	886	March 9, 2019
Challenge failed Help	3	534	November 15, 2019

Webroot challenge failing all of a sudden (IP issue) // Solution: A and AAAA need to point to the same server!

Related topics