VestaCP: Invalid HTTP-01 challenge response, XHTML document

Help
#1

Hi,

I’ve started to get random errors with my auto-renews:

Error: Invalid response from http://cdn.steampunkjunkies.com/.well-known/acme-challenge/-tuN8Fn96y1pClaOOlIAliFEIjoeRX2HqfTw1mN2eQQ: \

This is run using VestaCP, with:

sudo /usr/local/vesta/bin/v-update-letsencrypt-ssl

This works perfectly on most of them, but for some reason this one is causing me an issue now. If I go to that page, it works fine. It seems to work fine with a test file here with both ipv4 ipv6:

root@admin:~# curl -g -6 "http://cdn.steampunkjunkies.com/.well-known/acme-challenge/foo"
sdfsad

root@admin:~# curl -g "http://cdn.steampunkjunkies.com/.well-known/acme-challenge/foo"
sdfsad

Anyone got any suggestions? I’ve seen this pop up on several servers now, and its getting more frequest. I’m just worried that the SSL certs will expire if I can’t work it out, and break my sites :frowning :frowning: FWIW, it all worked fine before and it’s only in the last week I’ve started getting these messages.

Thanks

Andy

#2

if you are using Digital Ocean , they have DNS problems now causing LE to fail Renewals.

#3

Thanks for the reply. This is actually with Linode. I wonder if they are having the same problem :confused:

They don’t expire for another 2 weeks - but I always panic when I get errors, as there is the potential its not fixed by the time they expire :astonished:

Cheers

Andy

#4

I think they both use Cloudflare DNS Firewall.

#5
#6

Hi @steampunkjnkies, sorry to hear you’re running into issues renewing. The random timing definitely sounds frustrating!

This seems unrelated to the DNS problems from Digital Ocean. There’s also nothing to indicate there’s a problem with “Cloudflare DNS FIrewall”, the other linked thread was a result of a Digital Ocean DNS problem, not anything CloudFlare related.

I’m unfamiliar with VestaCP. Have you raised an issue with their support about this problem?

Looking at the logs from our end it seems like there is a misconfiguration on your challenge response webserver. Instead of returning the ACME challenge token contents it’s returning an XHTML document:

{“type”:“urn:acme:error:unauthorized”,“detail”:"Invalid response from http://cdn.steampunkjunkies.com/.well-known/acme-challenge/-tuN8Fn96y1pClaOOlIAliFEIjoeRX2HqfTw1mN2eQQ: “\u003c!DOCTYPE html PUBLIC “-//W3C//DTD XHTML 1.0 Strict//EN” “http://www.w3.org/TR/xhtml1/DTD/xhtml1-strict.dtd”\u003e\n\u003chtml xmlns=“http:””,“status”:403

I’m going to update the title of this thread to more prominently feature that this is a question about VestaCP with the hopes someone more familiar with its configuration/Let’s Encrypt plugin can help identify why the wrong response is being returned to the challenge server.

#7

Hi @cpu,

Thanks for the reply. It’s very odd, as it was working fine before, and I’ve not updated any of the core VestaCP software, so I don’t think thats the problem.

Looking at the logs from our end it seems like there is a misconfiguration on your challenge response webserver. Instead of returning the ACME challenge token contents it’s returning an XHTML document:

Mmm interesting. Do you have the IP address that it is calling for that?

Cheers

Andy

#8

It was 2a01:7e00::f03c:91ff:feac:4983 - which also explains why this was working for you before but fails now. We recently began preferring IPv6 addresses for dual-homed hosts.

It seems like your challenge webserver isn’t configured properly for that IPv6 address. You could fix this (ideal) or remove the AAAA record (less ideal).

Hope that helps!

#9

Hi,

Thanks for looking into this. I’m a bit confused as to why it doesn’t want to work with IPv6 :confused: From another server, I just run:

root@admin:~# curl -v -g -6 “http://cdn.steampunkjunkies.com/.well-known/acme-challenge/foo”

  • Trying 2a01:7e00::f03c:91ff:feac:4983…
  • Connected to cdn.steampunkjunkies.com (2a01:7e00::f03c:91ff:feac:4983) port 80 (#0)

GET /.well-known/acme-challenge/foo HTTP/1.1
Host: cdn.steampunkjunkies.com
User-Agent: curl/7.47.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx
< Date: Wed, 21 Jun 2017 14:23:47 GMT
< Content-Type: text/plain
< Content-Length: 10
< Last-Modified: Wed, 21 Jun 2017 05:47:17 GMT
< Connection: keep-alive
< Keep-Alive: timeout=60
< ETag: “594a0865-a”
< Accept-Ranges: bytes
<
sdfsad

… so I’m a bit baffled as to why its not working :confused:

As you said - removing the AAA record would fix it, but its not ideal (as I want to be IPv6 compatible).

UPDATE: I just saw this thread on VestaCP’s forum: https://forum.vestacp.com/viewtopic.php?f=11&t=14812&p=60947#p60947 - I wonder if they are having the same problem as me ;/

Cheers

Andy

#10

I’m still battling with this one :frowning:

It connects fine with ipv6 my end (from another server);

root@admin:~# curl -v -g -6 "http://admin.chambresdhotesfrance.com/.well-known/acme-challenge/foo"
*   Trying 2a01:7e00::f03c:91ff:febc:64f3...
* Connected to admin.chambresdhotesfrance.com (2a01:7e00::f03c:91ff:febc:64f3) port 80 (#0)
> GET /.well-known/acme-challenge/foo HTTP/1.1
> Host: admin.chambresdhotesfrance.com
> User-Agent: curl/7.47.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Server: nginx
< Date: Tue, 27 Jun 2017 16:56:28 GMT
< Content-Type: text/plain
< Content-Length: 6
< Last-Modified: Tue, 20 Jun 2017 09:44:11 GMT
< Connection: keep-alive
< Keep-Alive: timeout=60
< ETag: "5948ee6b-6"
< Accept-Ranges: bytes
<
* Connection #0 to host admin.chambresdhotesfrance.com left intact

Are you able to try that request your end, and see what it gets? (with curl) . The nginx server sections are setup correctly:

server {
    listen      213.219.38.44:80;
    listen      [::]:80;

I’m really not sure what else to try (apart from removing the AAAA records, which isn’t ideal), as the certs will run out in a just over 2 weeks :confused:

Cheers

Andy

#11

Hi,

OK, so I’m really not sure this is an IPv6 issue. I removed the AAAA records for cdn.steampunkjunkies.com, and have let it update mxtoolbox shows it as non-existing now. However, I still get this error:

sudo /usr/local/vesta/bin/v-update-letsencrypt-ssl
Error: Invalid response from http://cdn.steampunkjunkies.com/.well-known/acme-challenge/yZzKCdFnu3M4SE2HnlxFoOPsfZrFaCjlYvMnztLdoNA:

I really don’t get what is going on here :frowning: Do you see any more debugging your end @cpu

#12

Hi @steampunkjnkies,

I’m afraid there’s nothing else for me to report from the server-side. In the most recent attempt ( 28/06/2017
06:21:01.650 +0000) the validation authority connected to "http://cdn.steampunkjunkies.com/.well-known/acme-challenge/cQV_ngbHvv-EoYz84_RYSJymGbxJdFA5fqk7ZKN835A", (resolved to 213.219.38.44) and received back an Invalid Response containing what looks like the beginning of an HTML document instead of the expected HTTP-01 challenge response with a key authorization from your ACME client.

Wish I could offer some more concrete suggestions for actions to take. It certainly seems like a misconfiguration with either VestaCP or your webserver configuration.

#13

Thanks @cpu . I’ve actually just paid the VestaCP guys to see if they can track it down, as I’m drawing a blank. Are you able to see the full page that gets returned? That could help track down what its actually seeing (and why, maybe)

Cheers

Andy

#14

I don’t believe that the CA saves the full document, but it’s highly likely that it was the exact same 404 page that appears if you go to http://cdn.steampunkjunkies.com/.well-known/acme-challenge/cQV_ngbHvv-EoYz84_RYSJymGbxJdFA5fqk7ZKN835A in a browser right now (that is, that the web server was already responding with a 404 error for attempts to access this file).

1 Like
#15

Thanks. The weird part is, that it DOES show the files correctly in that folder:

http://cdn.steampunkjunkies.com/.well-known/acme-challenge/foo

Just for some reason its not writing them (although, that is hard to test and the cleanup happens quicker than you can check to see if that file was written, and what its contents were).

Cheers

Andy

#16

That’s correct - we only see the first 128 bytes.

#17

I don’t suppose that VestaCP uses Certbot under the hood? Certbot has a --debug-challenges option that will pause before cleaning up the challenge.

#18

Hi,

We finally got to the bottom of it :slight_smile:

Apparently, VestaCP uses keys for LE that are based on the email address. According to the Vesta tech I spoke to, you need to have a unique email address per account (not domain, but user account), otherwise it goes weird. I’m a bit confused as to why it worked before - but it all looks good now :slight_smile: Hopefully this will save someone else from the stress I had! (I’ve suggested they bring up an alert to warm people of this, what they try and add the same email address to a 2nd account)

Thats for all the help! :slight_smile:

Cheers

andy

#19

Hi @steampunkjnkies,

Interesting! I would never have thought to suggest checking this.[quote=“steampunkjnkies, post:18, topic:36527”]
(I’ve suggested they bring up an alert to warm people of this, what they try and add the same email address to a 2nd account)
[/quote]

That sounds like a very sensible idea.

Thanks for reporting back with a solution! I’m very glad you were able to get this resolved! :lock: :chart_with_upwards_trend:

#20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.