Problem with my domain kurilo.su

dkurilo · September 20, 2017, 7:53pm

My domain is: kurilo.su
I'm trying to obtain certificate with certbot manually. Command to launch certbot:
./certbot-auto --cb-auto-has-root --preferred-challenges http --manual certonly
Here is log:

$ ./certonly.sh
 Saving debug log to /var/log/letsencrypt/letsencrypt.log
 Plugins selected: Authenticator manual, Installer None
 Please enter in your domain name(s) (comma and/or space separated)  (Enter 'c'
 to cancel): kurilo.su www.kurilo.su
 Obtaining a new certificate
 Performing the following challenges:
 http-01 challenge for kurilo.su
 http-01 challenge for www.kurilo.su
-------------------------------------------------------------------------------
NOTE: The IP of this machine will be publicly logged as having requested this
certificate. If you're running certbot in manual mode on a machine that is not
your server, please ensure you're okay with that.

Are you OK with your IP being logged?
-------------------------------------------------------------------------------
(Y)es/(N)o: y

-------------------------------------------------------------------------------
Create a file containing just this data:

L-y6nLuH5N2irtD_EJyCTvcHaEZ-ai175lhAPbU48Cw.6wJUTun9R5XnbykZE3rbqmAGYcPEcAmkowRWDnUP90Y

And make it available on your web server at this URL:

http://kurilo.su/.well-known/acme-challenge/L-y6nLuH5N2irtD_EJyCTvcHaEZ-ai175lhAPbU48Cw

-------------------------------------------------------------------------------
Press Enter to Continue

-------------------------------------------------------------------------------
Create a file containing just this data:

tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk.6wJUTun9R5XnbykZE3rbqmAGYcPEcAmkowRWDnUP90Y

And make it available on your web server at this URL:

http://www.kurilo.su/.well-known/acme-challenge/tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk

-------------------------------------------------------------------------------
Press Enter to Continue
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. www.kurilo.su (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.kurilo.su/.well-known/acme-challenge/tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p", kurilo.su (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://kurilo.su/.well-known/acme-challenge/L-y6nLuH5N2irtD_EJyCTvcHaEZ-ai175lhAPbU48Cw: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.kurilo.su
   Type:   unauthorized
   Detail: Invalid response from
   http://www.kurilo.su/.well-known/acme-challenge/tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   Domain: kurilo.su
   Type:   unauthorized
   Detail: Invalid response from
   http://kurilo.su/.well-known/acme-challenge/L-y6nLuH5N2irtD_EJyCTvcHaEZ-ai175lhAPbU48Cw:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

But:

$ nc kurilo.su 80
GET /.well-known/acme-challenge/tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk HTTP/1.1
Host:www.kurilo.su

HTTP/1.1 200 OK
Server: nginx/Zenon version
Date: Wed, 20 Sep 2017 19:37:51 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 87
Connection: keep-alive
X-ACLR-Version: 0.05-zen
Last-Modified: Wed, 20 Sep 2017 19:30:59 GMT
Accept-Ranges: bytes
Cache-Control: max-age=2592000
Expires: Fri, 20 Oct 2017 19:37:51 GMT
X-UA-Compatible: IE=edge
X-Content-Type-Options: nosniff

tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk.6wJUTun9R5XnbykZE3rbqmAGYcPEcAmkowRWDnUP90Y

And I even can't find what server can answer with such 404 error.
Also in letsencrypt.log I have such lines:

{
  "type": "http-01",
  "status": "invalid",
  "error": {
    "type": "urn:acme:error:unauthorized",
    "detail": "Invalid response from http://www.kurilo.su/.well-known/acme-challenge/tzh47q_tZVjKjMY0MHzm-BLnQ-4WDEfwTjYECZPvEyk: \"\u003c!DOCTYPE HTML PUBLIC \"-//IETF//DTD HTML 2.0//EN\"\u003e\n\u003chtml\u003e\u003chead\u003e\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\n\u003c/head\u003e\u003cbody\u003e\n\u003ch1\u003eNot Found\u003c/h1\u003e\n\u003cp\"",
    "status": 403
  },

I think it can be a problem with domain .su. Can it be?

schoen · September 20, 2017, 9:04pm

Hi @dkurilo,

That’s very confusing because it also looks to me like you set up the challenges correctly, and so the certificate authority should not have given an error. A few thoughts:

Is it possible that your web server returns a different result depending on the IP address from which it receives an inbound connection?
Or based on the User-Agent string of the client?
Could there be an application firewall that somehow changes the responses that clients get based either of these factors?
Let’s Encrypt now validates challenges from multiple locations on the Internet; is there any way that the answer could have changed in response to a subsequent request?
Can you look at your nginx server logs to see if it was, in fact, your copy of nginx that gave the 404 error for some reason?

dkurilo · September 20, 2017, 9:37pm

Hi @schoen,
It’s shared hosting so unfortunately I can check very few things on the server.
It uses nginx proxy and Apache web server. It’s another thing that make difficult to find what is wrong.

Surely it’s possible. But hosting provider’s nginx has not the same response ( http://213.189.197.234/ ) then what I see in letsencrypt log file. And it never return 404 or 403. I tried.
What User-Agent string letsencrypt sends?
I hope no. I haven’t noticed such problems yet.
I don’t have access to nginx logs. But I don’t see any queries for these files in Apache logs. So it’s possible that nginx blocks queries from letsencrypt, but the answer in letsencrypt.log is very weird for me. Looks like it’s not my hosting provider nginx response. Is it possible to have full log (including user agent, Host, and other headers from both, the query and response)

schoen · September 20, 2017, 9:41pm

We could ask @cpu to try to find this information.

dkurilo · September 20, 2017, 11:15pm

Thank you.
It would be great if someone can help me.
I want to play with camera, but it’s impossible without SSL.

cpu · September 21, 2017, 5:00pm

I took a look at the server logs.

It's the primary VA that's failing - no remote VA checks from other vantage points are being done. E.g. there should only be one request in the nginx logs per validation attempt. The remote VAs are only asked to perform validation once the primary VA is satisfied as a measure of checking consensus on valid challenge responses.

We don't specifically advertise the UA that the validation authority uses on challenge requests. It's pretty easy to figure out but I don't want to encourage hacks that specifically UA match Let's Encrypts VA since we should be free to update/change it without worrying about breaking folks downstream.

@dkurilo Can you make a static test file in http://www.kurilo.su/.well-known/acme-challenge/ that I can try accessing?

schoen · September 21, 2017, 5:01pm

Thanks for looking into this, @cpu!

dkurilo · September 21, 2017, 5:51pm

Thanks, @schoen and @cpu.
I see no queries from VA, so it’s not important. I asked about UA just to be able to ask my hosting provider if they blocks it for some reason. But I hope they don’t do it. Is UA the same as in /test/config/va.json ?
@cpu, sure:

cpu · September 21, 2017, 7:07pm

I don't believe it exactly matches the default we have in the Boulder example configs but I would have to check.

What is the contents of this file supposed to be? I get back a XHTML document. Can you make a plaintext test file with no extension to more closely mimic the ACME challenge token file? (Sorry, I should have been more specific to start)

dkurilo · September 21, 2017, 7:13pm

@cpu, it’s plain text:

dima_000@Chth0n /cygdrive/d/work/letsencrypt
$ nc www.kurilo.su 80
GET /.well-known/acme-challenge/test HTTP/1.1
Host: www.kurilo.su

HTTP/1.1 200 OK
Server: nginx/Zenon version
Date: Thu, 21 Sep 2017 19:11:44 GMT
Content-Type: text/plain; charset=utf-8
Content-Length: 24
Connection: keep-alive
X-ACLR-Version: 0.05-zen
Last-Modified: Thu, 21 Sep 2017 17:04:00 GMT
Accept-Ranges: bytes
Cache-Control: max-age=2592000
Expires: Sat, 21 Oct 2017 19:11:44 GMT
X-UA-Compatible: IE=edge
X-Content-Type-Options: nosniff

c29tZSB0ZXN0IGNvbnRlbnQ=

jmorahan · September 21, 2017, 8:24pm

Not sure if this helps but I checked using Tor Browser and switching circuits a few times. Most times I get the text file as expected, but sometimes I get an nginx 403 error page.

Is it possible that your hosting provider is blacklisting some IP addresses or ranges?

dkurilo · September 21, 2017, 8:54pm

I think it’s possible. Every hosting provider make blacklist after DDOS attacks.
It means that letsencrypt IP address was involved in such attack. Surely it could be false detection.
But it can be not only my problem. So maybe it’s possible to extend logs or to find any other way to detect such situation.
If it’s impossible to find solution, I’ll just use dns challenge. It’s not a problem.

schoen · September 21, 2017, 8:56pm

I think the IP address blacklist is the most likely explanation when different people are seeing very different content for your test page.

dkurilo · September 21, 2017, 9:11pm

@schoen, it’s possible, but @jmorahan wrote he received 403 error page, but I have 404 error page in letsencrypt.log.
And I even don’t know what is the source of this response. Header from response in logs would make it a lot more clear.

ahaw021 · September 22, 2017, 4:20am

have you tried running the command again?

I would suggest adding the --staging flag and running the command a few times

You can then analyse the NGINX logs to see whats happening

I would also suggest adding --debug-challenges and checking the URL before submitting to let’s encrypt (to catch anything weird going on)

–debug-challenges After setting up challenges, wait for user input
before submitting to CA (default: False)

Andrei

cpu · September 22, 2017, 2:06pm

I suspect there is something happening at the network/ISP level. When I checked this file yesterday I got back an XHTML document that seemed similar to what the production VA received. When I checked today I got back a plaintext document.

I'm not sure what to suggest at this point. Can you contact your host to inquire about what may be causing this?

dkurilo · September 22, 2017, 2:34pm

@cpu, thank you.
Will think what to do.

@ahaw021, thank you for suggestion.

system · October 22, 2017, 2:34pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.