Urn:acme:error:unauthorized, 404, without objective reasons

I used a certificate for about three months and started getting an error.

Server: Ubuntu 16.04, NGINX

Command: certbot certonly --force-renewal -a webroot --webroot-path=/var/www/html/ -d www.***.com -d ***.com

Error:

Failed authorization procedure. www.***.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.***.com/.well-known/acme-challenge/***: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p", ***.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://***.com/.well-known/acme-challenge/***: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p..."

Rights to the directory: 777.

Can I access test files from browser? Yes.

Are the files being created? Yes.

Then I ran the test: After starting the command and before it finishes I refreshed my /.well-known/acme-challenge/ directory and I caught files with their names. After they were deleted I created new pair with copied names and my server returns it with status 200. I’m trying to find a solution for a few days.

What is the problem? Are DNS records of Let’s Encrypt ok?
I can provide domain name in PM.

Place a test file at: http:/***.com/.well-known/acme-challenge/test.txt

It works. Got 200 and content of the file.

Correct this If I’m wrong:
file
/var/www/html/.well-known/acme-challenge/test.txt exists
and it’s accessible from the Internet
but you can’t force renewal (even) with
certbot certonly --force-renewal -a webroot --webroot-path=/var/www/html/ -d www.***.com -d ***.com
and error message is still:
unauthorized :: The client lacks sufficient authorization :: Invalid response from http://***.com/.well-known/acme-challenge/***:

By the way, you can get Certbot to pause at the moment where the files exist with --debug-challenges if you want to experiment more with what’s going on. If I remember correctly, it will ask you to press Enter to continue.

You’re completely right.

Okay, thanks for info. I ran second test.

First terminal:

>> root@***5:~# certbot certonly -a webroot --webroot-path=/var/www/html/ -d www.***.com -d ***.com --debug-challenges

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.***.com
http-01 challenge for ***.com
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...

-------------------------------------------------------------------------------
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
-------------------------------------------------------------------------------
Press Enter to Continue

Second terminal:

>> root@***:/# cd /var/www/html/.well-known/acme-challenge && ls -l

-rw-r--r-- 1 root root 87 Aug 14 21:01 azzWj-KwjhMhHKfx9dwLT5AqCN580m7TGVlGYmqY8J8
-rw-r--r-- 1 root root 87 Aug 14 21:01 lCnlyyzGzoHmqMkmLhhehScELsIkBNWM3VnL4sI0ykY
-rw-r--r-- 1 root root  7 Aug 14 00:45 test.txt

>> root@***:/# curl -I -X GET http://***.com/.well-known/acme-challenge/azzWj-KwjhMhHKfx9dwLT5AqCN580m7TGVlGYmqY8J8

HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 14 Aug 2017 18:11:46 GMT
Content-Type: application/octet-stream
Content-Length: 87
Last-Modified: Mon, 14 Aug 2017 18:01:20 GMT
Connection: keep-alive
ETag: "5991e570-57"
Accept-Ranges: bytes

First terminal:

*<Enter>*

Resetting dropped connection: acme-v01.api.letsencrypt.org
Resetting dropped connection: acme-v01.api.letsencrypt.org
Cleaning up challenges
Unable to clean up challenge directory /var/www/html/.well-known/acme-challenge
Failed authorization procedure. www.***.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.***.com/.well-known/acme-challenge/lCnlyyzGzoHmqMkmLhhehScELsIkBNWM3VnL4sI0ykY: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p", ***.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://***.com/.well-known/acme-challenge/azzWj-KwjhMhHKfx9dwLT5AqCN580m7TGVlGYmqY8J8: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.***.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.***.com/.well-known/acme-challenge/lCnlyyzGzoHmqMkmLhhehScELsIkBNWM3VnL4sI0ykY:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   Domain: ***.com
   Type:   unauthorized
   Detail: Invalid response from
   http://***.com/.well-known/acme-challenge/azzWj-KwjhMhHKfx9dwLT5AqCN580m7TGVlGYmqY8J8:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

I don’t know what’s going on.

Maybe a difference between the forms with and without www? You didn’t indicate whether you used the leading www in your curl command.

Edit: Although that’s probably not it because it looks like they both failed in the Certbot output.

Can you run the curl command on a machine that’s not the server itself to see if it looks the same from inside and outside your network?

That seems to contradict ROOT created and owned files:

This is quite strange…

Are you using links?
Please show (as they relate to the challenges):
ls -l /var/www/html/
ls -l /var/www/html/.well-*
ls -l /var/www/html/.well-known/acme*

Third test here.

First terminal:

>> root@***:~# certbot certonly -a webroot --webroot-path=/var/www/html/ -d www.***.com -d ***.com --debug-challenges

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for www.***.com
http-01 challenge for ***.com
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...

-------------------------------------------------------------------------------
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
-------------------------------------------------------------------------------
Press Enter to Continue

Second terminal:

>> root@***:/# chmod 777 -R /var/www/html/

>> root@***:/# ls -l /var/www/html/

total 8
-rwxrwxrwx 1 root root 612 Dec 11  2016 index.nginx-debian.html
-rwxrwxrwx 1 root root  24 Jan  5  2017 index.php

>> root@***:/# ls -l /var/www/html/.well-*

total 4
drwxrwxrwx 2 root root 4096 Aug 15 00:00 acme-challenge

>> root@***:/# ls -l /var/www/html/.well-known/acme*

-rwxrwxrwx 1 root root 87 Aug 15 00:00 hNhL9nYJQ0VWjZJfHgzjyoz5imNHNM_5nC1FkLIJP30
-rwxrwxrwx 1 root root 87 Aug 15 00:00 I8DFqSVAbWSc9r3kDXKVJfD46UNjkdxARcnrR7PLFIw
-rwxrwxrwx 1 root root  7 Aug 14 00:45 test.txt

First terminal:

*<Enter>*

Resetting dropped connection: acme-v01.api.letsencrypt.org
Cleaning up challenges
Unable to clean up challenge directory /var/www/html/.well-known/acme-challenge
Failed authorization procedure. www.***.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.***.com/.well-known/acme-challenge/hNhL9nYJQ0VWjZJfHgzjyoz5imNHNM_5nC1FkLIJP30: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p", ***.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://***.com/.well-known/acme-challenge/I8DFqSVAbWSc9r3kDXKVJfD46UNjkdxARcnrR7PLFIw: "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
<html><head>
<title>404 Not Found</title>
</head><body>
<h1>Not Found</h1>
<p"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.***.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.***.com/.well-known/acme-challenge/hNhL9nYJQ0VWjZJfHgzjyoz5imNHNM_5nC1FkLIJP30:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   Domain: ***.com
   Type:   unauthorized
   Detail: Invalid response from
   http://***.com/.well-known/acme-challenge/I8DFqSVAbWSc9r3kDXKVJfD46UNjkdxARcnrR7PLFIw:
   "<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
   <html><head>
   <title>404 Not Found</title>
   </head><body>
   <h1>Not Found</h1>
   <p"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

P.S. Files are always deleted successfully after the command finishes.

Okay, fourth test here:

Another server:

>> root@AnotherServer:/# curl -I -X GET http://***.com/.well-known/acme-challenge/N57YuUay5FqxGsh-vB4dhHqhpGbfq4lXuNbcNRf_EWA

HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 14 Aug 2017 21:14:56 GMT
Content-Type: application/octet-stream
Content-Length: 87
Last-Modified: Mon, 14 Aug 2017 21:13:26 GMT
Connection: keep-alive
ETag: "59921276-57"
Accept-Ranges: bytes

Any ideas?

Can you find any associated 404s in your web server logs? Did those 404s actually come from the same server?

There are no records in the access.log from Let’s Encrypt, but are from cURL, Browsers, etc. This is the reason why I think Let’s Encrypt has DNS problem. Any ideas?

The auth requests returns 404 error - who is that from?

who is that from

What do you mean?

Who/What is returning the 404 error?
Is there an IPS/firewall in place, or any other device, that might be redirecting or answering those auth requests?

I have no firewall. Just clean Ubuntu. I don’t know what’s going on. Could that be error of Let’s Encrypt DNS?

doubtful, but without the domain name it is hard to be sure.

Provided in private message.