Failed authorization procedure - ubuntu 16.04 & nginx


#1

Hi all,

I’m having some problems issuing certs for my domain. I recently moved the site from an older server to a new server. The old server was running on ubuntu 14.04 with apache and using the let’s encrypt software to issue certs - I had a perfectly working cert issued for my site.

My new server is running ubuntu 16.04, uses nginx and uses certbot to issue certs. when doing a dry run for new certs, I keep getting authorization failures, but I’m not sure why as I can browse to the directory manually, the permissions are set for 755 and I don’t see anything in my nginx config that would prevent proper access.

In addition to the above, my site sites behind cloudflare. I have a cloudflare issue cert, which I turned off until I can get my issues sorted out with certbot.

site url in question: teistebrito.com

Here’s the command I’m running:

certbot certonly --email certs@mysite.com --text --renew-by-default --agree-tos --webroot -w /path/to/site/public_html -d teistebrito.com -d www.teistebrito.com --dry-run

which then outputs the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for teistebrito.com
http-01 challenge for www.teistebrito.com
Using the webroot path /path/to/site/public_html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /path/to/site/public_html/.well-known/acme-challenge
Failed authorization procedure. www.teistebrito.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.teistebrito.com/.well-known/acme-challenge/D0CwPvOEQUjMJGCGF39uT7LmRQ150-3OuGZlMrUVzsc: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>ngin"

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.teistebrito.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.teistebrito.com/.well-known/acme-challenge/D0CwPvOEQUjMJGCGF39uT7LmRQ150-3OuGZlMrUVzsc:
   "<html>
   <head><title>404 Not Found</title></head>
   <body bgcolor="white">
   <center><h1>404 Not Found</h1></center>
   <hr><center>ngin"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address.

I manually created the /acme-challenge directory and put a test index.html file within said directory. you can browse to it from a browser: http://teistebrito.com/.well-known/acme-challenge/

here’s the info from the letsencrypt.log file in /var/log:

2017-06-18 23:37:19,860:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: www.teistebrito.com
Type:   unauthorized
Detail: Invalid response from http://www.teistebrito.com/.well-known/acme-challenge/D0CwPvOEQUjMJGCGF39uT7LmRQ150-3OuGZlMrUVzsc: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>ngin"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address.
2017-06-18 23:37:19,860:INFO:certbot.auth_handler:Cleaning up challenges
2017-06-18 23:37:19,861:DEBUG:certbot.plugins.webroot:Removing /path/to/site/public_html/.well-known/acme-challenge/h9oep-03WkA_ondXUgvD_5aeydn5yUN1HyhkXmg00oU
2017-06-18 23:37:19,861:DEBUG:certbot.plugins.webroot:Removing /path/to/site/public_html/.well-known/acme-challenge/D0CwPvOEQUjMJGCGF39uT7LmRQ150-3OuGZlMrUVzsc
2017-06-18 23:37:19,861:INFO:certbot.plugins.webroot:Unable to clean up challenge directory /path/to/site/public_html/.well-known/acme-challenge
2017-06-18 23:37:19,862:DEBUG:certbot.plugins.webroot:Error was: [Errno 39] Directory not empty: '/path/to/site/public_html/.well-known/acme-challenge'
2017-06-18 23:37:19,863:DEBUG:certbot.main:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.11.1', 'console_scripts', 'certbot')()
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 882, in main
    return config.func(config, plugins)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 659, in obtain_cert
    action, _ = _auth_from_available(le_client, config, domains, certname, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 108, in _auth_from_available
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 294, in obtain_and_enroll_certificate
    certr, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 265, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 77, in get_authorizations
    self._respond(resp, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 134, in _respond
    self._poll_challenges(chall_update, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 198, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. www.teistebrito.com (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.teistebrito.com/.well-known/acme-challenge/D0CwPvOEQUjMJGCGF39uT7LmRQ150-3OuGZlMrUVzsc: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>ngin"

here’s my nginx vhost config:

server {
    listen 80;
    listen [::]:80;
    server_name teistebrito.com;


    root /path/to/site/public_html;
    index index.php index.html index.htm index.nginx-debian.html;

    # Logs
    access_log /logs/access.log;
    error_log /logs/error.log warn;

  #  server_name teistebrito.com;


         location / {

                autoindex on;
                try_files $uri $uri/ =404;
    }

    location ~ \.php$ {
        include snippets/fastcgi-php.conf;
        fastcgi_pass unix:/run/php/php7.0-fpm.sock;
    }

    location ~ /\.ht {
        deny all;
    }
}

does anyone have any ideas what’s going on here? any help is greatly appreciated!


#2

If
http://www.teistebrito.com/.well-known/acme-challenge/
does not point to:
/path/to/site/public_html/.well-known/acme-challenge/
then HTTP authentication will fail.

try placing a “test.txt” file in that folder.
echo “just a test” > /path/to/site/public_html/.well-known/acme-challenge/test.txt
and see if it can be reached from the Internet.

of course, it “public_html” is a file (not a folder) then there is your problem = all requests are forced to hit that html file.


#3

There is also an IPv6 address for this host, yet it appears to be configured properly and responding to IPv6 requests with the same content as to IPv4 requests (so that’s great). Still, when doing the test that @rg305 suggests, it would be worth checking that IPv6 answers the same way as IPv4, too.


#4

I ran the exact command you posted (changing the path to the correct 1, of course) and it is browseable on the net just fine: http://teistebrito.com/.well-known/acme-challenge/test.txt


#5

is there anything specific I can do to check the responses between the two types of IP addresses?


#6

Try from Internet:
curl -IkL4 http://teistebrito.com/.well-known/acme-challenge/test.txt
curl -IkL6 http://teistebrito.com/.well-known/acme-challenge/test.txt

Since Let’sEncrypt will prefer IPv6, you must ensure that IPv6 is working.


#7

the 1st one worked fine, the second one produced an error from my local server:

http://teistebrito.com/.well-known/acme-challenge/test.txt
-bash: http://teistebrito.com/.well-known/acme-challenge/test.txt: No such file or directory

I then tried the second command from a server in a proper data center and it errors out with:
curl: (7) Couldn’t connect to server


#8

DNS shows IPv4 + IPv6
Your server shows :80 + [::]:80
But I suspect the “gateway” is not participating with IPv6 flow.

You could either remove IPv6 entries from DNS or get the “gateway” configured properly.


#10

hmn…

the thing is, I do not have any AAAA records set for the domain name…I literally only have 2 records set in cloudflare, 1 A record for teistebrito.com and 1 A record for www.teistebrito.com…is cloudflare automatically proxying AAAA records?


#11

It seems CloudFlare automatically adds AAAA records:

Name: www.teistebrito.com
Addresses: 2400:cb00:2048:1::681f:43f9
2400:cb00:2048:1::681f:42f9
104.31.67.249
104.31.66.249

If they truly act as a proxy/cache, then the backend of that connection could be IPv4/IPv6 and/or HTTP/HTTPS.
I don’t really know how CloudFlare works.
But they will need to provide an SSL cert to cover your domain; for the clients that will hit their servers for HTTPS.
Which at the moment they already are:
DNS Name=sni109617.cloudflaressl.com

DNS Name=*.teistebrito.com

DNS Name=teistebrito.com


#12

For dual-stack origins, Cloudflare prefers IPv4. (For IPv6-only origins, Cloudflare uses IPv6.) The IP version between the client and Cloudflare, and between Cloudflare and the origin, aren’t related.

Edit: By default, Cloudflare’s caching shouldn’t be an issue. They only cache certain file extensions. And even for those file extensions, 404 errors are only cached 5 minutes. But that’s all configurable, via Cloudflare’s settings and cache headers from the origin.

Edit: Actually, 404 errors can’t be cached for more than 5 minutes. I think.


#13

Ok
But the clients will never see his cert - only the CloudFlare cert.


#14

Can you explain this statement?
Even if possible (aka: MITM Transparent SSL Proxy); they would NOT be able to cache any SSL content since they can’t see it. Defeating the whole point of the CloudFlare solution.


#15

yes…when my site was on the old server, I had a cert via let’s encrypt as well as a cloudflare issued cert for end-to-end encryption. when I moved to the new server, after not being able to get a cert issued cleanly from certbot, I set SSL in cloudflare from full (strict) to off. I’m still unable to get any certs from let’s encrypt…


#16

Try adding one of these:
–preferred-challenges http
–preferred-challenges http-01


#17

both failed with the same errors as before.


#18

What do you see in your logs?
Both of these show 404:
http://www.teistebrito.com/.well-known/acme-challenge/test.txt
https://www.teistebrito.com/.well-known/acme-challenge/test.txt

Surprisingly this works:
http://teistebrito.com/.well-known/acme-challenge/test.txt
And
https://teistebrito.com/.well-known/acme-challenge/test.txt
redirects to
http://teistebrito.com/.well-known/acme-challenge/test.txt

FYI: certbot won’t follow HTTPS redirects.

You seem to have two vhost files or are missing the server_alias for www
And you should probably turn CloudFlare SSL back on (if there is a less than strict mode, use it for now)

[EDIT]
In NGINX use:
server_name teistebrito.com www.teistebrito.com;


#19

ah, that was it!

I forgot to set up a redirect in nginx for www!

adding this to my vhost was the ticket:

server {
    server_name www.teistebrito.com;
    return 301 $scheme://teistebrito.com$request_uri;
}

thanks everyone for the help!


#20

sorry to bump this again gents but I’m having another problem for a different cert renewal for a different domain name on the same server.

I’m trying to renew my certs. when I run the renew command:
certbot renew --post-hook “systemctl restart nginx ; systemctl restart postfix” >> /var/log/le-renew.log

I get the following:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Cert not yet due for renewal
Cert is due for renewal, auto-renewing…
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for foroafeitado.com
http-01 challenge for charts.foroafeitado.com
http-01 challenge for mail.foroafeitado.com
http-01 challenge for storm.foroafeitado.com
http-01 challenge for www.foroafeitado.com
Waiting for verification…
Cleaning up challenges
Unable to clean up challenge directory /path/to/site/public_html/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/foroafeitado.com.conf produced an unexpected error: Failed authorization procedure. mail.foroafeitado.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mail.foroafeitado.com/.well-known/acme-challenge/oF1b9FLnxuN8vf8rB57mlwM30mUdLFgHLZbR_hC05OE: Timeout. Skipping.
Running post-hook command: systemctl restart nginx ; systemctl restart postfix
1 renew failure(s), 0 parse failure(s)

what would cause a timeout? I can manually browse to the directory:
https://mail.foroafeitado.com/.well-known/acme-challenge/test.txt

mail.foroafeitado.com is set up for postfix and to use as a PTR record.

when originally generating the certs, I ran the following command:
certbot certonly --email email@random.com --text --renew-by-default --agree-tos --webroot -w /path/to/site/public_html -d foroafeitado.com -d www.foroafeitado.com -d mail.foroafeitado.com -w /path/to/site/2/public_html -d charts.foroafeitado.com -w /path/to/site/3/public_html -d storm.foroafeitado.com

If I run the above the generate a new cert with --dry-run and remove -d mail.foroafeitado.com, it goes through just fine!

edited to add:
I don’t have anything set up in an nginx vhost for mail.foroafeitado.com, but I believe because I have default_server set up for the main domain, foroafeitado.com, it sends anything that doesn’t have a vhost through the main domain. I run a near identical set up on another server and it works just fine!

If this is not how it works, forgive me as I’m a sys admin in training during my free time!

any help is greatly appreciated!


#21

mail.foroafeitado.com has an IPv6 AAAA record but no web server responds at that address.