Renewal no longer works - webroot is accessible with curl

Hi all…

When I first set up letsencrypt a couple months ago for a few domains, I immediately tested certificate renewal with a dry run of certbot and everything seemed fine. Now it’s failing for all of them.

My server gives a 404 error on the challenge file, but when I created a file with the contents “Hello there” using the same name as the challenge file, I can download it just fine. I used tcpdump to see the HTTP request triggered by the renewal attempt, and even when I muck about with curl or telnet to issue a request that looks identical, I get no errors.

I have debug level logs turned on for nginx and for some reason the requests I issue are treated differently.

Failing Let’s Encrypt request:

test location: "/"
using configuration "/"

Test request:

test location: "/"
test location: ".well-known/acme-challenge"
using configuration "/.well-known/acme-challenge"

It looks like somehow the virtual host config for the domain is used for my test request but not for the Let’s Encrypt request.

There’s got to be something I’m overlooking. It looks like some weird clash between Let’s Encrypt’s client and my nginx config.

The request for both looks like:

GET /.well-known/acme-challenge/kjQjm_-x2gANBCuKWHwQ7kkMUiC4AT8xR4mKE4X4x4c HTTP/1.1
Host: tina.teeninga.ca
User-Agent: Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
Accept: */*
Accept-Encoding: gzip
Connection: close

I got curl to issue that request with the command: curl -v -A "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" -H "Accept-Encoding: gzip" -H "Connection: close" http://tina.teeninga.ca/.well-known/acme-challenge/kjQjm_-x2gANBCuKWHwQ7kkMUiC4AT8xR4mKE4X4x4c

My domain is: tina.teeninga.ca

I ran this command: certbot renew --dry-run --debug-challenges --post-hook "systemctl reload nginx" (The debug challenges option doesn’t seem to work for me.)

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/tina.teeninga.ca.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for tina.teeninga.ca
Waiting for verification...

-------------------------------------------------------------------------------
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.   
-------------------------------------------------------------------------------
Cleaning up challenges
Unable to clean up challenge directory /srv/letsencrypt/tina.teeninga.ca/.well-known/acme-challenge
Attempting to renew cert (tina.teeninga.ca) from /etc/letsencrypt/renewal/tina.teeninga.ca.conf produced an unexpected error: Failed authorization procedure. tina.
teeninga.ca (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://tina.teeninga.ca/.well-known/acme-c
hallenge/cOyg7m0pwNgZ2ELCOW0jt5vj-qk7lNS7o1iHw6MD1lc: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>". Skipping.

The failure to remove the directory is due to my test file being there. When I increase the verbosity, there are no errors in creating the challenge file.

My web server is (include version): nginx version: nginx/1.10.3 (Ubuntu)

My web server config is:

server {
        server_name tina.teeninga.ca;
        listen 80;
        # Directory to use for webroot challenge for certbot                                                                                                     
        #                                                                                                                                                          
        location ~ /.well-known/acme-challenge {
                allow all;
                default_type "text/plain";
                root /srv/letsencrypt/tina.teeninga.ca;
        }
        location / {
                return 301 https://$server_name$request_uri;
        }
}
server {
        server_name tina.teeninga.ca;
        listen 443 ssl;
...

The operating system my web server runs on is (include version): Ubuntu Server 16.04.3 LTS

My hosting provider, if applicable, is: linode

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

Hi @jbl,

Your “problem” is that tina.teeninga.ca resolves A (IPv4) and AAAA (IPv6) records:

$ dig tina.teeninga.ca A +short
qormi.dae.ca.
74.207.244.4

$ dig tina.teeninga.ca AAAA +short
qormi.dae.ca.
2600:3c01::f03c:91ff:fe60:ac0e

But your site is not configured properly for IPv6 requests, it doesn’t return the same content as for IPv4 requests.

Using IPv4 it is correct:

$ curl -4ikL http://tina.teeninga.ca/.well-known/acme-challenge/kjQjm_-x2gANBCuKWHwQ7kkMUiC4AT8xR4mKE4X4x4c
HTTP/1.1 200 OK
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 28 Aug 2017 08:26:21 GMT
Content-Type: text/plain
Content-Length: 12
Last-Modified: Mon, 28 Aug 2017 05:38:03 GMT
Connection: keep-alive
ETag: "59a3ac3b-c"
Accept-Ranges: bytes

Hello there

Using IPv6 it isn’t:

$ curl -6ikL http://tina.teeninga.ca/.well-known/acme-challenge/kjQjm_-x2gANBCuKWHwQ7kkMUiC4AT8xR4mKE4X4x4c
HTTP/1.1 404 Not Found
Server: nginx/1.10.3 (Ubuntu)
Date: Mon, 28 Aug 2017 08:25:58 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.10.3 (Ubuntu)</center>
</body>
</html>

Since a few months Let’s Encrypt prefers IPv6 over IPv4 so or you fix your web server configuration to answer IPv6 request correctly or remove the AAAA record (indeed it is a CNAME) for your domain.

Cheers,
sahsanu

3 Likes

Thanks, @sahsanu. Now that I see that, I don’t know why I would have left that out of the nginx config… Granted, I don’t actually have IPv6 connectivity at home that I could have tested it from. :flushed:

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.