I had a working setup until about a week ago. Something changed but I’m unable to point to it:
$ sudo certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/bookiza.io.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for bubbl.in
http-01 challenge for bubblin.io
http-01 challenge for www.bubbl.in
http-01 challenge for www.bubblin.io
http-01 challenge for bookiza.io
http-01 challenge for www.bookiza.io
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (bookiza.io) from /etc/letsencrypt/renewal/bookiza.io.conf produced an unexpected error: Failed authorization procedure. www.bookiza.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.bookiza.io/.well-known/acme-challenge/9YHK8HkRZrdirbDJrrC9kp--5z-8ampqHqGcCrcI190: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>", bookiza.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://bookiza.io/.well-known/acme-challenge/_Sw3GRDhf__OUm1QBxDGkaygx8gh7ztKoWuJbOi1wwA: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>". Skipping.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.bubbl.in.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.bubbl.in
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/www.bubbl.in/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/bubblin.io.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for bubblin.io
tls-sni-01 challenge for www.bubblin.io
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/bubblin.io/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/www.bubblin.io.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for www.bubblin.io
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/www.bubblin.io/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/bubbl.in.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
tls-sni-01 challenge for bubbl.in
Waiting for verification...
Cleaning up challenges
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/bubbl.in/fullchain.pem
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
The following certs could not be renewed:
/etc/letsencrypt/live/bookiza.io/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates below have not been saved.)
The following certs were successfully renewed:
/etc/letsencrypt/live/www.bubbl.in/fullchain.pem (success)
/etc/letsencrypt/live/bubblin.io/fullchain.pem (success)
/etc/letsencrypt/live/www.bubblin.io/fullchain.pem (success)
/etc/letsencrypt/live/bubbl.in/fullchain.pem (success)
The following certs could not be renewed:
/etc/letsencrypt/live/bookiza.io/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
** (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
**IMPORTANT NOTES:**
- The following errors were reported by the server:
Domain: www.bookiza.io
Type: unauthorized
Detail: Invalid response from
http://www.bookiza.io/.well-known/acme-challenge/9YHK8HkRZrdirbDJrrC9kp--5z-8ampqHqGcCrcI190:
"<html>\r\n<head><title>404 Not Found</title></head>\r\n<body
bgcolor=\"white\">\r\n<center><h1>404 Not
Found</h1></center>\r\n<hr><center>"
Domain: bookiza.io
Type: unauthorized
Detail: Invalid response from
http://bookiza.io/.well-known/acme-challenge/_Sw3GRDhf__OUm1QBxDGkaygx8gh7ztKoWuJbOi1wwA:
"<html>\r\n<head><title>404 Not Found</title></head>\r\n<body
bgcolor=\"white\">\r\n<center><h1>404 Not
Found</h1></center>\r\n<hr><center>"
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
So it’s working fine for one domain but not for the other. I can provide the contents of nginx for Bookiza here if that helps. DNS setup looks okay and nothing has changed on it fwiw.
could you copy/paste (as “preformatted text” with the [</>] button) the content of /etc/letsencrypt/renewal/bookiza.io.conf?
The bookiza.io cert contains 6 hostnames, three base domains (bubbl.in, bubblin.io and bookiza.io and their www subdomains). However, you are also getting separate certificates for bubbl.in, bubblin.ioand separate certificates for their www subdomain. Is there a good reason to get “double”, or even “triple” certificates for those base domains?
For those separate certificates: you’re using the tls-sni-01 challenge, which is being deprecated in 2019.
I remember running sudo certbot --nginx --preferred-challenges http a few weeks ago to not use tls-sni-01, but it appears that older installations are still lying around.
This server is mainly to host https://bubblin.io (a rails application handling www, root and other request situations) but I also set up a static site (https://bookiza.io) later on, so the installation may not one of the cleanest. I don't need to fetch double or triple instances of any of the certificates, how do I go about cleaning up now?
The redirections are messed up: http://www.bookiza.io/.well-known/acme-challenge/test
redirects to: https://www.bookiza.io/.well-known/acme-challenge/test/.well-known/acme-challenge/test
I now see the name resolves to IPv4 and IPv6 addresses.
LE will prefer IPv6 when present.
You must ensure that IPV6 is working or remove the AAAA record from DNS:
Name: www.bookiza.io
Addresses: 2600:3c00::f03c:91ff:fe56:3901
50.116.16.25
It would seem that your tool doesn’t check both IPv4 and IPv6 independently when present.
Or the misdirection is only related to IPv6 and the acme-challenge folder…
Interesting. So IPv6 appears to be configured correctly on my DNS manager. I have a quad-A record for Bookiza that’s mapped to the ipv6-ip provided by Linode. This is working fine for the other domain bubblin.io! To experiment I just added a blank quad-A record for bookiza (apart from *, www and @) on the DNS manager to see if that’s something it’s looking for.
Edit: It’s the same as @ hostname value so no changes were effected on the DNS of bookiza.
The IPv6 may be pointing to the correct system.
The problem is in how the redirection is being handled for IPv6.
Which is different than how it is being handled for IPv4.
Please find and show the vhost that listens to IPv6 (::80 or [::]:80).
start with: grep -r '::' /etc/nginx/
Edit/add: One more thing that’s specific to this server is that the other domain https://bubblin.io is on HSTS preload list. So I have a directive on its nginx conf to force browsers to use strict https only for that site.
I think you may need to remove the comments (# symbol) from those two lines so the redirects below them only apply to http connections.
They may be applying to http then also to https.
hm. when I do that on nginx.conf of bookiza, test with sudo nginx -t fails on nginx of the other site.
nginx: [emerg] duplicate listen options for [::]:80 in /etc/nginx/sites-enabled/bubblin_production:7
nginx: configuration file /etc/nginx/nginx.conf test failed
I do have a server block on nginx conf for bubblin.io like so:
Now thinking how I could set up a listener for bookiza without making any changes on the conf. of the other domain. Phew. Quite a circle this has been.
It looks like you have a site config named "~", which is usually an editor backup filename. Is that intentional? What happens if you delete this file or move it elsewhere?