Renewal is failing


#1

I had a working setup until about a week ago. Something changed but I’m unable to point to it:

$ sudo certbot renew --dry-run

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/bookiza.io.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert is due for renewal, auto-renewing...

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

http-01 challenge for bubbl.in

http-01 challenge for bubblin.io

http-01 challenge for www.bubbl.in

http-01 challenge for www.bubblin.io

http-01 challenge for bookiza.io

http-01 challenge for www.bookiza.io

Waiting for verification...

Cleaning up challenges

Attempting to renew cert (bookiza.io) from /etc/letsencrypt/renewal/bookiza.io.conf produced an unexpected error: Failed authorization procedure. www.bookiza.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.bookiza.io/.well-known/acme-challenge/9YHK8HkRZrdirbDJrrC9kp--5z-8ampqHqGcCrcI190: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>", bookiza.io (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://bookiza.io/.well-known/acme-challenge/_Sw3GRDhf__OUm1QBxDGkaygx8gh7ztKoWuJbOi1wwA: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body bgcolor=\"white\">\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>". Skipping.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/www.bubbl.in.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert not due for renewal, but simulating renewal for dry run

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

tls-sni-01 challenge for www.bubbl.in

Waiting for verification...

Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

new certificate deployed with reload of nginx server; fullchain is

/etc/letsencrypt/live/www.bubbl.in/fullchain.pem

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/bubblin.io.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert not due for renewal, but simulating renewal for dry run

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

tls-sni-01 challenge for bubblin.io

tls-sni-01 challenge for www.bubblin.io

Waiting for verification...

Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

new certificate deployed with reload of nginx server; fullchain is

/etc/letsencrypt/live/bubblin.io/fullchain.pem

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/www.bubblin.io.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert not due for renewal, but simulating renewal for dry run

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

tls-sni-01 challenge for www.bubblin.io

Waiting for verification...

Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

new certificate deployed with reload of nginx server; fullchain is

/etc/letsencrypt/live/www.bubblin.io/fullchain.pem

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Processing /etc/letsencrypt/renewal/bubbl.in.conf

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

Cert not due for renewal, but simulating renewal for dry run

Plugins selected: Authenticator nginx, Installer nginx

Renewing an existing certificate

Performing the following challenges:

tls-sni-01 challenge for bubbl.in

Waiting for verification...

Cleaning up challenges

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

new certificate deployed with reload of nginx server; fullchain is

/etc/letsencrypt/live/bubbl.in/fullchain.pem

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

The following certs could not be renewed:

/etc/letsencrypt/live/bookiza.io/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

** DRY RUN: simulating 'certbot renew' close to cert expiry

** (The test certificates below have not been saved.)

The following certs were successfully renewed:

/etc/letsencrypt/live/www.bubbl.in/fullchain.pem (success)

/etc/letsencrypt/live/bubblin.io/fullchain.pem (success)

/etc/letsencrypt/live/www.bubblin.io/fullchain.pem (success)

/etc/letsencrypt/live/bubbl.in/fullchain.pem (success)

The following certs could not be renewed:

/etc/letsencrypt/live/bookiza.io/fullchain.pem (failure)

** DRY RUN: simulating 'certbot renew' close to cert expiry

** (The test certificates above have not been saved.)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -

1 renew failure(s), 0 parse failure(s)

**IMPORTANT NOTES:**

 - The following errors were reported by the server:

Domain: www.bookiza.io

Type: unauthorized

Detail: Invalid response from

http://www.bookiza.io/.well-known/acme-challenge/9YHK8HkRZrdirbDJrrC9kp--5z-8ampqHqGcCrcI190:

"<html>\r\n<head><title>404 Not Found</title></head>\r\n<body

bgcolor=\"white\">\r\n<center><h1>404 Not

Found</h1></center>\r\n<hr><center>"

Domain: bookiza.io

Type: unauthorized

Detail: Invalid response from

http://bookiza.io/.well-known/acme-challenge/_Sw3GRDhf__OUm1QBxDGkaygx8gh7ztKoWuJbOi1wwA:

"<html>\r\n<head><title>404 Not Found</title></head>\r\n<body

bgcolor=\"white\">\r\n<center><h1>404 Not

Found</h1></center>\r\n<hr><center>"

To fix these errors, please make sure that your domain name was

entered correctly and the DNS A/AAAA record(s) for that domain

contain(s) the right IP address.

So it’s working fine for one domain but not for the other. I can provide the contents of nginx for Bookiza here if that helps. DNS setup looks okay and nothing has changed on it fwiw.


First certificates via webroot method - need guidance
#2

A few points:

  • could you copy/paste (as “preformatted text” with the [</>] button) the content of /etc/letsencrypt/renewal/bookiza.io.conf?
  • The bookiza.io cert contains 6 hostnames, three base domains (bubbl.in, bubblin.io and bookiza.io and their www subdomains). However, you are also getting separate certificates for bubbl.in, bubblin.io and separate certificates for their www subdomain. Is there a good reason to get “double”, or even “triple” certificates for those base domains?
  • For those separate certificates: you’re using the tls-sni-01 challenge, which is being deprecated in 2019.

#3

renew_before_expiry = 30 days

version = 0.26.1

archive_dir = /etc/letsencrypt/archive/bookiza.io

cert = /etc/letsencrypt/live/bookiza.io/cert.pem

privkey = /etc/letsencrypt/live/bookiza.io/privkey.pem

chain = /etc/letsencrypt/live/bookiza.io/chain.pem

fullchain = /etc/letsencrypt/live/bookiza.io/fullchain.pem

Options used in the renewal process

[renewalparams]

server = https://acme-v02.api.letsencrypt.org/directory

authenticator = nginx

account = 5269fe370dd90d04e2202ca26d61c139

installer = nginx

pref_challs = http-01,

~


#4

I remember running sudo certbot --nginx --preferred-challenges http a few weeks ago to not use tls-sni-01, but it appears that older installations are still lying around.

This server is mainly to host https://bubblin.io (a rails application handling www, root and other request situations) but I also set up a static site (https://bookiza.io) later on, so the installation may not one of the cleanest. I don’t need to fetch double or triple instances of any of the certificates, how do I go about cleaning up now?


#5

The redirections are messed up:
http://www.bookiza.io/.well-known/acme-challenge/test
redirects to:
https://www.bookiza.io/.well-known/acme-challenge/test/.well-known/acme-challenge/test


#6

How to fix it? I wonder why would redirections be messed up at all, I used the certbot through and through since the very beginning.


#7

please show the vhost config for bookiza.io
or wherever the redrection is being done.


#8

I can’t find these redirects. Checked with my own tool https://check-your-website.server-daten.de/?q=bookiza.io

There are redirects http -> https, these ends with a (correct) http status 404.


#9

A post was split to a new topic: Cert not yet due for renewal


#10

I’m using nginx for bookiza. It has the following configuration:

server {
    if ($host = www.bookiza.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = bookiza.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot




  #listen 80 default_server;
  #listen [::]:80 default_server ipv6only=on;
  server_name bookiza.io www.bookiza.io;
 # return 404; # managed by Certbot

}

Just FYI, the site has just one static index page to serve.


#11

I now see the name resolves to IPv4 and IPv6 addresses.
LE will prefer IPv6 when present.
You must ensure that IPV6 is working or remove the AAAA record from DNS:
Name: www.bookiza.io
Addresses: 2600:3c00::f03c:91ff:fe56:3901
50.116.16.25

IPv4 returns:
wget http://www.bookiza.io/.well-known/acme-challenge/test
–2018-11-26 10:11:22-- http://www.bookiza.io/.well-known/acme-challenge/test
Resolving www.bookiza.io (www.bookiza.io)… 50.116.16.25, 2600:3c00::f03c:91ff:fe56:3901
Connecting to www.bookiza.io (www.bookiza.io)|50.116.16.25|:80… connected.
HTTP request sent, awaiting response… 301 Moved Permanently
Location: https://www.bookiza.io/.well-known/acme-challenge/test [following]
–2018-11-26 10:11:22-- https://www.bookiza.io/.well-known/acme-challenge/test
Connecting to www.bookiza.io (www.bookiza.io)|50.116.16.25|:443… connected.
HTTP request sent, awaiting response… 404 Not Found
2018-11-26 10:11:22 ERROR 404: Not Found.

But IPv6 returns:
wget http://www.bookiza.io/.well-known/acme-challenge/test --no-check-certificate
–2018-11-26 15:16:23-- http://www.bookiza.io/.well-known/acme-challenge/test
Resolving www.bookiza.io (www.bookiza.io)… 2600:3c00::f03c:91ff:fe56:3901, 50.116.16.25
Connecting to www.bookiza.io (www.bookiza.io)|2600:3c00::f03c:91ff:fe56:3901|:80… connected.
HTTP request sent, awaiting response… 301 Moved Permanently
Location: https://www.bookiza.io/.well-known/acme-challenge/test/.well-known/acme-challenge/test [following]
–2018-11-26 15:16:24-- https://www.bookiza.io/.well-known/acme-challenge/test/.well-known/acme-challenge/test
Connecting to www.bookiza.io (www.bookiza.io)|2600:3c00::f03c:91ff:fe56:3901|:443… connected.
WARNING: cannot verify www.bookiza.io’s certificate, issued by ‘CN=Let’s Encrypt Authority X3,O=Let’s Encrypt,C=US’:
Unable to locally verify the issuer’s authority.
HTTP request sent, awaiting response… 404 Not Found
2018-11-26 15:16:24 ERROR 404: Not Found.


#12

It would seem that your tool doesn’t check both IPv4 and IPv6 independently when present.
Or the misdirection is only related to IPv6 and the acme-challenge folder…


#13

Interesting. So IPv6 appears to be configured correctly on my DNS manager. I have a quad-A record for Bookiza that’s mapped to the ipv6-ip provided by Linode. This is working fine for the other domain bubblin.io! To experiment I just added a blank quad-A record for bookiza (apart from *, www and @) on the DNS manager to see if that’s something it’s looking for.

Edit: It’s the same as @ hostname value so no changes were effected on the DNS of bookiza.


#14

The IPv6 may be pointing to the correct system.
The problem is in how the redirection is being handled for IPv6.
Which is different than how it is being handled for IPv4.
Please find and show the vhost that listens to IPv6 (::80 or [::]:80).
start with:
grep -r '::' /etc/nginx/


#15
server {
  root /var/www/bookiza.io;
  index index.html;
  server_name bookiza.io www.bookiza.io;

  location / {
         try_files $uri $uri/ =404;
  }

    location ~* \.(jpg|jpeg|png|gif|ico)$ {
       expires 30d;
    }

    location ~* \.(css|js)$ {
       expires 7d;
    }

    listen [::]:443 http2 ssl ipv6only=on; # managed by Certbot
    listen 443 http2 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/bookiza.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/bookiza.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

   gzip on;
   gzip_types application/javascript image/* text/css;
   gunzip on;

}

server {
    if ($host = www.bookiza.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = bookiza.io) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


  #listen 80 default_server;
  #listen [::]:80 default_server ipv6only=on;
  server_name bookiza.io www.bookiza.io;
                                                                                                                                            
}

This is pretty much all of it. For the other domain I have the following:

$ grep -r '::' /etc/nginx/
/etc/nginx/sites-available/~:    listen [::]:80 http2;
/etc/nginx/sites-available/~:  listen [::]:443 ssl http2;
/etc/nginx/sites-available/~:  listen [::]:443 ssl http2; # managed by Certbot
/etc/nginx/sites-available/default:#	listen [::]:80 default_server;
/etc/nginx/sites-available/default:#	# listen [::]:443 ssl default_server;
/etc/nginx/sites-available/default:##	listen [::]:80;
/etc/nginx/sites-available/bubblin_production:    listen [::]:80 ipv6only=on;
/etc/nginx/sites-available/bubblin_production:  listen [::]:443 ssl http2;
/etc/nginx/sites-available/bubblin_production:  listen [::]:443 ssl http2;
/etc/nginx/sites-available/bookiza_production:    listen [::]:443 http2 ssl ipv6only=on; # managed by Certbot
/etc/nginx/sites-available/bookiza_production:  #listen [::]:80 default_server ipv6only=on;

Edit/add: One more thing that’s specific to this server is that the other domain https://bubblin.io is on HSTS preload list. So I have a directive on its nginx conf to force browsers to use strict https only for that site.


#17

I think you may need to remove the comments (# symbol) from those two lines so the redirects below them only apply to http connections.
They may be applying to http then also to https.


#18

Yep, currently I can only check global, not ipv4/v6 - specific.

Yes, a redirect only via ipv6.


#19

hm. when I do that on nginx.conf of bookiza, test with sudo nginx -t fails on nginx of the other site.

nginx: [emerg] duplicate listen options for [::]:80 in /etc/nginx/sites-enabled/bubblin_production:7
nginx: configuration file /etc/nginx/nginx.conf test failed

I do have a server block on nginx conf for bubblin.io like so:

server {
    listen 80;
    listen [::]:80 ipv6only=on;
    server_name bubblin.io www.bubblin.io;
    rewrite ^(.*) https://$host$1$request_uri permanent;
}

Now thinking how I could set up a listener for bookiza without making any changes on the conf. of the other domain. Phew. Quite a circle this has been.


#20

Something is not right.
The two server_names are different - they should not be affecting each other.
You should review the entire config:
nginx -T


#21

It looks like you have a site config named “~”, which is usually an editor backup filename. Is that intentional? What happens if you delete this file or move it elsewhere?