CentOS 8, Nginx, ipv6 server blocks certbot failure

gumshoenoir.com www.gumshoenoir.com

certbot --nginx --agree-tos --redirect --hsts --staple-ocsp --email redacted@redacted.net -d gumshoenoir.com -d www.gumshoenoir.com --debug-challenges

CentOS8

[root@centos8 ~]# nginx -v
nginx version: nginx/1.14.1
[root@centos8 ~]#

vps at Hostinger

yup, I have root via ssh. Not a control pannel.

[root@centos8 ~]# certbot --version
certbot 1.6.0
[root@centos8 ~]#

Background
FWIW I've regularly used certbot and acme.sh to issue certificates for several years. My previous experience is with CentOS 7 / Apache on a vps hosted by Dediserve and Fedora and Debian buster at home.

Now with a new additional hosting provider, Hostinger, I'm taking my first stab at CentOS 8, nginx with server blocks and ipv6.I haven't previously worked with nginx or ipv6.

Summary
Dinkin' around for a couple days, scratching my head and trying to talk myself out of thinking I'm seeing inconsistent results today I make some headway.

I feel the problem may have begun with an incorrect or incomplete ipv6 config of my nameservers, which may have allowed the very first attempt to succeed over ipv4. Subsequent attempts with the same syntax resulted in failures -- after I had corrected the ipv6. I use name.com to host my DNS

Eventually I setup an alias for .well-know/acme-challenge in the root nginx server which is labeled default_server. Although the certbot considered the server blocks DNS correct it appeared to try and retrieve data from the root server not the server block.

Before I put up the alias/redirect for .well-know I got errors like detail ... Invalid response (like a not found) after adding the redirect I get detail Error getting validation data

Success?
Finally, for debugging I added --debug-challenges

certbot --nginx --agree-tos --redirect --hsts --staple-ocsp --email redacted@redacted.net -d gumshoenoir.com -d www.gumshoenoir.com --debug-challenges

when certbot paused I couldn't find anything in .well-know/acme-challenges but after hitting CR to resume, the cert was granted! (huh?) I did that twice in a row with different virtual domain names.

But certbot renew --dry-run failed.

Logs
I have many. I can share more if required. Meanwhile this is the last run I did. From certbot renew --dry-run (shoot, how do I put scroll bars on this lloonngg log?)

the run for gumshoenoir.com. had the .well-known/acme-challenge redirect but still failed renewal.
Doh! topic wouldn't post to many characters. Check this pastebin. https://pastebin.com/erZqBVQw

gumshoenoir.com.conf

server {

# listen       [::]:80;

server_name  gumshoenoir.com www.gumshoenoir.com;

root         /var/www/gumshoenoir.com;



access_log   /var/log/nginx/gumshoenoir.com_access.log main;

    error_log    /var/log/nginx/gumshoenoir.com_error.log error;

index   index.html index.htm;

location / {

    # First attempt to serve request as file, then

    # as directory, then fall back to displaying a 404.

    try_files $uri $uri/ =404;

}

    location ^~ /.well-known/acme-challenge/ {

# default_type "text/plain";

        root         /usr/share/nginx/html/letsencrypt;

    }

    location = /.well-known/acme-challenge/ {

        return 404;

    }

listen [::]:443 ssl; # managed by Certbot

listen 443 ssl; # managed by Certbot

ssl_certificate /etc/letsencrypt/live/gumshoenoir.com/fullchain.pem; # managed by Certbot

ssl_certificate_key /etc/letsencrypt/live/gumshoenoir.com/privkey.pem; # managed by Certbot

include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot

ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

add_header Strict-Transport-Security "max-age=31536000" always; # managed by Certbot

ssl_trusted_certificate /etc/letsencrypt/live/gumshoenoir.com/chain.pem; # managed by Certbot

ssl_stapling on; # managed by Certbot

ssl_stapling_verify on; # managed by Certbot

}

server {

if ($host = www.gumshoenoir.com) {

    return 301 https://$host$request_uri;

} # managed by Certbot

if ($host = gumshoenoir.com) {

    return 301 https://$host$request_uri;

} # managed by Certbot

listen       80;

# listen       [::]:80;

listen [::]:80 ipv6only=on;

server_name  gumshoenoir.com www.gumshoenoir.com;

root         /var/www/gumshoenoir.com;



access_log   /var/log/nginx/gumshoenoir.com_access.log main;

    error_log    /var/log/nginx/gumshoenoir.com_error.log error;

index   index.html index.htm;

location / {

    # First attempt to serve request as file, then

    # as directory, then fall back to displaying a 404.

    try_files $uri $uri/ =404;

}

    location ^~ /.well-known/acme-challenge/ {

# default_type "text/plain";

        root         /usr/share/nginx/html/letsencrypt;

    }

    location = /.well-known/acme-challenge/ {

        return 404;

    }

}

1 Like

In short:
try to use webroot if you have a redirection for that folder in place.
You might want to remove these ipv6only=on block.

This might be happening because Let's Encrypt cached your validation results. (However, it's extremely short)

I saw these two blocks from one of your configurations, did you add that by yourself?

    	location ^~ /.well-known/acme-challenge/ {
	# default_type "text/plain";
 
        	root         /usr/share/nginx/html/letsencrypt;
    	}
 
    	location = /.well-known/acme-challenge/ {
        	return 404;
    	}

If so, you might want to use webroot instead of regular Nginx, because from the logs (thank you for providing that) certbot is adding a static response to the specific file on the end of the server block configuration (line 905 - 909) , which (I guess) probably being overridden by the specific Nginx redirection block.

P.S. Your redirection block can change to (replace root by alias)

    	location ^~ /.well-known/acme-challenge/ {
	# default_type "text/plain";
 
        	alias /usr/share/nginx/html/letsencrypt;
    	}

In this case, challenge files will be placed into /usr/share/nginx/html/letsencrypt instead of /usr/share/nginx/html/letsencrypt/.well-known/acme-challenge/somefiles

2 Likes

Thanks for the tip. Looks a lot cleaner.

1 Like

Wrapup / Solution

I switched to webroot and left the .well-know/acme-challenge location remap out. Works on my machine, as does the renewal. I believe the renewals were failing because ngix was running. the /etc/letsencrypt/renewal/gumshoenoir.com.conf was previously set to authenticator = nginx using webroot then authenticator = webroot which works.

In all the how-tos and actual letsencrypt documentation I never read that using the -nginx installer I had to stop the ngix server first. Still not positive on that, except that if I did systemctl stop ngix first then certbot --nginx --agree-tos --redirect --hsts --staple-ocsp --email jeffa@jhalbrecht.net -d gumshoenoir.com -d www.gumshoenoir.com then the certbot started a non systemd ngix and the certificate request was granted, the ngix started by the script was still running after certbot completed. On my other apache servers I was using certonly and manually configuring the ssl configuration.

Moved unchanging ssl to an include file. Tested at qualsys where I got A+ for ipv4 and ipv6.

Thanks to all that helped.

1 Like

You don't need to.
certbot is going to modify your Nginx configuration file and reload/restart your web server, which generally worked if there's no similar rule.

In your case, it's purely because the remap, which I think certbot didn't recognize and still added a rule after that remap (which didn't get used at all).

Congrats on figuring out a solution :grin:

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.