Certificate installation worked, renewal dry-run fails

mnordhoff · May 8, 2018, 5:00pm

“certbot --nginx” temporarily modifies the Nginx configuration to serve the challenge file from a certain Certbot directory, then reverts the configuration change.

Your Nginx configuration shouldn’t normally need any references to /.well-known/acme-challenge/ or anything. Certbot’s supposed to be able to figure everything out on its own. If it can’t, it’s a bug. (Though it might not be a high priority to fix all bugs parsing egregiously weird configurations.)

In this case it’s not succeeding… So the questions are why, and what needs to be done to fix it. I’m suspicious of the port thing. There’s nothing wrong with doing that, and I think Cerbot’s current implementation is intended to support it, but it seems to be the only slightly unusual aspect of the configuration.

If you run Certbot with --debug-challenges, it will stop in the middle to let you examine the configuration changes. (I’m not certain renew supports that option, though.)

It may help to examine the configuration and see if anything looks wrong. Or run nginx -T and pastebin everything.

Edit: Also, what’s in the /etc/letsencrypt/renewal/ configuration file?

sicHiRsch · May 8, 2018, 10:44pm

I ran sudo nginx -t

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

I ran sudo certbot renew --dry-run --debug-challenges -v, here is the output:

https://pastebin.com/4dJ6RAwc

# renew_before_expiry = 30 days
version = 0.22.2
archive_dir = /etc/letsencrypt/archive/fritz.debler.net
cert = /etc/letsencrypt/live/fritz.debler.net/cert.pem
privkey = /etc/letsencrypt/live/fritz.debler.net/privkey.pem
chain = /etc/letsencrypt/live/fritz.debler.net/chain.pem
fullchain = /etc/letsencrypt/live/fritz.debler.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = nginx
installer = nginx
account = b33674c6f1afb1fe159c932f4b30e58b

mnordhoff · May 8, 2018, 10:45pm

nginx -T displays the configuration.

sicHiRsch · May 8, 2018, 10:49pm

sudo nginx -T
nginx: invalid option: "T"

not for my installation though...

EDIT: I see, that option is available as of nginx 1.9.2, mine is 1.4.6

mnordhoff · May 8, 2018, 11:02pm

While Certbot is waiting, can you check how it's modified the Nginx configuration?

sicHiRsch · May 8, 2018, 11:05pm

it doesn’t wait. It just pauses for a few seconds with this message

-------------------------------------------------------------------------------
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
-------------------------------------------------------------------------------

and then just continues

I’ll try to catch it at that point, but gotta run now. Thanks so far.

rg305 · May 9, 2018, 12:34am

I say built the -T option… by hand.
Start with the initial nginx.conf - post that.
grep it for includes - post those.
grep those for possible even more includes - post those as well
until all included files have been posted.

sicHiRsch · May 9, 2018, 12:44am

cat /etc/nginx/nginx.conf

user www-data;
    worker_processes 4;
    pid /run/nginx.pid;

    events {
            worker_connections 768;
            # multi_accept on;
    }

    http {

            ##
            # Basic Settings
            ##

            sendfile on;
            tcp_nopush on;
            tcp_nodelay on;
            keepalive_timeout 65;
            types_hash_max_size 2048;
            # server_tokens off;

            # server_names_hash_bucket_size 64;
            # server_name_in_redirect off;

            include /etc/nginx/mime.types;
            default_type application/octet-stream;

            ##
            # Logging Settings
            ##

            access_log /var/log/nginx/access.log;
            error_log /var/log/nginx/error.log;

            ##
            # Gzip Settings
            ##

            gzip on;
            gzip_disable "msie6";

            # gzip_vary on;
            # gzip_proxied any;
            # gzip_comp_level 6;
            # gzip_buffers 16 8k;
            # gzip_http_version 1.1;
            # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

            ##
            # nginx-naxsi config
            ##
            # Uncomment it if you installed nginx-naxsi
            ##

            #include /etc/nginx/naxsi_core.rules;

            ##
            # nginx-passenger config
            ##
            # Uncomment it if you installed nginx-passenger
            ##

            #passenger_root /usr;
            #passenger_ruby /usr/bin/ruby;

            ##
            # Virtual Host Configs
            ##

            include /etc/nginx/conf.d/*.conf;
            include /etc/nginx/sites-enabled/*;
    }


    #mail {
    #       # See sample authentication script at:
    #       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
    #
    #       # auth_http localhost/auth.php;
    #       # pop3_capabilities "TOP" "USER";
    #       # imap_capabilities "IMAP4rev1" "UIDPLUS";
    #
    #       server {
    #               listen     localhost:110;
    #               protocol   pop3;
    #               proxy      on;
    #       }
    #
    #       server {
    #               listen     localhost:143;
    #               protocol   imap;
    #               proxy      on;
    #       }
    #}

/etc/nginx/conf.d/*.conf has no conf files.

under sites-enabled I have one file, the one we have been playing with before

server {
        listen 81 default_server;
        server_name fritz.debler.net 192.168.178.36;
        root /var/www/html;
        index index.html index.htm;
        location ~ /.well-known {
                allow all;
        }
}

sicHiRsch · May 9, 2018, 12:49am

http {
include /etc/letsencrypt/le_http_01_cert_challenge.conf;
server_names_hash_bucket_size 128;

Compared to the above posted nginx.conf, these two lines are added during the --debug-challenges --dry-run

that le_http_01_cert_challenge.conf file contains the following:

server{listen 80;server_name fritz.debler.net;root /var/lib/letsencrypt/http_01_nonexistent;location = /.well-known/acme-challenge/NN-DLFKe0WL8_QU-bBhcv_frl0I8BQF9XhAePc4AAeA{default_type text/plain;return 200 NN-DLFKe0WL8_QU-bBhcv_frl0I8BQF9XhAePc4AAeA._9GQoHhPdssiGAgP3PeKvc9kbcJ74L7iDwVRMpYhQNs;}}

is this where we might be running into the port problems?

mnordhoff · May 9, 2018, 1:27am

Yes – since it’s adding a whole virtual host on port 80, but the port forwarding causes the requests to go to your other port 81 virtual host, which returns 404 Not Found.

Theoretically, you would perhaps want to run:

certbot --nginx --http-01-port 81 -d fritz.debler.net

Or:

certbot --nginx --http-01-port 81 --preferred-challenges http-01 -d fritz.debler.net

But now I’m confused… Why were you able to get a production certificate? At least 4 of them, in fact.

One of the (production) rate limits is 5 identical certificates per week, so it’s a bad idea to do something that might issue a new production certificate this week.

sicHiRsch · May 9, 2018, 1:32am

That is what I have been wondering all along. I am able to get a certificate, so somehow cerbot manages to pass the challenges there. That is why I repeated it a few times. But during the renewal process it fails…

mnordhoff · May 9, 2018, 1:37am

Don’t issue another production certificate now, but do you have the console output or letsencrypt.log from one of the times you issued one? In particular, the first time? It should be in /var/log/letsencrypt/letsencrypt.log or one of the older files in /var/log/letsencrypt/.

The question is what to do next… Under the circumstances, I’m not entirely certain, and I don’t want to guess wrong.

Edit:

I have an idea, but it’s not necessarily a wise idea:

Try adding “http01_port = 81” to the renewalparams section of /etc/letsencrypt/renewal/fritz.debler.net.conf – since it’s the last section of the file, appending it to the end is fine – and running “certbot renew --dry-run” again.

I think it will work. It probably won’t permanently damage anything.

sicHiRsch · May 9, 2018, 1:45am

I can't really make much sense of it, but you might.

http://fritz.debler.net/letsencrypt.log

sicHiRsch · May 9, 2018, 1:48am

Yay, it did in fact work! Thanks.

certbot renew --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/fritz.debler.net.conf
-------------------------------------------------------------------------------
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for fritz.debler.net
Waiting for verification...
Cleaning up challenges

-------------------------------------------------------------------------------
new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/fritz.debler.net/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/fritz.debler.net/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------

mnordhoff · May 9, 2018, 1:50am

Yay!

The question is whether it will work 60 days from now when you need to renew the production certificate… I don’t want to offer any false assurances.

sicHiRsch · May 9, 2018, 1:52am

We will see, I guess I’ll be back in 60 days to report

rg305 · May 9, 2018, 1:59am

This topic/thread will be closed in 30 days.
modify the renewal conf file to be larger than 60 (75?)
and it should try before then
and we can get to the bottom of this quicker.
Of course you would put it back to 30 once we are done testing/fixing

sicHiRsch · May 9, 2018, 2:46am

like this?

# renew_before_expiry = 75 days

does that hash work like a comment here, or does it need to stay?

EDIT:
Another related question. I see there is a cron job for renewal in /etc/cron.d/certbot
Do I need to add a post-hook script to restart nginx, or does it already restart it because nginx is mentioned in the renewal conf?

rg305 · May 9, 2018, 3:17am

I think yes the # is a comment.
Loads with 30, which is the default.
So it’s shown there just for moments like these…

sicHiRsch · May 24, 2018, 3:58am

OK, just to complete this thread, autorenewal has just worked last night. I am now changing the config back to renew 30 days before expiry.

The fix was adding http01_port = 81 to the renewal config.

Thanks guys

Topic		Replies	Views
Cert Renew Fail Help	19	1305	February 22, 2021
Challenges failed for renewal Help	27	3255	July 30, 2022
Unable to renew cert using nginx plugin, fresh cert creation succeeds Help	9	5967	November 8, 2019
Creating certificates with --nginx works. Renew or certonly doesn't Help	82	8406	March 18, 2019
Certbot renew failing Help	24	1943	February 14, 2021

Certificate installation worked, renewal dry-run fails

Related topics