Certbot-auto renewal 404 error only on primary domain

lordcase · October 10, 2020, 8:05am

My domain is:
lordcase.dev

I ran this command:
sudo /usr/local/bin/certbot-auto --debug-challenges -v

It produced this output:
Traceback (most recent call last):
File "/opt/eff.org/certbot/venv/bin/letsencrypt", line 11, in
load_entry_point('letsencrypt==0.7.0', 'console_scripts', 'letsencrypt')()
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/main.py", line 15, in main
return internal_main.main(cli_args)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/main.py", line 1362, in main
return config.func(config, plugins)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/main.py", line 1108, in run
certname, lineage)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/main.py", line 122, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/client.py", line 418, in obtain_and_enroll_certificate
cert, chain, key, _ = self.obtain_certificate(domains)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/client.py", line 351, in obtain_certificate
orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/client.py", line 398, in _get_order_and_authorizations
authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
self._poll_authorizations(authzrs, max_retries, best_effort)
File "/opt/eff.org/certbot/venv/lib64/python3.6/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Some challenges have failed.

My web server is (include version):
nginx version: nginx/1.14.1

The operating system my web server runs on is (include version):
CentOS Linux release 8.2.2004 (Core)

My hosting provider, if applicable, is:
Digital Ocean

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.9.0

My issue is that if I try to renew my cert on my primary domain, it fails.
All the "subdomains" work perfectly well.

My primary domain is accessible via curl.

I have tried to run certbot-auto in debug mode, at the pont it stopped after inserting its scripts to nginx.conf, i checked the conf files and the rewrite was there:

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot

and lower the token as well. However, while this config was in effect i tried to curl the acme-challenge url and it als returned a 404. My idea is some of my config files are corrupted, but cannot find the problem.

I'll paste my nginx.conf here, please let me know if anything else is needed.

For more information on configuration, see:

* Official English Documentation: http://nginx.org/en/docs/

* Official Russian Documentation: http://nginx.org/ru/docs/

user nginx;
worker_processes auto;
error_log /var/log/nginx/error.log;
pid /run/nginx.pid;

Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.

include /usr/share/nginx/modules/*.conf;

events {
worker_connections 1024;
}

http {
log_format main '$remote_addr - $remote_user [$time_local] "$request" '
'$status $body_bytes_sent "$http_referer" '
'"$http_user_agent" "$http_x_forwarded_for"';

access_log  /var/log/nginx/access.log  main;
client_max_body_size 20M;
sendfile            on;
tcp_nopush          on;
tcp_nodelay         on;
keepalive_timeout   65;
types_hash_max_size 2048;

include             /etc/nginx/mime.types;
default_type        application/octet-stream;

# Load modular configuration files from the /etc/nginx/conf.d directory.
# See http://nginx.org/en/docs/ngx_core_module.html#include
# for more information.
include /etc/nginx/conf.d/*.conf;

server {
    listen       80 default_server;
    listen       [::]:80 default_server;
    server_name  lordcase.dev;
    root         /var/www/html;

    # Load configuration files for the default server block.
    include /etc/nginx/default.d/*.conf;

    location / {
    }

    error_page 404 /404.html;
        location = /40x.html {
    }

    error_page 500 502 503 504 /50x.html;
        location = /50x.html {
    }
}

}

Thanks in advance,
Krisz

griffin · October 10, 2020, 8:34am

Welcome to the Let's Encrypt Community, Krisz

_az · October 10, 2020, 8:36am

The nginx plugin doesn't write the challenge response to the filesystem, it modifies the configuration inline like so:

    server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


        listen       80 default_server;
        listen       [::]:80 default_server;
        server_name  lordcase.dev;
        root         /usr/share/nginx/html;

        # Load configuration files for the default server block.
        include /etc/nginx/default.d/*.conf;

        location / {
        }

        error_page 404 /404.html;
            location = /40x.html {
        }

        error_page 500 502 503 504 /50x.html;
            location = /50x.html {
        }
    location = /.well-known/acme-challenge/60Ys81Bm1savmMCGVRyAjQzqkipx31EZU_fXswxgL3U{default_type text/plain;return 200 60Ys81Bm1savmMCGVRyAjQzqkipx31EZU_fXswxgL3U.-umPE5-6LMh6IO5nhSOsls10bJENdEdGT3TJDbfjo-w;} # managed by Certbot

}

Checking the webroot won't necessarily reveal the problem in such cases.

Could you try --debug-challenges again, and while Certbot is paused:

sudo nginx -T > /tmp/nginx-all.conf

and upload that somewhere or post it here?

lordcase · October 10, 2020, 8:46am

thx for the quick reply _az!

Yes, i have tried to curl the path that crtbot inserted into the conf but got a 404 sorry if it wasn't clearly put.

However i ran what you asked and i have a ffeling it doesn't look good
nginx: [warn] conflicting server name "beerbro.lordcase.dev" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "bogyo.lordcase.dev" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "nyul.lordcase.dev" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "lordcase.dev" on 0.0.0.0:80, ignored

nginx: [warn] conflicting server name "lordcase.dev" on [::]:80, ignored

nginx: the configuration file /etc/nginx/nginx.conf syntax is ok

nginx: configuration file /etc/nginx/nginx.conf test is successful

So the thing is i have moved my webroots to /var/www from the original recommendation but maybe did it in steps and not very proficiently

griffin · October 10, 2020, 8:52am

@_az

Learn something new every day.

@lordcase

I believe _az made the comment below for my benefit, so no worries.

_az · October 10, 2020, 8:54am

Those conflicting server names explain your error, so we don't need to see the rest of your configuration.

What's happening is that Certbot is adding the challenge response into only one of the lordcase.dev servers, as you saw yourself.

Only one of these lordcase.dev servers actually applies when serving traffic.

So when you're getting the 404, it's because nginx is looking at the other virtualhost.

What you should do is clean your overall config up to get rid of the duplicate servers, and you'll be good.

lordcase · October 10, 2020, 8:55am

ahhh i didn't catch your pre-edit

JuergenAuer · October 10, 2020, 8:55am

Hi @lordcase

you have duplicated definitions.

One combination port + domain -> multiple server blocks.

Remove all duplicated definitions.

Certbot picks the wrong server-block.

lordcase · October 10, 2020, 8:57am

awesome, thanks! would you have any recommendations which one to keep?

do I even need the one in .donf? can't i just create one main one for the primary domain?

Or if there are benefits, let's turn it around? what to move into conf-d/lordcase.dev.conf?

But if this is well beyond this boards scope, please just pint me to the right directin if you would

in the meantime I'll do some cleaning up and try the results

_az · October 10, 2020, 8:58am

I think it would make sense to get rid of the server {} in nginx.conf, and to keep the other one.

If you don't want to delete it, you could just change it so it says server_name _.

Rationale: this one doesn't contain any custom configuration, it's the stock CentOS 8 default virtualhost. So the other server for that domain is probably the relevant one.

lordcase · October 10, 2020, 9:02am

i did exactly that and it worked like a charm!

Thanks for everyone, awesome community u guys have here.

Sorry for not being able to pick 2 solutions, JuergenAuer's was just as precise, but _az you did all the heavy lifting walking me through it

Thx again for all!

Krisz

JuergenAuer · October 10, 2020, 9:04am

No problem.

Your question is a good sample that sharing one (the wrong) config file may not help to see the error.

Instead, the output of nginx -T shows the problem much better.

system · November 9, 2020, 9:04am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Certbot auto renewal problem Help	2	1568	December 8, 2021
Certbot-auto renew failed Help	2	1469	May 4, 2019
Renew problem on AWS instance Help	31	771	October 21, 2020
Certbot-auto 0.24.0 borked per issue 5369 Server	7	1104	June 3, 2018
Certbot renewal was working, but broke at some point Help	3	751	April 30, 2018

Certbot-auto renewal 404 error only on primary domain

For more information on configuration, see:

* Official English Documentation: http://nginx.org/en/docs/

* Official Russian Documentation: http://nginx.org/ru/docs/

Load dynamic modules. See /usr/share/doc/nginx/README.dynamic.

Related topics