Renew failure. Challenge Failed (nginx problem?)

[azazael]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

azabot.stream

I ran this command:

sudo certbot --nginx

It produced this output:

Which names would you like to activate HTTPS for?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: azabot.stream
2: unifi.azabot.stream
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate numbers separated by commas and/or spaces, or leave input
blank to select all options shown (Enter 'c' to cancel): 1
Attempting to parse the version 1.9.0 renewal configuration file found at /etc/letsencrypt/renewal/azabot.stream.conf with version 0.40.0 of Certbot. This might not work.
Attempting to parse the version 1.11.0 renewal configuration file found at /etc/letsencrypt/renewal/unifi.azabot.stream.conf with version 0.40.0 of Certbot. This might not work.
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for azabot.stream
nginx: [warn] conflicting server name "azabot.stream" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "azabot.stream" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "azabot.stream" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "azabot.stream" on [::]:80, ignored
Waiting for verification...
Challenge failed for domain azabot.stream
http-01 challenge for azabot.stream
Cleaning up challenges
nginx: [warn] conflicting server name "azabot.stream" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "azabot.stream" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "azabot.stream" on 0.0.0.0:80, ignored
nginx: [warn] conflicting server name "azabot.stream" on [::]:80, ignored
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: azabot.stream
   Type:   unauthorized
   Detail: Invalid response from
   http://azabot.stream/.well-known/acme-challenge

My web server is (include version):

nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version):

Ubuntu 20.04.1

My hosting provider, if applicable, is:

BuyVM

I can login to a root shell on my machine (yes or no, or I don't know):

Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 0.40.0

I am guessing that this is an issue of me not having my nginx config setup correctly and it not being able to access the challenge that it is creating. To that end here is the default sites config that I have. (I am still learning Nginx so I am sure this is ugly and there are things I can do better, feel free to help me learn)

server {
listen 80 default_server;
listen [::]:80 default_server;

root /var/www/html;

    index index.php index.html index.htm index.nginx-debian.html;

    server_name azabot.stream;

    location / {
            try_files $uri $uri/ =404;
    }

    location ~ \.php$ {
            include snippets/fastcgi-php.conf;
            fastcgi_pass unix:/run/php/php7.4-fpm.sock;
    }

    location ~ /\.ht {
            deny all;
    }
}
server {

    root /var/www/html;

    index index.html index.htm index.nginx-debian.html;
    server_name azabot.stream; # managed by Certbot

    location / {
            try_files $uri $uri/ =404;
    }

    location /twitch/eventsub/callback {
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_http_version 1.1;
        proxy_pass https://azabot.stream:8054/twitch/eventsub/callback;
    }

    location /azabot {
        proxy_set_header Host $host;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "Upgrade";
        proxy_http_version 1.1;
        proxy_pass https://azabot.stream:8054;
    }

listen [::]:443 ssl ipv6only=on; # managed by Certbot
listen 443 ssl; # managed by Certbot
ssl_certificate /etc/letsencrypt/live/azabot.stream/fullchain.pem; # managed by Certbot
ssl_certificate_key /etc/letsencrypt/live/azabot.stream/privkey.pem; # managed by Certbot
include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}

server {
if ($host = azabot.stream) {
    return 301 https://$host$request_uri;
} # managed by Certbot

listen 80 ;
listen [::]:80 ;
server_name azabot.stream;
return 404; # managed by Certbot
}

[_az]

What does this output:

sudo grep -R "server_name azabot" /etc/nginx

[azazael]

@_az

sudo grep -R "server_name azabot" /etc/nginx

/etc/nginx/sites-available/default: server_name azabot.stream;
/etc/nginx/sites-available/default: server_name azabot.stream; # managed by Certbot
/etc/nginx/sites-available/default: server_name azabot.stream;
/etc/nginx/sites-available/azabot.stream: server_name azabot.stream azabot.stream;
/etc/nginx/sites-enabled/default: server_name azabot.stream;
/etc/nginx/sites-enabled/default: server_name azabot.stream; # managed by Certbot
/etc/nginx/sites-enabled/default: server_name azabot.stream;
/etc/nginx/sites-enabled/azabot.stream: server_name azabot.stream azabot.stream;

[_az]

Thanks. The problem here is that you have duplicate virtualhosts in your nginx configuration.

Basically, you have multiple server blocks with port 80 + server_name azabot.stream.

You should aim to only have one such virtualhost, because having multiple is a conflict. That conflict also confuses Certbot.

If you look at /etc/nginx/sites-enabled/default, here is the first such virtualhost:

@azazael

server {
listen 80 default_server;
listen [::]:80 default_server;

root /var/www/html;

    index index.php index.html index.htm index.nginx-debian.html;

    server_name azabot.stream;

    location / {
            try_files $uri $uri/ =404;
    }

    location ~ \.php$ {
            include snippets/fastcgi-php.conf;
            fastcgi_pass unix:/run/php/php7.4-fpm.sock;
    }

    location ~ /\.ht {
            deny all;
    }
}

and here is the second one:

@azazael

server {
if ($host = azabot.stream) {
    return 301 https://$host$request_uri;
} # managed by Certbot

listen 80 ;
listen [::]:80 ;
server_name azabot.stream;
return 404; # managed by Certbot
}

There is also a third one in /etc/nginx/sites-enabled/azabot.stream.

Of these 3, you should keep only 1. Probably the one that does the HTTPS redirect is the one you want to keep, but that's for you to judge.

Once you've gotten rid of 2 of them, you can test to see whether nginx still complains about conflicts with:

sudo nginx -t

and if it doesn't, you can try the renewal again.

[azazael]

Makes perfect sense. I'll give it a shot later tonight and let you know what I find, thanks.

[azazael]

Actually that was faster than I thought to do

The first one was easy to remove because I foolish provided the server name there instead of the IP which was the default.

After that I just commented out the one in sites-enabled, tested nginx, restarted nginx, and then ran certbot.

As always thanks for the help for @_az