Certbot Nginx Authentication Failed for ai.chaoss.io - 404 Response

I am encountering an issue while trying to obtain an SSL certificate for my domain ai.chaoss.io using Certbot with the Nginx plugin. When I attempt to request the certificate, I receive the following error message:

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
Domain: ai.chaoss.io
Type: unauthorized
Detail: 67.6.94.219: Invalid response from https://ai.chaoss.io/.well-known/acme-challenge/5FbJtD7b_4og0nMJz7T1rlvZ6mrYw6pYCMFJMDS45O8: 404
Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Steps Taken:

DNS Resolution: I checked the DNS resolution for ai.chaoss.io, and it correctly points to the server with IP address 67.6.94.219:

nslookup ai.chaoss.io
Server: 127.0.0.53
Address: 127.0.0.53#53
Non-authoritative answer:
ai.chaoss.io canonical name = chaoss.tv.
Name: chaoss.tv
Address: 67.6.94.219
Name: chaoss.tv
Address: 64:ff9b::4306:5edb
Certbot Command: I ran the following command:

sudo certbot --nginx -d ai.chaoss.io
Issue: The error suggests that Certbot cannot verify the domain via Nginx because it receives a 404 error when trying to access the temporary challenge file at https://ai.chaoss.io/.well-known/acme-challenge/.

Nginx Configuration: I have verified that Nginx is running and correctly serving the site at ai.chaoss.io, but the temporary files needed for the verification challenge aren't being served.

Challenge URL Issue: The challenge URL seems to return a 404, indicating that the file is either not accessible or not placed in the right location by Certbot.

Hello @Akshatb2006,

Please show the output of each of the following commands

  • sudo certbot --version
  • sudo certbot certificates
  • sudo nginx -T that is a capital T
2 Likes

akshat-baranwal@AKB:~/Github/augur$ sudo certbot --version
[sudo] password for akshat-baranwal:
certbot 2.9.0
akshat-baranwal@AKB:~/Github/augur$ sudo certbot certificates
Saving debug log to /var/log/letsencrypt/letsencrypt.log


No certificates found.


akshat-baranwal@AKB:~/Github/augur$ sudo nginx -T
2025/05/13 23:25:01 [warn] 48441#48441: conflicting server name "ai.chaoss.io" on 0.0.0.0:80, ignored
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
error_log /var/log/nginx/error.log;
include /etc/nginx/modules-enabled/*.conf;

events {
        worker_connections 768;
        # multi_accept on;
}

http {

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;
        # server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # SSL Settings
        ##

        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;

        ##
        # Logging Settings
        ##

        access_log /var/log/nginx/access.log;

        ##
        # Gzip Settings
        ##

        gzip on;

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # Virtual Host Configs
        ##

        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}


#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
#
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/avif                                       avif;
    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/wasm                                 wasm;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/ogg                                        ogv;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-matroska                                 mkv;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/sites-enabled/augur:
server {
    listen 80;
    server_name ai.chaoss.io;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        if ($request_uri !~ "^/\.well-known/acme-challenge/") {
            return 301 https://$host$request_uri;
        }
    }
}
server {
        server_name  ai.chaoss.io;

        location /api/unstable/ {
                proxy_pass http://ai.chaoss.io:5038;
                proxy_set_header Host $host;
        }

        location / {
                proxy_pass http://127.0.0.1:5038;
        }

        location /settings {

                proxy_read_timeout 800;
                proxy_connect_timeout 800;
                proxy_send_timeout 800;
        }

        error_log /var/log/nginx/augurview.osshealth.error.log;
        access_log /var/log/nginx/augurview.osshealth.access.log;

}

# configuration file /etc/nginx/sites-enabled/default:
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
        listen 80 default_server;
        listen [::]:80 default_server;

        # SSL configuration
        #
        # listen 443 ssl default_server;
        # listen [::]:443 ssl default_server;
        #
        # Note: You should disable gzip for SSL traffic.
        # See: https://bugs.debian.org/773332
        #
        # Read up on ssl_ciphers to ensure a secure configuration.
        # See: https://bugs.debian.org/765782
        #
        # Self signed certs generated by the ssl-cert package
        # Don't use them in a production server!
        #
        # include snippets/snakeoil.conf;

        root /var/www/html;

        # Add index.php to the list if you are using PHP
        index index.html index.htm index.nginx-debian.html;

        server_name _;

        location / {
                # First attempt to serve request as file, then
                # as directory, then fall back to displaying a 404.
                try_files $uri $uri/ =404;
        }

        # pass PHP scripts to FastCGI server
        #
        #location ~ \.php$ {
        #       include snippets/fastcgi-php.conf;
        #
        #       # With php-fpm (or other unix sockets):
        #       fastcgi_pass unix:/run/php/php7.4-fpm.sock;
        #       # With php-cgi (or other tcp sockets):
        #       fastcgi_pass 127.0.0.1:9000;
        #}

        # deny access to .htaccess files, if Apache's document root
        # concurs with nginx's one
        #
        #location ~ /\.ht {
        #       deny all;
        #}
}


# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
#       listen 80;
#       listen [::]:80;
#
#       server_name example.com;
#
#       root /var/www/example.com;
#       index index.html;
#
#       location / {
#               try_files $uri $uri/ =404;
#       }
#}

Are you intending or expecting the redirection from HTTP to HTTPS
for locations in .well-known/acme-challenge/?

$ curl -Ii http://ai.chaoss.io/.well-known/acme-challenge/4v4XdDwRTKIIGc9wVtYMv66I5WgDZnnA0zrgOoJ4SmH
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 13 May 2025 18:11:49 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://ai.chaoss.io/.well-known/acme-challenge/4v4XdDwRTKIIGc9wVtYMv66I5WgDZnnA0zrgOoJ4SmH

And the results of following the redirection to HTTPS

$ curl -Ii https://ai.chaoss.io/.well-known/acme-challenge/4v4XdDwRTKIIGc9wVtYMv66I5WgDZnnA0zrgOoJ4SmH
HTTP/1.1 404 NOT FOUND
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 13 May 2025 18:13:33 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 8146
Connection: keep-alive
Access-Control-Allow-Origin: *
2 Likes

No, I did not intend or expect a redirection from HTTP to HTTPS for the .well-known/acme-challenge/ location. The redirection is causing the challenge to fail, as Certbot is unable to access the verification file over HTTPS, resulting in a 404 Not Found error. I believe the challenge file needs to be accessible directly via HTTP, without the redirection to HTTPS during the certificate verification process.

Redirection to Port 443 (HTTPS) is acceptable after starting on Port 80 (HTTP).
Certainly not required to do so.

From HTTP-01 challenge states:
"The HTTP-01 challenge can only be done on port 80."
"Our implementation of the HTTP-01 challenge follows redirects, up to 10 redirects deep. It only accepts redirects to “http:” or “https:”, and only to ports 80 or 443. It does not accept redirects to IP addresses. When redirected to an HTTPS URL, it does not validate certificates (since this challenge is intended to bootstrap valid certificates, it may encounter self-signed or expired certificates along the way)."

3 Likes

You have two server blocks listening on the same port for the same domain name. Certbot probably updated just one of them but yet nginx is choosing the other when replying to HTTP requests. Hence the '404 Not Found'

Note these two server blocks. The second is missing a listen statement so defaults to port 80 leaving you with a duplicate. Once you remove/fix the duplicate problem the Certbot command should be fine

server {
    listen 80;
    server_name ai.chaoss.io;

    location /.well-known/acme-challenge/ {
        root /var/www/html;
    }

    location / {
        if ($request_uri !~ "^/\.well-known/acme-challenge/") {
            return 301 https://$host$request_uri;
        }
    }
}
server {
        server_name  ai.chaoss.io;

        location /api/unstable/ {
                proxy_pass http://ai.chaoss.io:5038;
                proxy_set_header Host $host;
        }
(... rest omitted but no listen statement ...)
4 Likes

Thank you for the feedback! I've reviewed the configuration and made adjustments to ensure no conflicts between the server blocks.

Currently, I have the following setup:

The first server block listens on HTTP (port 80) and handles the ACME challenge. It redirects all other traffic to HTTPS.

The second block listens on HTTPS (port 443) for the ACME challenge.

The third block listens on HTTPS (port 443) for the API and other routes.

Despite these changes, I'm still facing the same issue. I verified that the ACME challenge test file is accessible via HTTP (http://ai.chaoss.io/.well-known/acme-challenge/testfile), but Certbot is still returning a "404 Not Found" error.

Could you please advise on any other troubleshooting steps I should try? Perhaps there's a deeper issue related to how the server blocks are being selected for requests or how Certbot is interacting with Nginx.

1 Like

Did you leave the testfile up?

Checking HTTP and it redirects to HTTPS

$ curl -Ii http://ai.chaoss.io/.well-known/acme-challenge/testfile
HTTP/1.1 301 Moved Permanently
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 13 May 2025 19:32:01 GMT
Content-Type: text/html
Content-Length: 178
Connection: keep-alive
Location: https://ai.chaoss.io/.well-known/acme-challenge/testfile

Redirection to HTTPS

$ curl -Ii https://ai.chaoss.io/.well-known/acme-challenge/testfile
HTTP/1.1 404 NOT FOUND
Server: nginx/1.18.0 (Ubuntu)
Date: Tue, 13 May 2025 19:32:10 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 8146
Connection: keep-alive
Access-Control-Allow-Origin: *
2 Likes

That isn't a very important test when using the --nginx plugin. The --nginx plugin inserts code into your server block dynamically to handle the challenge reply. Once the request is over (for good or bad) the temp code is removed.

We can't mimic what happens during --nginx using standalone tests.

Would you please upload the entire /var/log/letsencrypt/letsencrypt.log file?

I can probably figure it out from that. The temp changes are shown as well as other key parts.

2 Likes

Oh, and that doesn't make sense. If port 80 handles the ACME Challenge what do you have in the port 443 server block?

In any case, you don't need any ACME specific "stuff" in the server blocks when using the --nginx option. That is one of the benefits of that option in that it handles it all for you.

4 Likes
2025-05-14 01:18:12,692:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2025-05-14 01:18:12,692:DEBUG:certbot._internal.error_handler:Calling registered functions
2025-05-14 01:18:12,692:INFO:certbot._internal.auth_handler:Cleaning up challenges
2025-05-14 01:18:13,772:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 33, in <module>
    sys.exit(load_entry_point('certbot==2.9.0', 'console_scripts', 'certbot')())
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1894, in main
    return config.func(config, plugins)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 1450, in run
    new_lineage = _get_and_save_cert(le_client, config, domains,
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/main.py", line 143, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 517, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2025-05-14 01:18:13,777:ERROR:certbot._internal.log:Some challenges have failed.

akshat-baranwal@AKB:~/Github/augur$

That was just the last part of the log. The important parts are much earlier. Really need the whole thing. Or, show us the output of nginx -T again and maybe can spot the problem in that.

3 Likes

Thank you for your help! It looks like the issue is resolved now, and everything is working correctly. I appreciate the guidance you provided.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.