Certificate renewal suddenly stopped working. No changes to server or DNS

Help
1

Hi,

I've a server that has been up and running for years, with certbot dutifully renewing my LE certificate. There has been no changes to the server, DNS records or pfSense, but suddenly the renewals have begun to fail. Anyone have any insights?

Thanks in advance,

@ my Wits-end

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mail.wits-end.ca

I ran this command: certbot renew -w /var/www/html/ (via cron or via sudo with same results)

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/mail.wits-end.ca.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.wits-end.ca
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Challenge failed for domain mail.wits-end.ca
http-01 challenge for mail.wits-end.ca
Cleaning up challenges
Attempting to renew cert (mail.wits-end.ca) from /etc/letsencrypt/renewal/mail.wits-end.ca.conf produced an unexpected error: Some challenges have failed.. Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.wits-end.ca/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/mail.wits-end.ca/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

My web server is (include version): nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 20.04.6 LTS

My hosting provider, if applicable, is: n/a

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.40.0 (Ubuntu apt install)

2

Welcome @Wits-end

The 404 is an HTTP error code meaning "not found"

For an HTTP Challenge using --webroot (that you are using) that usually means the webroot-path given to Certbot does not match the one used by your nginx server.

It can happen for other reasons. For example, a residential router with new firmware might be intercepting the HTTP challenge request and replying rather than your nginx server. Or, you are routing the incoming port 80 request to the wrong server. Or maybe a firewall rule has changed that affects the HTTP Challenge URLs.

Note you shouldn't need the -w /var/www/html for that command. The renew command will use the options from the original successful cert issuance. These options are stored in a conf file in /etc/letsencrypt/renewal

The first things to check are that the nginx server block for this domain has the correct root directory name and matches the Certbot webroot-path.

The other is to check your nginx access log and make sure it shows the 404 error.

I know you said nothing changed but something must have changed. A 404 error is pretty specific and almost certainly something on your end. Focus on any changes since Dec31 when your last cert was issued.

3 Likes
3

Hi MikeMcQ,

Thanks for getting back to me.

I'm very familiar with 404... my Teams picture at work is "Error 404... Technician not found" in an attempt to get people to ask someone else... LOL. It doesn't work... :frowning:

Looking at the others, will get back to it after a trip to the Hall to deal with Union stuff. I don't recall making any changes though as the server, firewall, and dns have been rock-solid for years. I do not use a residential router. I have a bridged router going to a pfSense box, with the server on it's own interface with routing that prevents the server lan from accessing the primary lan. The only changes to the ubuntu box have been apt update / upgrade. (unless it's something that iredmail did.... hmmm...)

If I create the ".well-known/acme-challenges/" directory in /var/www/html and place a simple html file in there, nginx serves up the site without any fuss.

@ my Wits-end

1 Like
4

Can you try .well-known/acme-challenge directory instead? :slight_smile:

And, use HTTP:// (not HTTPS://). You are allowed to redirect them but the original request is HTTP

Also, check your nginx access log for the 404. Does it show up there?

Note this

curl -i http://mail.wits-end.ca/.well-known/acme-challenge/Test404 

HTTP/1.1 404 Not Found
Server: nginx
Date: Tue, 17 Mar 2026 21:04:47 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive

Is different headers than below. Kind of looks like a different responder

curl -i http://mail.wits-end.ca/.well-known/acme-challenges/PluralTest404 

HTTP/1.1 301 Moved Permanently
Server: nginx
Date: Tue, 17 Mar 2026 21:06:39 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive
Location: https://mail.wits-end.ca/.well-known/acme-challenges/PluralTest404
X-Frame-Options: sameorigin
X-Content-Type-Options: nosniff
(... others omitted...)

Mind you, it is good practice to use a location block in the port 80 server block to handle the acme-challenge. So, maybe that accounts for the difference. But, maybe it is something else.

2 Likes
5

typo when I made the directory, and I just copy/pasted the link. I changed it to acme-challenge, and browsed to http://mail.wits-end.ca/.well-known/acme-challenge/hello.html and was served up a fresh page with new improved flavours.

57.103.73.161 - - [17/Mar/2026:14:12:52 -0700] "GET /.well-known/acme-challenge/aGbKWLGsZ6PeFcXjd3m2fF01nC5pcTtiS6j8Yspr9e0: HTTP/1.1" 404 146 "->
57.103.66.223 - - [17/Mar/2026:14:18:17 -0700] "GET /.well-known/acme-challenge/aGbKWLGsZ6PeFcXjd3m2fF01nC5pcTtiS6j8Yspr9e0: HTTP/1.1" 404 146 "->

6

I get 404. Did you try from the public internet?

curl -i http://mail.wits-end.ca/.well-known/acme-challenge/hello.html 

HTTP/1.1 404 Not Found
Server: nginx
Date: Tue, 17 Mar 2026 22:31:10 GMT
Content-Type: text/html
Content-Length: 146
Connection: keep-alive
1 Like
7

Ok, that's weird. Browsing on my phone on data brings up the page, mind you it serves it up as https instead of http. Curl brings up 404...

8

Some browsers try HTTP and HTTPS at the same time. They will favor HTTPS if they find it working. Check your routings for Port 80. Or perhaps firewall rules affecting just port 80. Anyway that kind of thing

2 Likes
9

I've verified that certbot has the ability to write to /var/www/html/.well-known/acme-challenge/ as when I run in debug-challenge, I can start up another terminal session and see that the directories have been created and that the challenge token is there...

so, not a file permission issue... which means it has to be how the file is being served up (or not) to LE...

10

Please show the nginx server block for port 80 for this domain.

Better, show the entire output of sudo nginx -T

If you prefer, pipe that output to a file outside of the directory structure of nginx and upload that.

Agree, nginx isn't handling it as you expect

Is there anything unusual in the nginx error log?

1 Like
11

Here is the results of nginx -T (warning - LONG)

root@mail:/var/www/html# nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes 1;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/conf-enabled/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
}

# configuration file /etc/nginx/conf-enabled/0-general.conf:
map_hash_bucket_size 1024;

# configuration file /etc/nginx/conf-enabled/cache.conf:
map $sent_http_content_type $expires {
    default                     off;
    application/x-javascript    1d;
    text/css                    1d;
    ~image/                     1d;
}

expires $expires;

# configuration file /etc/nginx/conf-enabled/client_max_body_size.conf:
client_max_body_size 12m;

# configuration file /etc/nginx/conf-enabled/default_type.conf:
default_type application/octet-stream;

# configuration file /etc/nginx/conf-enabled/gzip.conf:
gzip on;
gzip_vary on;
gzip_http_version 1.0;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_min_length 10240;
gzip_proxied any;
gzip_disable "MSIE [1-6]\.";

# text/html is always compressed.
gzip_types
    text/plain
    text/css
    text/xml
    text/javascript
    text/json
    text/vcard
    text/cache-manifest
    text/vnd.rim.location.xloc
    text/vtt
    text/x-component
    text/x-cross-domain-policy
    image/bmp
    image/vnd.microsoft.icon
    image/x-icon
    image/svg+xml
    font/truetype
    font/opentype
    application/atom+xml
    application/javascript
    application/json
    application/ld+json
    application/vnd.geo+json
    application/manifest+json
    application/x-javascript
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/xml
    application/xml+rss
    application/xhtml+xml
    application/vnd.ms-fontobject;

# configuration file /etc/nginx/conf-enabled/headers.conf:
add_header X-Frame-Options sameorigin;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection '1; mode=block';
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Content-Security-Policy "default-src https: data: 'unsafe-inline' 'unsafe-eval'";
add_header Referrer-Policy strict-origin;

# configuration file /etc/nginx/conf-enabled/log.conf:
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

# configuration file /etc/nginx/conf-enabled/mime_types.conf:
include /etc/nginx/mime.types;

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/png                             png;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;
    image/svg+xml                         svg svgz;
    image/webp                            webp;

    application/font-woff                 woff;
    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.wap.wmlc              wmlc;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet          xlsx;
    application/vnd.openxmlformats-officedocument.presentationml.presentation  pptx;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/conf-enabled/php_fpm.conf:
upstream php_workers {
    server 127.0.0.1:9999;
}

# configuration file /etc/nginx/conf-enabled/sendfile.conf:
sendfile on;

# configuration file /etc/nginx/conf-enabled/server_tokens.conf:
# Hide Nginx version number
server_tokens off;

# configuration file /etc/nginx/conf-enabled/types_hash_max_size.conf:
types_hash_max_size 2048;

# configuration file /etc/nginx/sites-enabled/00-default-ssl.conf:
#
# Note: This file must be loaded before other virtual host config files,
#
# HTTPS
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name _;

    root /var/www/html;
    index index.php index.html;

    include /etc/nginx/templates/misc.tmpl;
    include /etc/nginx/templates/ssl.tmpl;
    include /etc/nginx/templates/iredadmin.tmpl;
    include /etc/nginx/templates/roundcube.tmpl;
    include /etc/nginx/templates/sogo.tmpl;
    include /etc/nginx/templates/netdata.tmpl;
    include /etc/nginx/templates/php-catchall.tmpl;
    include /etc/nginx/templates/stub_status.tmpl;
}

# configuration file /etc/nginx/templates/misc.tmpl:
# Allow access to '^/.well-known/'
location ~ ^/.well-known/ {
    allow all;
    access_log off;
    log_not_found off;
    autoindex off;
    #root /var/www/html;
}

# Deny all attempts to access hidden files such as .htaccess.
location ~ /\. { deny all; }

# Handling noisy messages
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }

# configuration file /etc/nginx/templates/ssl.tmpl:
ssl_protocols TLSv1.2 TLSV1.3;

# Fix 'The Logjam Attack'.
ssl_ciphers EECDH+CHACHA20:EECDH+AESGCM:EDH+AESGCM:AES256+EECDH;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/ssl/dh2048_param.pem;

# To use your own ssl cert (e.g. "Let's Encrypt"), please create symbol link to
# ssl cert/key used below, so that we can manage this config file with Ansible.
#
# For example:
#
# rm -f /etc/ssl/private/iRedMail.key
# rm -f /etc/ssl/certs/iRedMail.crt
# ln -s /etc/letsencrypt/live/<domain>/privkey.pem /etc/ssl/private/iRedMail.key
# ln -s /etc/letsencrypt/live/<domain>/fullchain.pem /etc/ssl/certs/iRedMail.crt
#
# To request free "Let's Encrypt" cert, please check our tutorial:
# https://docs.iredmail.org/letsencrypt.html
# ssl_certificate /etc/ssl/certs/iRedMail.crt;
# ssl_certificate_key /etc/ssl/private/iRedMail.key;

ssl_certificate /etc/letsencrypt/live/mail.wits-end.ca/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/mail.wits-end.ca/privkey.pem;

# Greatly improve the performance of keep-alive connections over SSL.
# With this enabled, client is not necessary to do a full SSL-handshake for
# every request, thus saving time and cpu-resources.
ssl_session_cache shared:SSL:10m;

# configuration file /etc/nginx/templates/iredadmin.tmpl:
# Settings for iRedAdmin.

# static files under /iredadmin/static
location ~ ^/iredadmin/static/(.*) {
    alias /opt/www/iredadmin/static/$1;
}

# Python scripts
location ~ ^/iredadmin(.*) {
    rewrite ^/iredadmin(/.*)$ $1 break;

    include /etc/nginx/templates/hsts.tmpl;

    include uwsgi_params;
    uwsgi_pass 127.0.0.1:7791;
    uwsgi_param UWSGI_CHDIR /opt/www/iredadmin;
    uwsgi_param UWSGI_SCRIPT iredadmin;
    uwsgi_param SCRIPT_NAME /iredadmin;

    # Access control
    #allow 127.0.0.1;
    #allow 192.168.1.10;
    #allow 192.168.1.0/24;
    #deny all;
}

# iRedAdmin: redirect /iredadmin to /iredadmin/
location = /iredadmin {
    rewrite ^ /iredadmin/;
}

# Handle newsletter-style subscription/unsubscription supported in iRedAdmin-Pro.
location ~ ^/newsletter/ {
    rewrite /newsletter/(.*) /iredadmin/newsletter/$1 last;
}

# configuration file /etc/nginx/templates/hsts.tmpl:
# Use HTTP Strict Transport Security to force client to use secure
# connections only. References:
#
# * RFC Document (6797): HTTP Strict Transport Security (HSTS)
#   https://tools.ietf.org/html/rfc6797#section-6.1.2
#
# * Short tutorial from Mozilla:
#   https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security
#
# WARNING: According to RFC document, HSTS will fail with self-signed SSL
#          certificate.
#          https://tools.ietf.org/html/rfc6797#page-27
#
# Syntax:
#
#   Strict-Transport-Security: max-age=expireTime [; includeSubDomains] [; preload]
add_header Strict-Transport-Security "max-age=31536000";

# configuration file /etc/nginx/uwsgi_params:

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  REQUEST_SCHEME     $scheme;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;

# configuration file /etc/nginx/templates/roundcube.tmpl:
#
# Running Roundcube as a subfolder on an existing virtual host
#
# Block access to default directories and files under these directories
location ~ ^/mail/(bin|config|installer|logs|SQL|temp|vendor)($|/.*) { deny all; }

# Block access to default files under top-directory and files start with same name.
location ~ ^/mail/(CHANGELOG|composer.json|INSTALL|jsdeps.json|LICENSE|README|UPGRADING)($|.*) { deny all; }

# Block plugin config files and sample config files.
location ~ ^/mail/plugins/.*/config.inc.php.* { deny all; }

# Block access to plugin data
location ~ ^/mail/plugins/enigma/home($|/.*) { deny all; }

# Redirect URI `/mail` to `/mail/`.
location = /mail {
    return 301 /mail/;
}

location ~ ^/mail/(.*\.php)$ {
    include /etc/nginx/templates/hsts.tmpl;
    include /etc/nginx/templates/fastcgi_php.tmpl;
    fastcgi_param SCRIPT_FILENAME /opt/www/roundcubemail/$1;
}

location ~ ^/mail/(.*) {
    alias /opt/www/roundcubemail/$1;
    index index.php;
}

# configuration file /etc/nginx/templates/fastcgi_php.tmpl:
#
# Template used to handle PHP fastcgi applications
#
# You still need to define `SCRIPT_FILENAME` for your PHP application, and
# probably `fastcgi_index` if your application use different index file.
#
include fastcgi_params;

# Directory index file
fastcgi_index index.php;

# Handle PHP files with upstream handler
fastcgi_pass php_workers;

# Fix the HTTPROXY issue.
# Reference: https://httpoxy.org/
fastcgi_param HTTP_PROXY '';

# configuration file /etc/nginx/fastcgi_params:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  REQUEST_SCHEME     $scheme;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

# configuration file /etc/nginx/templates/sogo.tmpl:
# Settings for SOGo Groupware

# SOGo
location ~ ^/sogo { rewrite ^ https://$host/SOGo; }
location ~ ^/SOGO { rewrite ^ https://$host/SOGo; }

# Redirect /mail to /SOGo
#location ~ ^/mail { rewrite ^ https://$host/SOGo; }

# For Mac OS X and iOS devices.
rewrite ^/.well-known/caldav    /SOGo/dav permanent;
rewrite ^/.well-known/carddav   /SOGo/dav permanent;
rewrite ^/principals            /SOGo/dav permanent;

location ^~ /SOGo {
    include /etc/nginx/templates/hsts.tmpl;

    proxy_pass http://127.0.0.1:20000;

    # forward user's IP address
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;

    # always use https
    proxy_set_header x-webobjects-server-port $server_port;
    proxy_set_header x-webobjects-server-name $host;
    proxy_set_header x-webobjects-server-url  https://$host;

    proxy_set_header x-webobjects-server-protocol HTTP/1.0;

    proxy_busy_buffers_size   64k;
    proxy_buffers             8 64k;
    proxy_buffer_size         64k;
}

location ^~ /Microsoft-Server-ActiveSync {
    proxy_pass http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync;

    proxy_connect_timeout 3540;
    proxy_send_timeout 3540;
    proxy_read_timeout 3540;

    proxy_busy_buffers_size   64k;
    proxy_buffers             8 64k;
    proxy_buffer_size         64k;
}

location ^~ /SOGo/Microsoft-Server-ActiveSync {
    proxy_pass http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync;

    proxy_connect_timeout 3540;
    proxy_send_timeout 3540;
    proxy_read_timeout 3540;

    proxy_busy_buffers_size   64k;
    proxy_buffers             8 64k;
    proxy_buffer_size         64k;
}

location /SOGo.woa/WebServerResources/ {
    alias /usr/lib/GNUstep/SOGo/WebServerResources/;
    expires max;
}
location /SOGo/WebServerResources/ {
    alias /usr/lib/GNUstep/SOGo/WebServerResources/;
    expires max;
}
location ^/SOGo/so/ControlPanel/Products/([^/]*)/Resources/(.*)$ {
    alias /usr/lib/GNUstep/SOGo/$1.SOGo/Resources/$2;
    expires max;
}

# configuration file /etc/nginx/templates/netdata.tmpl:
# Running netdata as a subfolder to an existing virtual host
# FYI: https://github.com/firehol/netdata/wiki/Running-behind-nginx

location = /netdata {
    return 301 /netdata/;
}

location ~ /netdata/(?<ndpath>.*) {
    proxy_redirect off;
    proxy_set_header Host $host;

    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_pass_request_headers on;
    proxy_set_header Connection "keep-alive";
    proxy_store off;
    proxy_pass http://127.0.0.1:19999/$ndpath$is_args$args;

    gzip on;
    gzip_proxied any;
    gzip_types *;

    auth_basic "Authentication Required";
    auth_basic_user_file /etc/nginx/netdata.users;
}

# configuration file /etc/nginx/templates/php-catchall.tmpl:
# Normal PHP scripts
location ~ \.php$ {
    include /etc/nginx/templates/fastcgi_php.tmpl;

    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}

# configuration file /etc/nginx/templates/stub_status.tmpl:
location = /stub_status {
    stub_status on;
    access_log off;
    allow 127.0.0.1;
    deny all;
}

location = /status {
    include fastcgi_params;
    fastcgi_pass php_workers;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
    access_log off;
    allow 127.0.0.1;
    deny all;
}

# configuration file /etc/nginx/sites-enabled/00-default.conf:
#
# Note: This file must be loaded before other virtual host config files,
#
# HTTP
server {
    # Listen on ipv4
    listen 80;
    # listen [::]:80;

    server_name _;

    # Allow ACME challenge to be served over HTTP (don't redirect to HTTPS).
    location ~* ^/.well-known/acme-challenge/ {
        root /opt/www/well_known;
        try_files $uri =404;
        allow all;
    }

    # Redirect all insecure http:// requests to https://
    location / {
        return 301 https://$host$request_uri;
    }
}
12

The only odd thing relating to cert renewal that I could spot in the error.log file is

2026/03/17 15:12:14 [error] 14104#14104: *3967 directory index of "/var/www/html/.well-known/acme-challenge/" is forbidden, client: 184.105.99.10

There are other errors, but relating to .xml files that I know do not exist on the server. directories like blog, wordpress, wp, news, 2018, 2019, etc...

13

Earlier you said this:

But based on this configuration:

listen 80;
# listen [::]:80;
server_name _;

# Allow ACME challenge to be served over HTTP (don't redirect to HTTPS).
location ~* ^/.well-known/acme-challenge/ {
    root /opt/www/well_known;
    try_files $uri =404;
    allow all;
}

# Redirect all insecure http:// requests to https://
location / {
    return 301 https://$host$request_uri;
}

You probably need to put the files into /opt/www/well_known

3 Likes
14

Definitely :slight_smile:

@Wits-end I'd be curious to see the contents of the renewal config file in
/etc/letsencrypt/renewal
for this certificate. You can redact the account number if you wish

4 Likes
15

considering that certbot sets the location of the file, I tried sudo certbot renew -w /opt/www/well_known, with the same error 404.

contents of mail.wits-end.ca.conf

renew_before_expiry = 30 days

version = 0.40.0
archive_dir = /etc/letsencrypt/archive/mail.wits-end.ca
cert = /etc/letsencrypt/live/mail.wits-end.ca/cert.pem
privkey = /etc/letsencrypt/live/mail.wits-end.ca/privkey.pem
chain = /etc/letsencrypt/live/mail.wits-end.ca/chain.pem
fullchain = /etc/letsencrypt/live/mail.wits-end.ca/fullchain.pem

Options used in the renewal process

[renewalparams]
account =
authenticator = webroot
webroot_path = /var/www/html,
server= https://acme-v02.api.letsencrypt.org/directory
[[webroot_map]]
mail.wits-end.ca = /var/www/html

16

That probably should have worked. That 0.40 version of Certbot you are using is ancient (from 2019). Perhaps it didn't handle a -w override w/renew back then. We'll talk about upgrading that once we get past this initial problem :slight_smile:

But, let's do a better experiment. Try this

sudo certbot certonly --dry-run --webroot -w /opt/www/well_known -d mail.wits-end.ca

If that gets 404 review the nginx access and error logs for the same time and show what's there.

Something has definitely changed. We can see the renewal config has /var/www/html. That was what was used when the cert was originally issued. But, the nginx server block for port 80 is using a different directory.

UPDATE: As an aside, your hunch that iRedMail is involved looks correct. That exact directory is used in their setup guide: iRedMail Easy: Request a free cert from Let's Encrypt

2 Likes
17

alright, back from a nights sleep, and the dry-run was successful with that command, so I'll renew the cert without the dry-run option and then we'll get on with certbot. It's that old as the apt install for Ubuntu 20.04 doesn't upgrade it from that, but perhaps replacing it with a snap install will do.

1 Like
18

Ok then, removed the apt version of certbot, installed the snap version with classic flag, set a new symbolic link. Renewed the cert again and updated it to EDCSA. I think that about covers it.

Thank you kindly for your help.

2 Likes
19

Excellent. Re-issuing the cert with the new webroot-path should have updated the renewal config file too.

Generally should avoid using -w or other options with the renew command. You can test the renew using
sudo certbot renew --dry-run

Options other than that with renew would apply to every cert. With just one it isn't harmful they are just unnecessary. And, would create issues if you ever got another cert.

3 Likes