Certbot renew suddenly failing (Challenge failed for domain)

Help
21

Dollars to donuts we're gonna see some duplicate/incomplete server blocks. Just my hunch.

2 Likes
22

I concur with that conclusion. It might lead to some confusion on occasion though, as seen here.

2 Likes
23

Very likely although nginx is far less prone to such things compared to Apache. nginx actually warns about common overlaps :slight_smile:

Still, problems with listen statements or even possibly location blocks are common causes. Given they are not aware of changes it might be something more subtle but we'll see.

I wouldn't bet dollars or donuts at the moment :slight_smile:

2 Likes
24

Nginx could be internally proxying the challenge file requests to somewhere else. Only the config shall tell. :crystal_ball:

Using webroot method means no interaction/manipulation with nginx internal behavior.

(I know you know this, @MikeMcQ. Just saying for our studio audience.)

3 Likes
25

Here's the config. Sorry but putting it in a separate file is a bit beyond me in the 90 seconds I could get away from something even more urgent.

[root@elm html]# nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes 1;
pid /var/run/nginx.pid;

events {
    worker_connections 1024;
}

http {
    include /etc/nginx/conf-enabled/*.conf;
    include /etc/nginx/sites-enabled/*.conf;
}

# configuration file /etc/nginx/conf-enabled/0-general.conf:
map_hash_bucket_size 1024;

# configuration file /etc/nginx/conf-enabled/cache.conf:
map $sent_http_content_type $expires {
    default                     off;
    application/x-javascript    1d;
    text/css                    1d;
    ~image/                     1d;
}

expires $expires;

# configuration file /etc/nginx/conf-enabled/client_max_body_size.conf:
client_max_body_size 50m;

# configuration file /etc/nginx/conf-enabled/default_type.conf:
default_type application/octet-stream;

# configuration file /etc/nginx/conf-enabled/gzip.conf:
gzip on;
gzip_vary on;
gzip_http_version 1.0;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_min_length 10240;
gzip_proxied any;
gzip_disable "MSIE [1-6]\.";

# text/html is always compressed.
gzip_types
    text/plain
    text/css
    text/xml
    text/javascript
    text/json
    text/vcard
    text/cache-manifest
    text/vnd.rim.location.xloc
    text/vtt
    text/x-component
    text/x-cross-domain-policy
    image/bmp
    image/vnd.microsoft.icon
    image/x-icon
    image/svg+xml
    font/truetype
    font/opentype
    application/atom+xml
    application/javascript
    application/json
    application/ld+json
    application/vnd.geo+json
    application/manifest+json
    application/x-javascript
    application/x-font-ttf
    application/x-web-app-manifest+json
    application/xml
    application/xml+rss
    application/xhtml+xml
    application/vnd.ms-fontobject;

# configuration file /etc/nginx/conf-enabled/headers.conf:
add_header X-Frame-Options sameorigin;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection '1; mode=block';
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Content-Security-Policy "default-src https: data: 'unsafe-inline' 'unsafe-eval'";
add_header Referrer-Policy strict-origin;

# configuration file /etc/nginx/conf-enabled/log.conf:
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;

# configuration file /etc/nginx/conf-enabled/mime_types.conf:
include /etc/nginx/mime.types;

# configuration file /etc/nginx/mime.types:

types {
    text/html                                        html htm shtml;
    text/css                                         css;
    text/xml                                         xml;
    image/gif                                        gif;
    image/jpeg                                       jpeg jpg;
    application/javascript                           js;
    application/atom+xml                             atom;
    application/rss+xml                              rss;

    text/mathml                                      mml;
    text/plain                                       txt;
    text/vnd.sun.j2me.app-descriptor                 jad;
    text/vnd.wap.wml                                 wml;
    text/x-component                                 htc;

    image/png                                        png;
    image/svg+xml                                    svg svgz;
    image/tiff                                       tif tiff;
    image/vnd.wap.wbmp                               wbmp;
    image/webp                                       webp;
    image/x-icon                                     ico;
    image/x-jng                                      jng;
    image/x-ms-bmp                                   bmp;

    font/woff                                        woff;
    font/woff2                                       woff2;

    application/java-archive                         jar war ear;
    application/json                                 json;
    application/mac-binhex40                         hqx;
    application/msword                               doc;
    application/pdf                                  pdf;
    application/postscript                           ps eps ai;
    application/rtf                                  rtf;
    application/vnd.apple.mpegurl                    m3u8;
    application/vnd.google-earth.kml+xml             kml;
    application/vnd.google-earth.kmz                 kmz;
    application/vnd.ms-excel                         xls;
    application/vnd.ms-fontobject                    eot;
    application/vnd.ms-powerpoint                    ppt;
    application/vnd.oasis.opendocument.graphics      odg;
    application/vnd.oasis.opendocument.presentation  odp;
    application/vnd.oasis.opendocument.spreadsheet   ods;
    application/vnd.oasis.opendocument.text          odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation
                                                     pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
                                                     xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document
                                                     docx;
    application/vnd.wap.wmlc                         wmlc;
    application/x-7z-compressed                      7z;
    application/x-cocoa                              cco;
    application/x-java-archive-diff                  jardiff;
    application/x-java-jnlp-file                     jnlp;
    application/x-makeself                           run;
    application/x-perl                               pl pm;
    application/x-pilot                              prc pdb;
    application/x-rar-compressed                     rar;
    application/x-redhat-package-manager             rpm;
    application/x-sea                                sea;
    application/x-shockwave-flash                    swf;
    application/x-stuffit                            sit;
    application/x-tcl                                tcl tk;
    application/x-x509-ca-cert                       der pem crt;
    application/x-xpinstall                          xpi;
    application/xhtml+xml                            xhtml;
    application/xspf+xml                             xspf;
    application/zip                                  zip;

    application/octet-stream                         bin exe dll;
    application/octet-stream                         deb;
    application/octet-stream                         dmg;
    application/octet-stream                         iso img;
    application/octet-stream                         msi msp msm;

    audio/midi                                       mid midi kar;
    audio/mpeg                                       mp3;
    audio/ogg                                        ogg;
    audio/x-m4a                                      m4a;
    audio/x-realaudio                                ra;

    video/3gpp                                       3gpp 3gp;
    video/mp2t                                       ts;
    video/mp4                                        mp4;
    video/mpeg                                       mpeg mpg;
    video/quicktime                                  mov;
    video/webm                                       webm;
    video/x-flv                                      flv;
    video/x-m4v                                      m4v;
    video/x-mng                                      mng;
    video/x-ms-asf                                   asx asf;
    video/x-ms-wmv                                   wmv;
    video/x-msvideo                                  avi;
}

# configuration file /etc/nginx/conf-enabled/php_fpm.conf:
upstream php_workers {
    server 127.0.0.1:9999;
}

# configuration file /etc/nginx/conf-enabled/sendfile.conf:
sendfile on;

# configuration file /etc/nginx/conf-enabled/server_tokens.conf:
# Hide Nginx version number
server_tokens off;

# configuration file /etc/nginx/conf-enabled/types_hash_max_size.conf:
types_hash_max_size 2048;

# configuration file /etc/nginx/sites-enabled/00-default-ssl.conf:
#
# Note: This file must be loaded before other virtual host config files,
#
# HTTPS
server {
    listen 443 ssl http2;
    listen [::]:443 ssl http2;
    server_name _;

    root /var/www/html;
    index index.php index.html;

    include /etc/nginx/templates/misc.tmpl;
    include /etc/nginx/templates/ssl.tmpl;
    include /etc/nginx/templates/iredadmin.tmpl;
    include /etc/nginx/templates/roundcube.tmpl;
    include /etc/nginx/templates/sogo.tmpl;
    include /etc/nginx/templates/netdata.tmpl;
    include /etc/nginx/templates/php-catchall.tmpl;
    include /etc/nginx/templates/stub_status.tmpl;
}

# configuration file /etc/nginx/templates/misc.tmpl:
# Allow access to '^/.well-known/'
location ~ ^/.well-known/ {
    allow all;
    access_log off;
    log_not_found off;
    autoindex off;
    #root /var/www/html;
}

# Deny all attempts to access hidden files such as .htaccess.
location ~ /\. { deny all; }

# Handling noisy messages
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }

# configuration file /etc/nginx/templates/ssl.tmpl:
ssl_protocols TLSv1.2 TLSv1.3;

# Fix 'The Logjam Attack'.
ssl_ciphers EECDH+CHACHA20:EECDH+AESGCM:EDH+AESGCM:AES256+EECDH;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/pki/tls/dh2048_param.pem;

# Greatly improve the performance of keep-alive connections over SSL.
# With this enabled, client is not necessary to do a full SSL-handshake for
# every request, thus saving time and cpu-resources.
ssl_session_cache shared:SSL:10m;

# To use your own ssl cert (e.g. "Let's Encrypt"), please create symbol link to
# ssl cert/key used below, so that we can manage this config file with Ansible.
#
# For example:
#
# rm -f /etc/pki/tls/private/iRedMail.key
# rm -f /etc/pki/tls/certs/iRedMail.crt
# ln -s /etc/letsencrypt/live/<domain>/privkey.pem /etc/pki/tls/private/iRedMail.key
# ln -s /etc/letsencrypt/live/<domain>/fullchain.pem /etc/pki/tls/certs/iRedMail.crt
#
# To request free "Let's Encrypt" cert, please check our tutorial:
# https://docs.iredmail.org/letsencrypt.html
ssl_certificate /etc/pki/tls/certs/iRedMail.crt;
ssl_certificate_key /etc/pki/tls/private/iRedMail.key;

# configuration file /etc/nginx/templates/iredadmin.tmpl:
# Settings for iRedAdmin.

# static files under /iredadmin/static
location ~ ^/iredadmin/static/(.*) {
    alias /opt/www/iredadmin/static/$1;
}

# Python scripts
location ~ ^/iredadmin(.*) {
    rewrite ^/iredadmin(/.*)$ $1 break;

    include /etc/nginx/templates/hsts.tmpl;

    include uwsgi_params;
    uwsgi_pass 127.0.0.1:7791;
    uwsgi_param UWSGI_CHDIR /opt/www/iredadmin;
    uwsgi_param UWSGI_SCRIPT iredadmin;
    uwsgi_param SCRIPT_NAME /iredadmin;

    # Access control
    #allow 127.0.0.1;
    #allow 192.168.1.10;
    #allow 192.168.1.0/24;
    #deny all;
}

# iRedAdmin: redirect /iredadmin to /iredadmin/
location = /iredadmin {
    rewrite ^ /iredadmin/;
}

# Handle newsletter-style subscription/unsubscription supported in iRedAdmin-Pro.
location ~ ^/newsletter/ {
    rewrite /newsletter/(.*) /iredadmin/newsletter/$1 last;
}

# configuration file /etc/nginx/templates/hsts.tmpl:
# Use HTTP Strict Transport Security to force client to use secure
# connections only. References:
#
# * RFC Document (6797): HTTP Strict Transport Security (HSTS)
#   https://tools.ietf.org/html/rfc6797#section-6.1.2
#
# * Short tutorial from Mozilla:
#   https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security
#
# WARNING: According to RFC document, HSTS will fail with self-signed SSL
#          certificate.
#          https://tools.ietf.org/html/rfc6797#page-27
#
# Syntax:
#
#   Strict-Transport-Security: max-age=expireTime [; includeSubDomains] [; preload]
add_header Strict-Transport-Security "max-age=31536000";

# configuration file /etc/nginx/uwsgi_params:

uwsgi_param  QUERY_STRING       $query_string;
uwsgi_param  REQUEST_METHOD     $request_method;
uwsgi_param  CONTENT_TYPE       $content_type;
uwsgi_param  CONTENT_LENGTH     $content_length;

uwsgi_param  REQUEST_URI        $request_uri;
uwsgi_param  PATH_INFO          $document_uri;
uwsgi_param  DOCUMENT_ROOT      $document_root;
uwsgi_param  SERVER_PROTOCOL    $server_protocol;
uwsgi_param  REQUEST_SCHEME     $scheme;
uwsgi_param  HTTPS              $https if_not_empty;

uwsgi_param  REMOTE_ADDR        $remote_addr;
uwsgi_param  REMOTE_PORT        $remote_port;
uwsgi_param  SERVER_PORT        $server_port;
uwsgi_param  SERVER_NAME        $server_name;

# configuration file /etc/nginx/templates/roundcube.tmpl:
#
# Running Roundcube as a subfolder on an existing virtual host
#
# Block access to default directories and files under these directories
location ~ ^/mail/(bin|config|installer|logs|SQL|temp|vendor)($|/.*) { deny all; }

# Block access to default files under top-directory and files start with same name.
location ~ ^/mail/(CHANGELOG|composer.json|INSTALL|jsdeps.json|LICENSE|README|UPGRADING)($|.*) { deny all; }

# Block plugin config files and sample config files.
location ~ ^/mail/plugins/.*/config.inc.php.* { deny all; }

# Block access to plugin data
location ~ ^/mail/plugins/enigma/home($|/.*) { deny all; }

# Redirect URI `/mail` to `/mail/`.
location = /mail {
    return 301 /mail/;
}

location ~ ^/mail/(.*\.php)$ {
    include /etc/nginx/templates/hsts.tmpl;
    include /etc/nginx/templates/fastcgi_php.tmpl;
    fastcgi_param SCRIPT_FILENAME /opt/www/roundcubemail/$1;
}

location ~ ^/mail/(.*) {
    alias /opt/www/roundcubemail/$1;
    index index.php;
}

# configuration file /etc/nginx/templates/fastcgi_php.tmpl:
#
# Template used to handle PHP fastcgi applications
#
# You still need to define `SCRIPT_FILENAME` for your PHP application, and
# probably `fastcgi_index` if your application use different index file.
#
include fastcgi_params;

# Directory index file
fastcgi_index index.php;

# Handle PHP files with upstream handler
fastcgi_pass php_workers;

# Fix the HTTPROXY issue.
# Reference: https://httpoxy.org/
fastcgi_param HTTP_PROXY '';

# configuration file /etc/nginx/fastcgi_params:

fastcgi_param  QUERY_STRING       $query_string;
fastcgi_param  REQUEST_METHOD     $request_method;
fastcgi_param  CONTENT_TYPE       $content_type;
fastcgi_param  CONTENT_LENGTH     $content_length;

fastcgi_param  SCRIPT_NAME        $fastcgi_script_name;
fastcgi_param  REQUEST_URI        $request_uri;
fastcgi_param  DOCUMENT_URI       $document_uri;
fastcgi_param  DOCUMENT_ROOT      $document_root;
fastcgi_param  SERVER_PROTOCOL    $server_protocol;
fastcgi_param  REQUEST_SCHEME     $scheme;
fastcgi_param  HTTPS              $https if_not_empty;

fastcgi_param  GATEWAY_INTERFACE  CGI/1.1;
fastcgi_param  SERVER_SOFTWARE    nginx/$nginx_version;

fastcgi_param  REMOTE_ADDR        $remote_addr;
fastcgi_param  REMOTE_PORT        $remote_port;
fastcgi_param  SERVER_ADDR        $server_addr;
fastcgi_param  SERVER_PORT        $server_port;
fastcgi_param  SERVER_NAME        $server_name;

# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param  REDIRECT_STATUS    200;

# configuration file /etc/nginx/templates/sogo.tmpl:
# Settings for SOGo Groupware

# SOGo
location ~ ^/sogo { rewrite ^ https://$host/SOGo; }
location ~ ^/SOGO { rewrite ^ https://$host/SOGo; }

# Redirect /mail to /SOGo
#location ~ ^/mail { rewrite ^ https://$host/SOGo; }

# For Mac OS X and iOS devices.
rewrite ^/.well-known/caldav    /SOGo/dav permanent;
rewrite ^/.well-known/carddav   /SOGo/dav permanent;
rewrite ^/principals            /SOGo/dav permanent;

location ^~ /SOGo {
    include /etc/nginx/templates/hsts.tmpl;

    proxy_pass http://127.0.0.1:20000;

    # forward user's IP address
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header Host $host;

    # always use https
    proxy_set_header x-webobjects-server-port $server_port;
    proxy_set_header x-webobjects-server-name $host;
    proxy_set_header x-webobjects-server-url  https://$host;
    proxy_set_header x-webobjects-server-protocol HTTP/1.0;

    proxy_busy_buffers_size   64k;
    proxy_buffers             8 64k;
    proxy_buffer_size         64k;
}

location ^~ /Microsoft-Server-ActiveSync {
    proxy_pass http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync;

    proxy_connect_timeout 3540;
    proxy_send_timeout 3540;
    proxy_read_timeout 3540;

    proxy_busy_buffers_size   64k;
    proxy_buffers             8 64k;
    proxy_buffer_size         64k;
}

location ^~ /SOGo/Microsoft-Server-ActiveSync {
    proxy_pass http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync;

    proxy_connect_timeout 3540;
    proxy_send_timeout 3540;
    proxy_read_timeout 3540;

    proxy_busy_buffers_size   64k;
    proxy_buffers             8 64k;
    proxy_buffer_size         64k;
}

location /SOGo.woa/WebServerResources/ {
    alias /usr/lib64/GNUstep/SOGo/WebServerResources/;
    expires max;
}
location /SOGo/WebServerResources/ {
    alias /usr/lib64/GNUstep/SOGo/WebServerResources/;
    expires max;
}
location ^/SOGo/so/ControlPanel/Products/([^/]*)/Resources/(.*)$ {
    alias /usr/lib64/GNUstep/SOGo/$1.SOGo/Resources/$2;
    expires max;
}

# configuration file /etc/nginx/templates/netdata.tmpl:
# Running netdata as a subfolder to an existing virtual host
# FYI: https://github.com/firehol/netdata/wiki/Running-behind-nginx

location = /netdata {
    return 301 /netdata/;
}

location ~ /netdata/(?<ndpath>.*) {
    proxy_redirect off;
    proxy_set_header Host $host;

    proxy_set_header X-Forwarded-Host $host;
    proxy_set_header X-Forwarded-Server $host;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_http_version 1.1;
    proxy_pass_request_headers on;
    proxy_set_header Connection "keep-alive";
    proxy_store off;
    proxy_pass http://127.0.0.1:19999/$ndpath$is_args$args;

    gzip on;
    gzip_proxied any;
    gzip_types *;

    auth_basic "Authentication Required";
    auth_basic_user_file /etc/nginx/netdata.users;
}

# configuration file /etc/nginx/templates/php-catchall.tmpl:
# Normal PHP scripts
location ~ \.php$ {
    include /etc/nginx/templates/fastcgi_php.tmpl;

    fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}

# configuration file /etc/nginx/templates/stub_status.tmpl:
location = /stub_status {
    stub_status on;
    access_log off;
    allow 127.0.0.1;
    deny all;
}

location = /status {
    include fastcgi_params;
    fastcgi_pass php_workers;
    fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
    access_log off;
    allow 127.0.0.1;
    deny all;
}

# configuration file /etc/nginx/sites-enabled/00-default.conf:
#
# Note: This file must be loaded before other virtual host config files,
#
# HTTP
server {
    # Listen on ipv4
    listen 80;
    #listen [::]:80;

    server_name _;

    # Allow ACME challenge to be served over HTTP (don't redirect to HTTPS).
    location ~* ^/.well-known/acme-challenge/ {
        root /opt/www/well_known;
        try_files $uri =404;
        allow all;
    }

    # Redirect all insecure http requests to https.
    location / {
        return 301 https://$host$request_uri;
    }
}

[root@elm html]#
2 Likes
26

Your only port 80 server block is the one below so handles any HTTP (port 80) request inbound to your nginx

Notice the root directory in the location block for acme-challenge is not what Certbot used (/var/www/html). Hence the 404 Not Found.

So, either this root setting was changed or something happened to the Certbot renewal config file. Probably the former.

The easiest "fix" is just to change this root to be /var/www/html and test it with

sudo certbot renew --dry-run

If that works run it without --dry-run

server {
    # Listen on ipv4
    listen 80;
    #listen [::]:80;

    server_name _;

    # Allow ACME challenge to be served over HTTP (don't redirect to HTTPS).
    location ~* ^/.well-known/acme-challenge/ {
        root /opt/www/well_known;
        try_files $uri =404;
        allow all;
    }

    # Redirect all insecure http requests to https.
    location / {
        return 301 https://$host$request_uri;
    }
}
3 Likes
27

I'm sorry I don't understand the syntax well enough to understand how to implement your suggestion. I see the second text block you've included matches the bottom of the existing config.
Are you suggesting I change

       root /opt/www/well_known;

to

       root /var/www/html;

?

2 Likes
28

Yes, exactly that.

3 Likes
29

Okay, that worked perfectly. Both the dry run and the live execution ("wet run"?) worked correctly and my cert now expires in August :slight_smile:

I'll leave it like this but I have no explanation at all for how this stopped working. It's certainly nothing I changed, and I can't believe the iRedMail people would change anything like this in one of their version update scripts. Baffling.

An extra special thanks -- letsencrypt has the very best support anywhere!

5 Likes
30

:folded_hands:

4 Likes