Dollars to donuts we're gonna see some duplicate/incomplete server blocks. Just my hunch.
I concur with that conclusion. It might lead to some confusion on occasion though, as seen here.
Very likely although nginx is far less prone to such things compared to Apache. nginx actually warns about common overlaps
Still, problems with listen
statements or even possibly location
blocks are common causes. Given they are not aware of changes it might be something more subtle but we'll see.
I wouldn't bet dollars or donuts at the moment
Nginx could be internally proxying the challenge file requests to somewhere else. Only the config shall tell.
Using webroot method means no interaction/manipulation with nginx internal behavior.
(I know you know this, @MikeMcQ. Just saying for our studio audience.)
Here's the config. Sorry but putting it in a separate file is a bit beyond me in the 90 seconds I could get away from something even more urgent.
[root@elm html]# nginx -T
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful
# configuration file /etc/nginx/nginx.conf:
user nginx;
worker_processes 1;
pid /var/run/nginx.pid;
events {
worker_connections 1024;
}
http {
include /etc/nginx/conf-enabled/*.conf;
include /etc/nginx/sites-enabled/*.conf;
}
# configuration file /etc/nginx/conf-enabled/0-general.conf:
map_hash_bucket_size 1024;
# configuration file /etc/nginx/conf-enabled/cache.conf:
map $sent_http_content_type $expires {
default off;
application/x-javascript 1d;
text/css 1d;
~image/ 1d;
}
expires $expires;
# configuration file /etc/nginx/conf-enabled/client_max_body_size.conf:
client_max_body_size 50m;
# configuration file /etc/nginx/conf-enabled/default_type.conf:
default_type application/octet-stream;
# configuration file /etc/nginx/conf-enabled/gzip.conf:
gzip on;
gzip_vary on;
gzip_http_version 1.0;
gzip_comp_level 6;
gzip_buffers 16 8k;
gzip_min_length 10240;
gzip_proxied any;
gzip_disable "MSIE [1-6]\.";
# text/html is always compressed.
gzip_types
text/plain
text/css
text/xml
text/javascript
text/json
text/vcard
text/cache-manifest
text/vnd.rim.location.xloc
text/vtt
text/x-component
text/x-cross-domain-policy
image/bmp
image/vnd.microsoft.icon
image/x-icon
image/svg+xml
font/truetype
font/opentype
application/atom+xml
application/javascript
application/json
application/ld+json
application/vnd.geo+json
application/manifest+json
application/x-javascript
application/x-font-ttf
application/x-web-app-manifest+json
application/xml
application/xml+rss
application/xhtml+xml
application/vnd.ms-fontobject;
# configuration file /etc/nginx/conf-enabled/headers.conf:
add_header X-Frame-Options sameorigin;
add_header X-Content-Type-Options nosniff;
add_header X-XSS-Protection '1; mode=block';
add_header X-Download-Options noopen;
add_header X-Permitted-Cross-Domain-Policies none;
add_header Content-Security-Policy "default-src https: data: 'unsafe-inline' 'unsafe-eval'";
add_header Referrer-Policy strict-origin;
# configuration file /etc/nginx/conf-enabled/log.conf:
access_log /var/log/nginx/access.log;
error_log /var/log/nginx/error.log;
# configuration file /etc/nginx/conf-enabled/mime_types.conf:
include /etc/nginx/mime.types;
# configuration file /etc/nginx/mime.types:
types {
text/html html htm shtml;
text/css css;
text/xml xml;
image/gif gif;
image/jpeg jpeg jpg;
application/javascript js;
application/atom+xml atom;
application/rss+xml rss;
text/mathml mml;
text/plain txt;
text/vnd.sun.j2me.app-descriptor jad;
text/vnd.wap.wml wml;
text/x-component htc;
image/png png;
image/svg+xml svg svgz;
image/tiff tif tiff;
image/vnd.wap.wbmp wbmp;
image/webp webp;
image/x-icon ico;
image/x-jng jng;
image/x-ms-bmp bmp;
font/woff woff;
font/woff2 woff2;
application/java-archive jar war ear;
application/json json;
application/mac-binhex40 hqx;
application/msword doc;
application/pdf pdf;
application/postscript ps eps ai;
application/rtf rtf;
application/vnd.apple.mpegurl m3u8;
application/vnd.google-earth.kml+xml kml;
application/vnd.google-earth.kmz kmz;
application/vnd.ms-excel xls;
application/vnd.ms-fontobject eot;
application/vnd.ms-powerpoint ppt;
application/vnd.oasis.opendocument.graphics odg;
application/vnd.oasis.opendocument.presentation odp;
application/vnd.oasis.opendocument.spreadsheet ods;
application/vnd.oasis.opendocument.text odt;
application/vnd.openxmlformats-officedocument.presentationml.presentation
pptx;
application/vnd.openxmlformats-officedocument.spreadsheetml.sheet
xlsx;
application/vnd.openxmlformats-officedocument.wordprocessingml.document
docx;
application/vnd.wap.wmlc wmlc;
application/x-7z-compressed 7z;
application/x-cocoa cco;
application/x-java-archive-diff jardiff;
application/x-java-jnlp-file jnlp;
application/x-makeself run;
application/x-perl pl pm;
application/x-pilot prc pdb;
application/x-rar-compressed rar;
application/x-redhat-package-manager rpm;
application/x-sea sea;
application/x-shockwave-flash swf;
application/x-stuffit sit;
application/x-tcl tcl tk;
application/x-x509-ca-cert der pem crt;
application/x-xpinstall xpi;
application/xhtml+xml xhtml;
application/xspf+xml xspf;
application/zip zip;
application/octet-stream bin exe dll;
application/octet-stream deb;
application/octet-stream dmg;
application/octet-stream iso img;
application/octet-stream msi msp msm;
audio/midi mid midi kar;
audio/mpeg mp3;
audio/ogg ogg;
audio/x-m4a m4a;
audio/x-realaudio ra;
video/3gpp 3gpp 3gp;
video/mp2t ts;
video/mp4 mp4;
video/mpeg mpeg mpg;
video/quicktime mov;
video/webm webm;
video/x-flv flv;
video/x-m4v m4v;
video/x-mng mng;
video/x-ms-asf asx asf;
video/x-ms-wmv wmv;
video/x-msvideo avi;
}
# configuration file /etc/nginx/conf-enabled/php_fpm.conf:
upstream php_workers {
server 127.0.0.1:9999;
}
# configuration file /etc/nginx/conf-enabled/sendfile.conf:
sendfile on;
# configuration file /etc/nginx/conf-enabled/server_tokens.conf:
# Hide Nginx version number
server_tokens off;
# configuration file /etc/nginx/conf-enabled/types_hash_max_size.conf:
types_hash_max_size 2048;
# configuration file /etc/nginx/sites-enabled/00-default-ssl.conf:
#
# Note: This file must be loaded before other virtual host config files,
#
# HTTPS
server {
listen 443 ssl http2;
listen [::]:443 ssl http2;
server_name _;
root /var/www/html;
index index.php index.html;
include /etc/nginx/templates/misc.tmpl;
include /etc/nginx/templates/ssl.tmpl;
include /etc/nginx/templates/iredadmin.tmpl;
include /etc/nginx/templates/roundcube.tmpl;
include /etc/nginx/templates/sogo.tmpl;
include /etc/nginx/templates/netdata.tmpl;
include /etc/nginx/templates/php-catchall.tmpl;
include /etc/nginx/templates/stub_status.tmpl;
}
# configuration file /etc/nginx/templates/misc.tmpl:
# Allow access to '^/.well-known/'
location ~ ^/.well-known/ {
allow all;
access_log off;
log_not_found off;
autoindex off;
#root /var/www/html;
}
# Deny all attempts to access hidden files such as .htaccess.
location ~ /\. { deny all; }
# Handling noisy messages
location = /favicon.ico { access_log off; log_not_found off; }
location = /robots.txt { access_log off; log_not_found off; }
# configuration file /etc/nginx/templates/ssl.tmpl:
ssl_protocols TLSv1.2 TLSv1.3;
# Fix 'The Logjam Attack'.
ssl_ciphers EECDH+CHACHA20:EECDH+AESGCM:EDH+AESGCM:AES256+EECDH;
ssl_prefer_server_ciphers on;
ssl_dhparam /etc/pki/tls/dh2048_param.pem;
# Greatly improve the performance of keep-alive connections over SSL.
# With this enabled, client is not necessary to do a full SSL-handshake for
# every request, thus saving time and cpu-resources.
ssl_session_cache shared:SSL:10m;
# To use your own ssl cert (e.g. "Let's Encrypt"), please create symbol link to
# ssl cert/key used below, so that we can manage this config file with Ansible.
#
# For example:
#
# rm -f /etc/pki/tls/private/iRedMail.key
# rm -f /etc/pki/tls/certs/iRedMail.crt
# ln -s /etc/letsencrypt/live/<domain>/privkey.pem /etc/pki/tls/private/iRedMail.key
# ln -s /etc/letsencrypt/live/<domain>/fullchain.pem /etc/pki/tls/certs/iRedMail.crt
#
# To request free "Let's Encrypt" cert, please check our tutorial:
# https://docs.iredmail.org/letsencrypt.html
ssl_certificate /etc/pki/tls/certs/iRedMail.crt;
ssl_certificate_key /etc/pki/tls/private/iRedMail.key;
# configuration file /etc/nginx/templates/iredadmin.tmpl:
# Settings for iRedAdmin.
# static files under /iredadmin/static
location ~ ^/iredadmin/static/(.*) {
alias /opt/www/iredadmin/static/$1;
}
# Python scripts
location ~ ^/iredadmin(.*) {
rewrite ^/iredadmin(/.*)$ $1 break;
include /etc/nginx/templates/hsts.tmpl;
include uwsgi_params;
uwsgi_pass 127.0.0.1:7791;
uwsgi_param UWSGI_CHDIR /opt/www/iredadmin;
uwsgi_param UWSGI_SCRIPT iredadmin;
uwsgi_param SCRIPT_NAME /iredadmin;
# Access control
#allow 127.0.0.1;
#allow 192.168.1.10;
#allow 192.168.1.0/24;
#deny all;
}
# iRedAdmin: redirect /iredadmin to /iredadmin/
location = /iredadmin {
rewrite ^ /iredadmin/;
}
# Handle newsletter-style subscription/unsubscription supported in iRedAdmin-Pro.
location ~ ^/newsletter/ {
rewrite /newsletter/(.*) /iredadmin/newsletter/$1 last;
}
# configuration file /etc/nginx/templates/hsts.tmpl:
# Use HTTP Strict Transport Security to force client to use secure
# connections only. References:
#
# * RFC Document (6797): HTTP Strict Transport Security (HSTS)
# https://tools.ietf.org/html/rfc6797#section-6.1.2
#
# * Short tutorial from Mozilla:
# https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security
#
# WARNING: According to RFC document, HSTS will fail with self-signed SSL
# certificate.
# https://tools.ietf.org/html/rfc6797#page-27
#
# Syntax:
#
# Strict-Transport-Security: max-age=expireTime [; includeSubDomains] [; preload]
add_header Strict-Transport-Security "max-age=31536000";
# configuration file /etc/nginx/uwsgi_params:
uwsgi_param QUERY_STRING $query_string;
uwsgi_param REQUEST_METHOD $request_method;
uwsgi_param CONTENT_TYPE $content_type;
uwsgi_param CONTENT_LENGTH $content_length;
uwsgi_param REQUEST_URI $request_uri;
uwsgi_param PATH_INFO $document_uri;
uwsgi_param DOCUMENT_ROOT $document_root;
uwsgi_param SERVER_PROTOCOL $server_protocol;
uwsgi_param REQUEST_SCHEME $scheme;
uwsgi_param HTTPS $https if_not_empty;
uwsgi_param REMOTE_ADDR $remote_addr;
uwsgi_param REMOTE_PORT $remote_port;
uwsgi_param SERVER_PORT $server_port;
uwsgi_param SERVER_NAME $server_name;
# configuration file /etc/nginx/templates/roundcube.tmpl:
#
# Running Roundcube as a subfolder on an existing virtual host
#
# Block access to default directories and files under these directories
location ~ ^/mail/(bin|config|installer|logs|SQL|temp|vendor)($|/.*) { deny all; }
# Block access to default files under top-directory and files start with same name.
location ~ ^/mail/(CHANGELOG|composer.json|INSTALL|jsdeps.json|LICENSE|README|UPGRADING)($|.*) { deny all; }
# Block plugin config files and sample config files.
location ~ ^/mail/plugins/.*/config.inc.php.* { deny all; }
# Block access to plugin data
location ~ ^/mail/plugins/enigma/home($|/.*) { deny all; }
# Redirect URI `/mail` to `/mail/`.
location = /mail {
return 301 /mail/;
}
location ~ ^/mail/(.*\.php)$ {
include /etc/nginx/templates/hsts.tmpl;
include /etc/nginx/templates/fastcgi_php.tmpl;
fastcgi_param SCRIPT_FILENAME /opt/www/roundcubemail/$1;
}
location ~ ^/mail/(.*) {
alias /opt/www/roundcubemail/$1;
index index.php;
}
# configuration file /etc/nginx/templates/fastcgi_php.tmpl:
#
# Template used to handle PHP fastcgi applications
#
# You still need to define `SCRIPT_FILENAME` for your PHP application, and
# probably `fastcgi_index` if your application use different index file.
#
include fastcgi_params;
# Directory index file
fastcgi_index index.php;
# Handle PHP files with upstream handler
fastcgi_pass php_workers;
# Fix the HTTPROXY issue.
# Reference: https://httpoxy.org/
fastcgi_param HTTP_PROXY '';
# configuration file /etc/nginx/fastcgi_params:
fastcgi_param QUERY_STRING $query_string;
fastcgi_param REQUEST_METHOD $request_method;
fastcgi_param CONTENT_TYPE $content_type;
fastcgi_param CONTENT_LENGTH $content_length;
fastcgi_param SCRIPT_NAME $fastcgi_script_name;
fastcgi_param REQUEST_URI $request_uri;
fastcgi_param DOCUMENT_URI $document_uri;
fastcgi_param DOCUMENT_ROOT $document_root;
fastcgi_param SERVER_PROTOCOL $server_protocol;
fastcgi_param REQUEST_SCHEME $scheme;
fastcgi_param HTTPS $https if_not_empty;
fastcgi_param GATEWAY_INTERFACE CGI/1.1;
fastcgi_param SERVER_SOFTWARE nginx/$nginx_version;
fastcgi_param REMOTE_ADDR $remote_addr;
fastcgi_param REMOTE_PORT $remote_port;
fastcgi_param SERVER_ADDR $server_addr;
fastcgi_param SERVER_PORT $server_port;
fastcgi_param SERVER_NAME $server_name;
# PHP only, required if PHP was built with --enable-force-cgi-redirect
fastcgi_param REDIRECT_STATUS 200;
# configuration file /etc/nginx/templates/sogo.tmpl:
# Settings for SOGo Groupware
# SOGo
location ~ ^/sogo { rewrite ^ https://$host/SOGo; }
location ~ ^/SOGO { rewrite ^ https://$host/SOGo; }
# Redirect /mail to /SOGo
#location ~ ^/mail { rewrite ^ https://$host/SOGo; }
# For Mac OS X and iOS devices.
rewrite ^/.well-known/caldav /SOGo/dav permanent;
rewrite ^/.well-known/carddav /SOGo/dav permanent;
rewrite ^/principals /SOGo/dav permanent;
location ^~ /SOGo {
include /etc/nginx/templates/hsts.tmpl;
proxy_pass http://127.0.0.1:20000;
# forward user's IP address
proxy_set_header X-Real-IP $remote_addr;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_set_header Host $host;
# always use https
proxy_set_header x-webobjects-server-port $server_port;
proxy_set_header x-webobjects-server-name $host;
proxy_set_header x-webobjects-server-url https://$host;
proxy_set_header x-webobjects-server-protocol HTTP/1.0;
proxy_busy_buffers_size 64k;
proxy_buffers 8 64k;
proxy_buffer_size 64k;
}
location ^~ /Microsoft-Server-ActiveSync {
proxy_pass http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync;
proxy_connect_timeout 3540;
proxy_send_timeout 3540;
proxy_read_timeout 3540;
proxy_busy_buffers_size 64k;
proxy_buffers 8 64k;
proxy_buffer_size 64k;
}
location ^~ /SOGo/Microsoft-Server-ActiveSync {
proxy_pass http://127.0.0.1:20000/SOGo/Microsoft-Server-ActiveSync;
proxy_connect_timeout 3540;
proxy_send_timeout 3540;
proxy_read_timeout 3540;
proxy_busy_buffers_size 64k;
proxy_buffers 8 64k;
proxy_buffer_size 64k;
}
location /SOGo.woa/WebServerResources/ {
alias /usr/lib64/GNUstep/SOGo/WebServerResources/;
expires max;
}
location /SOGo/WebServerResources/ {
alias /usr/lib64/GNUstep/SOGo/WebServerResources/;
expires max;
}
location ^/SOGo/so/ControlPanel/Products/([^/]*)/Resources/(.*)$ {
alias /usr/lib64/GNUstep/SOGo/$1.SOGo/Resources/$2;
expires max;
}
# configuration file /etc/nginx/templates/netdata.tmpl:
# Running netdata as a subfolder to an existing virtual host
# FYI: https://github.com/firehol/netdata/wiki/Running-behind-nginx
location = /netdata {
return 301 /netdata/;
}
location ~ /netdata/(?<ndpath>.*) {
proxy_redirect off;
proxy_set_header Host $host;
proxy_set_header X-Forwarded-Host $host;
proxy_set_header X-Forwarded-Server $host;
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
proxy_http_version 1.1;
proxy_pass_request_headers on;
proxy_set_header Connection "keep-alive";
proxy_store off;
proxy_pass http://127.0.0.1:19999/$ndpath$is_args$args;
gzip on;
gzip_proxied any;
gzip_types *;
auth_basic "Authentication Required";
auth_basic_user_file /etc/nginx/netdata.users;
}
# configuration file /etc/nginx/templates/php-catchall.tmpl:
# Normal PHP scripts
location ~ \.php$ {
include /etc/nginx/templates/fastcgi_php.tmpl;
fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
}
# configuration file /etc/nginx/templates/stub_status.tmpl:
location = /stub_status {
stub_status on;
access_log off;
allow 127.0.0.1;
deny all;
}
location = /status {
include fastcgi_params;
fastcgi_pass php_workers;
fastcgi_param SCRIPT_FILENAME $fastcgi_script_name;
access_log off;
allow 127.0.0.1;
deny all;
}
# configuration file /etc/nginx/sites-enabled/00-default.conf:
#
# Note: This file must be loaded before other virtual host config files,
#
# HTTP
server {
# Listen on ipv4
listen 80;
#listen [::]:80;
server_name _;
# Allow ACME challenge to be served over HTTP (don't redirect to HTTPS).
location ~* ^/.well-known/acme-challenge/ {
root /opt/www/well_known;
try_files $uri =404;
allow all;
}
# Redirect all insecure http requests to https.
location / {
return 301 https://$host$request_uri;
}
}
[root@elm html]#
Your only port 80 server block is the one below so handles any HTTP (port 80) request inbound to your nginx
Notice the root
directory in the location block for acme-challenge is not what Certbot used (/var/www/html
). Hence the 404 Not Found.
So, either this root setting was changed or something happened to the Certbot renewal config file. Probably the former.
The easiest "fix" is just to change this root
to be /var/www/html
and test it with
sudo certbot renew --dry-run
If that works run it without --dry-run
server {
# Listen on ipv4
listen 80;
#listen [::]:80;
server_name _;
# Allow ACME challenge to be served over HTTP (don't redirect to HTTPS).
location ~* ^/.well-known/acme-challenge/ {
root /opt/www/well_known;
try_files $uri =404;
allow all;
}
# Redirect all insecure http requests to https.
location / {
return 301 https://$host$request_uri;
}
}
I'm sorry I don't understand the syntax well enough to understand how to implement your suggestion. I see the second text block you've included matches the bottom of the existing config.
Are you suggesting I change
root /opt/www/well_known;
to
root /var/www/html;
?
Yes, exactly that.
Okay, that worked perfectly. Both the dry run and the live execution ("wet run"?) worked correctly and my cert now expires in August
I'll leave it like this but I have no explanation at all for how this stopped working. It's certainly nothing I changed, and I can't believe the iRedMail people would change anything like this in one of their version update scripts. Baffling.
An extra special thanks -- letsencrypt has the very best support anywhere!