Mastodon, Type: Unauthorized, Invalid response from :404

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mas.greatlakes.club

I ran this command: certbot certonly -d mas.greatlakes.club (after systemctl stop nginx)

It produced this output: Detail: 24.247.183.12 (correct ip) : Invalid response from http://mas.greatlakes.club/.well-known/acme-challenge/ ...

My web server is (include version): Nginx (standard mastodon install Nov 2022)

The operating system my web server runs on is (include version): Ubuntu 20

My hosting provider, if applicable, is: self

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 1.29.0

Mastodon host installed Nov 2022 worked perfectly until June 22. certbot renew tuns on standard systemd schedule.

Remember having to use --standalone at initial install because of ubuntu/nginx weirdness. Went through all remembered iterations of how to check or renew SSL but all yield same result. It is as if acme challenge file is missing or something. ((404??))

How to fix?

2

Hi @greatlakesclub, and welcome to the LE community forum :slight_smile:

I suppose the "Mastadon" in the topic title has delayed any responses.
I, for one, can't say I know anything about "Mastadon".

That said, you mentioned nginx as well.
That, I think we can troubleshoot.
With the output of:
nginx -T

3 Likes
3

This machine hosts two different domains: mast.thomasdmn.net cames first and then mas.greatlakes.club was added. Both worked perfectly until less than a week ago and now both say SSL error or failure to all incoming requests.

mas.greatlakes.club settings were added via the /sites-enabled/ directory and separate conf file. These settings can be seen starting line 352 in the output below.

# configuration file /etc/nginx/nginx.conf:
user www-data;
worker_processes auto;
pid /run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
	worker_connections 768;
	# multi_accept on;
}

http {

	##
	# Basic Settings
	##

	sendfile on;
	tcp_nopush on;
	types_hash_max_size 2048;
	# server_tokens off;

	# server_names_hash_bucket_size 64;
	# server_name_in_redirect off;

	include /etc/nginx/mime.types;
	default_type application/octet-stream;

	##
	# SSL Settings
	##

	ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
	ssl_prefer_server_ciphers on;

	##
	# Logging Settings
	##

	access_log /var/log/nginx/access.log;
	error_log /var/log/nginx/error.log;

	##
	# Gzip Settings
	##

	gzip on;

	# gzip_vary on;
	# gzip_proxied any;
	# gzip_comp_level 6;
	# gzip_buffers 16 8k;
	# gzip_http_version 1.1;
	# gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript;

	##
	# Virtual Host Configs
	##

	include /etc/nginx/conf.d/*.conf;
	include /etc/nginx/sites-enabled/*;
}


#mail {
#	# See sample authentication script at:
#	# http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#	# auth_http localhost/auth.php;
#	# pop3_capabilities "TOP" "USER";
#	# imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#	server {
#		listen     localhost:110;
#		protocol   pop3;
#		proxy      on;
#	}
#
#	server {
#		listen     localhost:143;
#		protocol   imap;
#		proxy      on;
#	}
#}

# configuration file /etc/nginx/modules-enabled/50-mod-http-geoip2.conf:
load_module modules/ngx_http_geoip2_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-image-filter.conf:
load_module modules/ngx_http_image_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf:
load_module modules/ngx_http_xslt_filter_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-mail.conf:
load_module modules/ngx_mail_module.so;

# configuration file /etc/nginx/modules-enabled/50-mod-stream.conf:
load_module modules/ngx_stream_module.so;

# configuration file /etc/nginx/modules-enabled/70-mod-stream-geoip2.conf:
load_module modules/ngx_stream_geoip2_module.so;

# configuration file /etc/nginx/mime.types:

types {
    text/html                             html htm shtml;
    text/css                              css;
    text/xml                              xml;
    image/gif                             gif;
    image/jpeg                            jpeg jpg;
    application/javascript                js;
    application/atom+xml                  atom;
    application/rss+xml                   rss;

    text/mathml                           mml;
    text/plain                            txt;
    text/vnd.sun.j2me.app-descriptor      jad;
    text/vnd.wap.wml                      wml;
    text/x-component                      htc;

    image/avif                            avif;
    image/png                             png;
    image/svg+xml                         svg svgz;
    image/tiff                            tif tiff;
    image/vnd.wap.wbmp                    wbmp;
    image/webp                            webp;
    image/x-icon                          ico;
    image/x-jng                           jng;
    image/x-ms-bmp                        bmp;

    font/woff                             woff;
    font/woff2                            woff2;

    application/java-archive              jar war ear;
    application/json                      json;
    application/mac-binhex40              hqx;
    application/msword                    doc;
    application/pdf                       pdf;
    application/postscript                ps eps ai;
    application/rtf                       rtf;
    application/vnd.apple.mpegurl         m3u8;
    application/vnd.google-earth.kml+xml  kml;
    application/vnd.google-earth.kmz      kmz;
    application/vnd.ms-excel              xls;
    application/vnd.ms-fontobject         eot;
    application/vnd.ms-powerpoint         ppt;
    application/vnd.oasis.opendocument.graphics        odg;
    application/vnd.oasis.opendocument.presentation    odp;
    application/vnd.oasis.opendocument.spreadsheet     ods;
    application/vnd.oasis.opendocument.text            odt;
    application/vnd.openxmlformats-officedocument.presentationml.presentation    pptx;
    application/vnd.openxmlformats-officedocument.spreadsheetml.sheet    xlsx;
    application/vnd.openxmlformats-officedocument.wordprocessingml.document    docx;
    application/vnd.wap.wmlc              wmlc;
    application/wasm                      wasm;
    application/x-7z-compressed           7z;
    application/x-cocoa                   cco;
    application/x-java-archive-diff       jardiff;
    application/x-java-jnlp-file          jnlp;
    application/x-makeself                run;
    application/x-perl                    pl pm;
    application/x-pilot                   prc pdb;
    application/x-rar-compressed          rar;
    application/x-redhat-package-manager  rpm;
    application/x-sea                     sea;
    application/x-shockwave-flash         swf;
    application/x-stuffit                 sit;
    application/x-tcl                     tcl tk;
    application/x-x509-ca-cert            der pem crt;
    application/x-xpinstall               xpi;
    application/xhtml+xml                 xhtml;
    application/xspf+xml                  xspf;
    application/zip                       zip;

    application/octet-stream              bin exe dll;
    application/octet-stream              deb;
    application/octet-stream              dmg;
    application/octet-stream              iso img;
    application/octet-stream              msi msp msm;

    audio/midi                            mid midi kar;
    audio/mpeg                            mp3;
    audio/ogg                             ogg;
    audio/x-m4a                           m4a;
    audio/x-realaudio                     ra;

    video/3gpp                            3gpp 3gp;
    video/mp2t                            ts;
    video/mp4                             mp4;
    video/mpeg                            mpeg mpg;
    video/quicktime                       mov;
    video/webm                            webm;
    video/x-flv                           flv;
    video/x-m4v                           m4v;
    video/x-mng                           mng;
    video/x-ms-asf                        asx asf;
    video/x-ms-wmv                        wmv;
    video/x-msvideo                       avi;
}

# configuration file /etc/nginx/sites-enabled/default:
##
# You should look at the following URL's in order to grasp a solid understanding
# of Nginx configuration files in order to fully unleash the power of Nginx.
# https://www.nginx.com/resources/wiki/start/
# https://www.nginx.com/resources/wiki/start/topics/tutorials/config_pitfalls/
# https://wiki.debian.org/Nginx/DirectoryStructure
#
# In most cases, administrators will remove this file from sites-enabled/ and
# leave it as reference inside of sites-available where it will continue to be
# updated by the nginx packaging team.
#
# This file will automatically load configuration files provided by other
# applications, such as Drupal or Wordpress. These applications will be made
# available underneath a path with that package name, such as /drupal8.
#
# Please see /usr/share/doc/nginx-doc/examples/ for more detailed examples.
##

# Default server configuration
#
server {
	listen 80 default_server;
	listen [::]:80 default_server;

	# SSL configuration
	#
	# listen 443 ssl default_server;
	# listen [::]:443 ssl default_server;
	#
	# Note: You should disable gzip for SSL traffic.
	# See: https://bugs.debian.org/773332
	#
	# Read up on ssl_ciphers to ensure a secure configuration.
	# See: https://bugs.debian.org/765782
	#
	# Self signed certs generated by the ssl-cert package
	# Don't use them in a production server!
	#
	# include snippets/snakeoil.conf;

	root /var/www/html;

	# Add index.php to the list if you are using PHP
	index index.html index.htm index.nginx-debian.html;

	server_name _;

	location / {
		# First attempt to serve request as file, then
		# as directory, then fall back to displaying a 404.
		try_files $uri $uri/ =404;
	}

	# pass PHP scripts to FastCGI server
	#
	#location ~ \.php$ {
	#	include snippets/fastcgi-php.conf;
	#
	#	# With php-fpm (or other unix sockets):
	#	fastcgi_pass unix:/run/php/php7.4-fpm.sock;
	#	# With php-cgi (or other tcp sockets):
	#	fastcgi_pass 127.0.0.1:9000;
	#}

	# deny access to .htaccess files, if Apache's document root
	# concurs with nginx's one
	#
	#location ~ /\.ht {
	#	deny all;
	#}
}


# Virtual Host configuration for example.com
#
# You can move that to a different file under sites-available/ and symlink that
# to sites-enabled/ to enable it.
#
#server {
#	listen 80;
#	listen [::]:80;
#
#	server_name example.com;
#
#	root /var/www/example.com;
#	index index.html;
#
#	location / {
#		try_files $uri $uri/ =404;
#	}
#}

# configuration file /etc/nginx/sites-enabled/greatlakes:
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream backend2 {
    server 127.0.0.1:3001 fail_timeout=0;
}

upstream streaming2 {
    server 127.0.0.1:4001 fail_timeout=0;
}

proxy_cache_path /var/cache/nginx2 levels=1:2 keys_zone=CACHE2:10m inactive=7d max_size=1g;


# from orig conf file deleted by certbot edits
#server {
#  listen 80;
#  listen [::]:80;
#  server_name mast.thomasdmn.net;
#  root /home/mastodon/live/public;
#  location /.well-known/acme-challenge/ { allow all; }
#  location / { return 301 https://$host$request_uri; }
#}


# insterted by certbot -- redirects all HTTP traffic from 80 to 443 HTTPS
server {
    if ($host = mas.greatlakes.club) { return 301 https://$host$request_uri; } # managed by Certbot
  listen 80;
  listen [::]:80;
  server_name mas.greatlakes.club;
    return 404; # managed by Certbot
}


# inserted by CERTBOT killed all locations
#server {
 # server_name mast.thomasdmn.net;
 # root /home/mastodon/live/public;
 # location /.well-known/acme-challenge/ { allow all; }
 # location / { return 301 https://$host$request_uri; }

   #listen [::]:443 ssl ipv6only=on; # managed by Certbot
   # listen 443 ssl; # managed by Certbot
   # ssl_certificate /etc/letsencrypt/live/mast.thomasdmn.net/fullchain.pem; # managed by Certbot
   # ssl_certificate_key /etc/letsencrypt/live/mast.thomasdmn.net/privkey.pem; # managed by Certbot
   # include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
   # ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

#}


###########################################
#  insert new server for mas.greatlakes.club 12-17-22
###########################################


server {
  #listen 443 ssl http2;
  #listen [::]:443 ssl http2;
  #server_name mast.thomasdmn.net;
  #rewrote edits from certbot -- orig cause too many redirects.


   listen [::]:443 ssl http2 ipv6only=on; # managed by Certbot
   listen 443 ssl http2; # managed by Certbot
   server_name mas.greatlakes.club;
   root /home/mastodon/live/public;
   
   location /.well-known/acme-challenge/ { allow all; }
# inserted by certbot causes too many redirects - duplicate location   location / { return 301 https://$host$request_uri; }
    ssl_certificate /etc/letsencrypt/live/mas.greatlakes.club/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mas.greatlakes.club/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


#  ssl_protocols TLSv1.2 TLSv1.3;
#  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
#  ssl_prefer_server_ciphers on;
#  ssl_session_cache shared:SSL:10m;
#  ssl_session_tickets off;

  # Uncomment these lines once you acquire a certificate:
  # ssl_certificate     /etc/letsencrypt/live/mast.thomasdmn.net/fullchain.pem;
  # ssl_certificate_key /etc/letsencrypt/live/mast.thomasdmn.net/privkey.pem;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 80m;

  #root /home/mastodon/live/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml image/x-icon;

  location / {
    try_files $uri @proxy;
  }

  # If Docker is used for deployment and Rails serves static files,
  # then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.
  location = /sw.js {
    add_header Cache-Control "public, max-age=604800, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/assets/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/avatars/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/emoji/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/headers/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/packs/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/shortcuts/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/sounds/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/system/ {
    root /mnt/greatlakes/live/public;
    add_header Cache-Control "public, max-age=2419200, immutable";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ^~ /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";

    proxy_pass http://streaming2;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";

    tcp_nodelay on;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://backend2;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache CACHE2;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;

    tcp_nodelay on;
  }

  error_page 404 500 501 502 503 504 /500.html;

}




# certbot inserted end of file I moved to front as is
#server {
#    if ($host = mast.thomasdmn.net) {
#        return 301 https://$host$request_uri;
#    } # managed by Certbot

#  listen 80;
#  listen [::]:80;
#  server_name mast.thomasdmn.net;
#    return 404; # managed by Certbot
#}

# configuration file /etc/letsencrypt/options-ssl-nginx.conf:
# This file contains important security parameters. If you modify this file
# manually, Certbot will be unable to automatically provide future security
# updates. Instead, Certbot will print and log an error message with a path to
# the up-to-date file that you will need to refer to when manually updating
# this file. Contents are based on https://ssl-config.mozilla.org

ssl_session_cache shared:le_nginx_SSL:10m;
ssl_session_timeout 1440m;
ssl_session_tickets off;

ssl_protocols TLSv1.2 TLSv1.3;
ssl_prefer_server_ciphers off;

ssl_ciphers "ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384";

# configuration file /etc/nginx/sites-enabled/mastodon:
map $http_upgrade $connection_upgrade {
  default upgrade;
  ''      close;
}

upstream backend {
    server 127.0.0.1:3000 fail_timeout=0;
}

upstream streaming {
    server 127.0.0.1:4000 fail_timeout=0;
}

proxy_cache_path /var/cache/nginx levels=1:2 keys_zone=CACHE:10m inactive=7d max_size=1g;


# from orig conf file deleted by certbot edits
#server {
#  listen 80;
#  listen [::]:80;
#  server_name mast.thomasdmn.net;
#  root /home/mastodon/live/public;
#  location /.well-known/acme-challenge/ { allow all; }
#  location / { return 301 https://$host$request_uri; }
#}


# insterted by certbot -- redirects all HTTP traffic from 80 to 443 HTTPS
server {
    if ($host = mast.thomasdmn.net) { return 301 https://$host$request_uri; } # managed by Certbot
  listen 80;
  listen [::]:80;
  server_name mast.thomasdmn.net;
    return 404; # managed by Certbot
}


# inserted by CERTBOT killed all locations
#server {
 # server_name mast.thomasdmn.net;
 # root /home/mastodon/live/public;
 # location /.well-known/acme-challenge/ { allow all; }
 # location / { return 301 https://$host$request_uri; }

   #listen [::]:443 ssl ipv6only=on; # managed by Certbot
   # listen 443 ssl; # managed by Certbot
   # ssl_certificate /etc/letsencrypt/live/mast.thomasdmn.net/fullchain.pem; # managed by Certbot
   # ssl_certificate_key /etc/letsencrypt/live/mast.thomasdmn.net/privkey.pem; # managed by Certbot
   # include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
   # ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

#}

server {
  #listen 443 ssl http2;
  #listen [::]:443 ssl http2;
  #server_name mast.thomasdmn.net;

   #rewrote edits from certbot -- orig cause too many redirects.

   listen [::]:443 ssl;  # broke when added greatlakes  http2;     ipv6only=on; # managed by Certbot
   listen 443 ssl http2; # managed by Certbot
   server_name mast.thomasdmn.net;
   root /home/mastodon/live/public;

   location /.well-known/acme-challenge/ { allow all; }
# inserted by certbot casues too many redirects - duplicate location   location / { return 301 https://$host$request_uri; }
    ssl_certificate /etc/letsencrypt/live/mast.thomasdmn.net/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/mast.thomasdmn.net/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot


#  ssl_protocols TLSv1.2 TLSv1.3;
#  ssl_ciphers HIGH:!MEDIUM:!LOW:!aNULL:!NULL:!SHA;
#  ssl_prefer_server_ciphers on;
#  ssl_session_cache shared:SSL:10m;
#  ssl_session_tickets off;

  # Uncomment these lines once you acquire a certificate:
  # ssl_certificate     /etc/letsencrypt/live/mast.thomasdmn.net/fullchain.pem;
  # ssl_certificate_key /etc/letsencrypt/live/mast.thomasdmn.net/privkey.pem;

  keepalive_timeout    70;
  sendfile             on;
  client_max_body_size 80m;

  #root /home/mastodon/live/public;

  gzip on;
  gzip_disable "msie6";
  gzip_vary on;
  gzip_proxied any;
  gzip_comp_level 6;
  gzip_buffers 16 8k;
  gzip_http_version 1.1;
  gzip_types text/plain text/css application/json application/javascript text/xml application/xml application/xml+rss text/javascript image/svg+xml image/x-icon;

  location / {
    try_files $uri @proxy;
  }

  # If Docker is used for deployment and Rails serves static files,
  # then needed must replace line `try_files $uri =404;` with `try_files $uri @proxy;`.
  location = /sw.js {
    add_header Cache-Control "public, max-age=604800, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/assets/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/avatars/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/emoji/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/headers/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/packs/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/shortcuts/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/sounds/ {
    add_header Cache-Control "public, max-age=2419200, must-revalidate";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ~ ^/system/ {
    root /mnt/mastodon/live/public;
    add_header Cache-Control "public, max-age=2419200, immutable";
    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";
    try_files $uri =404;
  }

  location ^~ /api/v1/streaming {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";

    proxy_pass http://streaming;
    proxy_buffering off;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    add_header Strict-Transport-Security "max-age=63072000; includeSubDomains";

    tcp_nodelay on;
  }

  location @proxy {
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
    proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
    proxy_set_header X-Forwarded-Proto $scheme;
    proxy_set_header Proxy "";
    proxy_pass_header Server;

    proxy_pass http://backend;
    proxy_buffering on;
    proxy_redirect off;
    proxy_http_version 1.1;
    proxy_set_header Upgrade $http_upgrade;
    proxy_set_header Connection $connection_upgrade;

    proxy_cache CACHE;
    proxy_cache_valid 200 7d;
    proxy_cache_valid 410 24h;
    proxy_cache_use_stale error timeout updating http_500 http_502 http_503 http_504;
    add_header X-Cached $upstream_cache_status;

    tcp_nodelay on;
  }

  error_page 404 500 501 502 503 504 /500.html;

}




# certbot inserted end of file I moved to front as is
#server {
#    if ($host = mast.thomasdmn.net) {
#        return 301 https://$host$request_uri;
#    } # managed by Certbot

#  listen 80;
#  listen [::]:80;
#  server_name mast.thomasdmn.net;
#    return 404; # managed by Certbot
#}
4

Sorry. Boy did that make a mess. Any way to paste a conf file without it parsing thinking it is code?

5

nginxt_062723.txt (23.6 KB)
oh! well there ya go...

1 Like
6

Yes, by adding three backticks before and after the content. Or, using the Preformatted Text formatting tool in the post menu.

Was great you uploaded it. That's the best. Someone will review and reply

3 Likes
7

What was the rest of that command this time? This command would have prompted for the kind of authentication to use.

Note when you stop nginx like you note but then try the nginx authentication that can cause trouble. You only stop nginx for standalone method. But, if you have a working nginx there is rarely any need to stop it to use standalone and instead should use webroot or --nginx plug-in.

Also, can you show the .conf file for greatlakes in the /etc/letsencrypt/renewal folder? You can try the new 3 backticks technique :slight_smile:

3 Likes
8

2 Standalone. I never could figure out why certbot commands using nginx always failed so Installed and renew using standalone each command.

Some command I ran yest showed cert valid until Aug 2023 so shouldnt be expired. SSL issue appeared after we had a power blackout and the server went dark for 6 hours after UPS ran dry.

# renew_before_expiry = 30 days
version = 1.29.0
archive_dir = /etc/letsencrypt/archive/mas.greatlakes.club
cert = /etc/letsencrypt/live/mas.greatlakes.club/cert.pem
privkey = /etc/letsencrypt/live/mas.greatlakes.club/privkey.pem
chain = /etc/letsencrypt/live/mas.greatlakes.club/chain.pem
fullchain = /etc/letsencrypt/live/mas.greatlakes.club/fullchain.pem

# Options used in the renewal process
[renewalparams]
account = b201d051fdfa01a0d65b989ed5fbd8e0
authenticator = nginx
server = https://acme-v02.api.letsencrypt.org/directory
key_type = rsa
installer = nginx
9

Well, standalone didn't work either so I think it best if we sort out why nginx plug-in failed. Or, at least use --webroot instead and get away from standalone.

Notice in the renewal conf file it shows using nginx as authenticator and installer. So, if you have a certbot renew scheduled in cron or systemd timer that is what is getting tried. And, is what we want working so renewal is automated.

To begin, have nginx running then try this

certbot certonly --nginx --dry-run -d mas.greatlakes.club

This is purely a test. Just show result of this and we'll proceed from that.

3 Likes
10 
Simulating renewal of an existing certificate for mas.greatlakes.club

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: mas.greatlakes.club
  Type:   unauthorized
  Detail: 24.247.183.12: Invalid response from http://mas.greatlakes.club/.well-known/acme-challenge/hKbrTV9WaCQA3ML4rfX-QilcwdfUUEnx1FwGOHo5RqY: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.
11

also not shown in text dump:

"Hint: The Certificate Authority failed to verify the temporary nginx confiiguration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet."

Shown IP addr is correct

porkbun is dns host:

TYPE 	HOST	ANSWER	TTL
|A|*.greatlakes.club|24.247.183.12|600|||
| --- | --- | --- | --- | --- | --- |
|A|mas.greatlakes.club|24.247.183.12|600|||
|A|greatlakes.club|24.247.183.12|600|
12

Can you upload the log file from that failed run?

It's at /var/log/letsencrypt/letsencrypt.log

The 404 Not Found means the nginx plug-in was not able to effect its temp server block changes. The log should help us see why

3 Likes
13

certbot_certificates_062723.txt (904 Bytes)
letsencrypt.log.txt (480.7 KB)

14

Only error I can see is both domains use root /home/mastodon/live/public which shouldnt be a problem? To keep the domains clean from each other I should change mas.greatlakes.club to use root /home/greatlakes/live/public

But both are avail perms for read and write of web server files so should not be our crunkle-dug

I can update after we figure out why I am pulling out my nginx hair. = : )

15

It is not a problem. Don't change anything. I am reviewing. The root folder is not involved when using the --nginx plug-in. It inserts its own rewrite and return statements into the server block to provide the response.

3 Likes
16

Can you try with this option?

certbot certonly --nginx --dry-run -d mas.greatlakes.club --nginx-sleep-seconds 15

And, please use above command. It looks like you used renew for both domains in previous log. This keeps it easier for me to debug - which is nice :slight_smile:

Just show the result of this don't need a full log again. If this doesn't work we'll try --webroot as I don't see any conflicts with certbot and your nginx config in the log.

3 Likes
17

I am still curious about results of my previous comment. But, I think you have something more fundamental failing. It looks like you have a Cisco router pre-processing the HTTP requests before it reaches your nginx. And, it isn't aware of your ACME cert requests so prevents it working.

Why do I say this?

I am seeing this which is odd. I did not see where you disabled HEAD requests (-I) in the nginx config to explain this

curl -I http://mas.greatlakes.club
HTTP/1.1 500 Internal Server Error
Server: nginx/1.16.1

But, even a GET request (-i) does this. Which looks OK until we note that the nginx HTTP server block is supposed to redirect this to HTTPS. And, it isn't. Instead, we see some sort of page with lots of Cisco notices.

curl -i http://mas.greatlakes.club
HTTP/1.1 200 OK
Server: nginx/1.16.1
Date: Tue, 27 Jun 2023 21:26:31 GMT
Content-Type: text/html
Transfer-Encoding: chunked
Connection: keep-alive

<!DOCTYPE HTML PUBLIC "-//W3C//DTD HTML 4.01 Transitional//EN">
<html xmlns="http://www.w3.org/1999/xhtml">
<head>
<meta name="Pragma" content="No-Cache">
<link type="image/x-icon" rel="shortcut icon" href="/images/cisco.png">
...
<td valign="bottom" height="20" class="Copyright" colspan="3" style="text-align:justify">&#169; 
2015 Cisco Systems, Inc. All Rights Reserved.
Cisco, Cisco Systems, and the Cisco Systems logo are registered trademarks 
or trademarks of Cisco Systems, Inc. and/or it's affiliates in the United States 
and certain other countries.
3 Likes
18

--nginx-sleep-seconds 15:

failed invalid response from (same error above above)

--webroot:

too many flags. what is thr proper context for --webroot?

19

We cross-posted. I'll give sample --webroot when you respond to my findings about Cisco

2 Likes
20

You are correct.
Spectrum cable modem (residential)
Cisco RV320 firewall
Server is DMZ port (192.168.9.100)

all working fine until this week

Do you think maybe Cisco FW is grabbing port 80 requests for cert stuff and jumbling instead of letting server handle like it was told to do?