SOLVED Cert Renewal Failure with NGINX Reverse Proxy

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
auth.legionofdeath.ru
I ran this command:

It produced this output:
sudo certbot certonly -v --dry-run -d auth.legionofdeath.ru
My web server is (include version):
nginx version: nginx/1.24.0
The operating system my web server runs on is (include version):
Ubuntu 22.04.4 LTS
My hosting provider, if applicable, is:
selfhosted
I can login to a root shell on my machine (yes or no, or I don't know):
yes sudoer
I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 2.10.0

Hi all! I was notified today that my certificate had expired and went to investigate why it didnt autorenew only to find certbot was having issues. I have all SSL terminating on my nginx reverse proxy and each site located on a separate vm with the reverse proxy exposed publicly. Everything on the site works fine. I have tried creating a catchall for acme challenges, i have tried converting the site to http only, and i am out of ideas at this point. permissions seem ok. i run certbot from my normal sudoer user revproxyadmin. nginx is using www-data for its user. everything in /var/www/html is owned by www-data... my internal dns points the domain name to the correct internal ip... I also created a test file in the acme folder i created and can access that fine internally on my network but not externally. If the site wasnt working properly I would think my portforwards were messed up... I will post my configs for nginx maybe at some point i messed something up? Please help!!!

Original config:

server {
     if ($host = auth.legionofdeath.ru) {
         return 301 https://$host$request_uri;
     } # managed by Certbot Redirect all HTTP requests to HTTPS with a 301 Moved Permanently response.

    listen 80;

    server_name auth.legionofdeath.ru;

}

server {
    listen 443 ssl http2;

    server_name auth.legionofdeath.ru;

    client_max_body_size 0;

    resolver 10.0.1.46 valid=30s ipv6=off;

    add_header X-Xss-Protection "1; mode=block" always;
    add_header X-Content-Type-Options "nosniff" always;
    add_header Strict-Transport-Security "max-age=15552000; preload" always;
    add_header X-Frame-Options "SAMEORIGIN" always;
    add_header 'Referrer-Policy' 'origin-when-cross-origin';
    add_header Content-Security-Policy "frame-ancestors legionofdeath.ru auth.legionofdeath.ru;";

    proxy_hide_header X-Powered-By;
    proxy_buffering off; ## Sends data as fast as it can not buffering large chunks.

    access_log /var/log/nginx/auth.legionofdeath.ru.access.log;
    error_log /var/log/nginx/auth.legionofdeath.ru.error.log warn;

    proxy_headers_hash_max_size 512;
    proxy_headers_hash_bucket_size 128;


    location / {
        set $upstream http://auth.legionofdeath.ru;

        proxy_pass $upstream;
        # proxy_pass http://10.0.1.76:80;

        proxy_set_header Range $http_range;
        proxy_set_header If-Range $http_if_range;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_redirect off;

        # The next three lines allow websockets
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Fix: upstream server temporarily disabled while connecting to upstream
        proxy_buffer_size 128k;
        proxy_buffers 4 256k;
        proxy_busy_buffers_size 256k;

        # Fix: a client request body is buffered to a temporary file
        client_body_buffer_size 2048M;
        # client_max_body_size 1024M;

        # Fix: upstream timed out (110: Connection timed out) while reading response header from upstream
        proxy_read_timeout 600;

        # Fix: upstream prematurely closed connection while reading response header from upstream
        proxy_connect_timeout 300;
    }
    ssl_certificate /etc/letsencrypt/live/curtpme.com-0001/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/curtpme.com-0001/privkey.pem; # managed by Certbot



}

Here is the latest config still not working:

server {
    listen 80;
    server_name auth.legionofdeath.ru;

    # Location block for ACME Challenge responses
    location ^~ /.well-known/acme-challenge/ {
        allow all;
        root /var/www/html;  # Adjust this to your webroot directory where Certbot will place challenge files
        default_type "text/plain";
        try_files $uri =404;  # Serve file if present, respond with 404 if not found
    }

    # Logging configuration
    access_log /var/log/nginx/auth.legionofdeath.ru.access.log;
    error_log /var/log/nginx/auth.legionofdeath.ru.error.log warn;

    # Proxy settings for all other requests
    location / {
        # Proxy all requests to the internal server on port 80
        proxy_pass http://10.0.1.76:80;

        # Proxy headers to ensure the remote server knows the original request details
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header X-Forwarded-Host $http_host;
        proxy_set_header X-Forwarded-Port $server_port;
        proxy_redirect off;

        # Enable support for Websockets
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";

        # Proxy timeout settings
        proxy_read_timeout 600;  # Timeout for reading a response from the proxy server
        proxy_connect_timeout 300;  # Timeout for establishing a connection to the proxy server
    }
}

Here is the nginx.conf for the reverse proxy:

user  www-data;
worker_processes  auto;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections  1024;
}


http {

    sendfile on;
    tcp_nopush on;
    tcp_nodelay on;
    keepalive_timeout 65;
    types_hash_max_size 2048;
    proxy_connect_timeout 1h;
    proxy_send_timeout 1h;
    proxy_read_timeout 1h;


    server_names_hash_bucket_size 64;


    include /etc/nginx/mime.types;
    default_type application/octet-stream;


    ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
    ssl_prefer_server_ciphers on;


    access_log /var/log/nginx/access.log;
    error_log /var/log/nginx/error.log;
    log_format default '$remote_addr - $http_CF_Connecting_IP - $remote_user [$time_local] "$request" '
    '$status $body_bytes_sent "$http_referer" '
    '"$http_user_agent" "$http_x_forwarded_for" $request_time $server_port';


    gzip_disable "msie6";
    gzip_comp_level 6;
    gzip_min_length 1100;
    gzip_buffers 16 8k;
    gzip_proxied any;
    gzip_types
    text/plain
    text/css
    text/js
    text/xml
    text/javascript
    application/javascript
    application/x-javascript
    application/json
    application/xml
    application/rss+xml
    image/svg+xml;


    include /etc/nginx/conf.d/*.conf;
    include /etc/nginx/sites-enabled/*;
}

Here is the nginx conf for the proxied server:

server {
    listen 80;
    server_name auth.legionofdeath.ru;

    location = /favicon.ico { access_log off; log_not_found off; }

    location /static {
        alias /var/www/myauth/static;
        autoindex off;
    }

    location /robots.txt {
        alias /var/www/myauth/static/robots.txt;
    }

    # Gunicorn config goes below
    location / {
        include proxy_params;
        proxy_pass http://127.0.0.1:8000;
    }
}

Here is the nginx.conf for the proxied server:

user  www-data;
worker_processes  auto;

error_log  /var/log/nginx/error.log;
pid        /var/run/nginx.pid;
include /etc/nginx/modules-enabled/*.conf;

events {
    worker_connections  1024;
}


http {

        sendfile on;
        tcp_nopush on;
        types_hash_max_size 2048;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;


        ssl_protocols TLSv1 TLSv1.1 TLSv1.2 TLSv1.3; # Dropping SSLv3, ref: POODLE
        ssl_prefer_server_ciphers on;


        access_log /var/log/nginx/access.log;
        error_log /var/log/nginx/error.log;


        gzip on;


        include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}
2

Heading to bed so I apologize in advance for slow replies. I restored the VM to the original config in the meantime while I wait. Thank you!!

3

So I ended up staying up and figured out it was actually my ubiquity gateway that was causing the problem. somehow its configs got messed up and even though everything looked right and the logs showed the http traffic going where it was supposed to I rebooted it and now everything is fine. problem solved!

1 Like