Nginx Renewal Failing

Renewal is failing for two subdomains.

I ran this command: certbot renew --nginx

It produced this output:

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Identifier: immich.hickshome.dev
  Type:   unauthorized
  Detail: ***.***.***.***: Invalid response from https://immich.hickshome.dev/.well-known/acme-challenge/pKnMV-pXymwkyKy6sAVh-iqqlYwaoFCu8HYwq67QSuQ: 404

  Identifier: proxmox.hickshome.dev
  Type:   unauthorized
  Detail: ***.***.***.***: Invalid response from https://proxmox.hickshome.dev/.well-known/acme-challenge/G71VYIG5kN_LJ3Y93jJiEa3BqZQ3U01CumajLrTCa4w: 404

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

My web server is (include version): nginx 1.26.3

Here is my nginx conf:

server {
    # Redirect HTTP traffic to HTTPS
    listen 80;
    listen [::]:80 ipv6only=off;
    server_name _;
    return 301 https://$host$request_uri;
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name proxmox.hickshome.dev;
    ssl_certificate /etc/letsencrypt/live/proxmox.hickshome.dev/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/proxmox.hickshome.dev/privkey.pem;
    proxy_redirect off;
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass https://localhost:8006;
        proxy_buffering off;
        client_max_body_size 0;
        proxy_connect_timeout  3600s;
        proxy_read_timeout  3600s;
        proxy_send_timeout  3600s;
        send_timeout  3600s;
    }
    location ~ /.well-known/acme-challenge/ {
        allow all;
    }
}

server {
    listen 443 ssl;
    listen [::]:443 ssl;
    server_name immich.hickshome.dev;
    ssl_certificate /etc/letsencrypt/live/proxmox.hickshome.dev/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/proxmox.hickshome.dev/privkey.pem;
    proxy_redirect off;
    location / {
        proxy_http_version 1.1;
        proxy_set_header Upgrade $http_upgrade;
        proxy_set_header Connection "upgrade";
        proxy_pass http://10.0.0.**:2283;
    }
    location ~ /.well-known/acme-challenge/ {
        allow all;
    }

}

The operating system my web server runs on is (include version): proxmox VE 9.0.6

My hosting provider, if applicable, is: GoDaddy

The version of my client is: certbot 5.6.0

The 404 in the error message while using the --nginx method is unusual. It can mean the nginx system that is responding is not the same as the one you ran Certbot along-side.

You said you have version 1.26.3. But, v1.22.1 is responding. Was that just a typo or could you have a mis-routing of HTTP requests in your system?

curl -i immich.hickshome.dev 

HTTP/1.1 301 Moved Permanently
Server: nginx/1.22.1

Interesting. I'm not sure why it's showing 1.22.1. I ran nginx -v and it returned 1.26.3 directly on my proxmox host. I wonder if there is something in my router serving it.

I could never get certbot to work with NGINX during certificate creation, so i ran it as standalone certonly to get me by.

This does seem odd, but everything works fine with my reverse proxy for both the sites that are coming off of this server

You could setup access_log files with unique names in your 1.26 server. Then, retry that curl that says v1.22 and see if that request shows up in the log you expect. This may help track down the reason. Perhaps your system just overrides the default Server response header. But, that is unlikely. There probably is some other nginx system involved somehow.

Is Proxmox a systemd system?

You could also compare the pids of the running tasks with the one from systemd. Such as

sudo ps -eF | grep nginx
sudo systemctl status --no-pager -l nginx

Those work on Debian 13, adjust for yours if needed.

Standalone needs exclusive use of port 80. What commands did you use to stop nginx so they did not conflict?

Looks like the main PID does match from both commands.

I have also tried

systemctl stop nginx
certbot renew

i have also tried with --standalone on certbot renew, both with the same errors.
when doing this, i get the following output.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/proxmox.hickshome.dev.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for proxmox.hickshome.dev and immich.hickshome.dev

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
  Identifier: immich.hickshome.dev
  Type:   connection
  Detail: ***.***.***.***: Fetching https://immich.hickshome.dev/.well-known/acme-challenge/UNDPEDlePF73H2Mc3gfp0VXZde_Ui4UAuvnhgb5Lvrw: Connection refused

  Identifier: proxmox.hickshome.dev
  Type:   connection
  Detail: ***.***.***.***: Fetching https://proxmox.hickshome.dev/.well-known/acme-challenge/sWf8nElfBJF78sTeAGbXC1OxFzpHryhwLtUsxMjAMqg: Connection refused

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

I have port forwards setup for 80 and 443 in Omada Software Controller to point directly at the proxmox server.

I should also mention that once I do these, port 80 becomes stuck and nginx will not start back up.

Jun 02 10:25:20 HICKS-SERVER01 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Jun 02 10:25:20 HICKS-SERVER01 nginx[407701]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:25:21 HICKS-SERVER01 nginx[407701]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:25:21 HICKS-SERVER01 nginx[407701]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:25:22 HICKS-SERVER01 nginx[407701]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:25:22 HICKS-SERVER01 nginx[407701]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:25:23 HICKS-SERVER01 nginx[407701]: nginx: [emerg] still could not bind()
Jun 02 10:25:23 HICKS-SERVER01 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 02 10:25:23 HICKS-SERVER01 systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 02 10:25:23 HICKS-SERVER01 systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.

Stopping nginx before renewing can cause system stability problems if your Certbot renewal config file uses the --nginx option. You can get multiple nginx systems running - one under systemd and one not.

That is exactly what is causing:

You now either need to carefully kill the non-systemd nginx pids for nginx (harder) or reboot your server (easier).

Standalone with connection-refused means Let's Encrypt could not reach you. Standalone is much harder to debug than --nginx.

You should not be mixing these kinds of requests without careful management of your environ.

After you reboot, let's start by showing the contents of the contents of your renewal file in here:

/etc/letsencrypt/renewal/

If there is more than one please show output of this

sudo certbot certificates

Well something interesting happened. After the reboot, nginx still wouldn't startup.

root@HICKS-SERVER01:/etc/letsencrypt/renewal# sudo netstat -tulnp | grep :80
tcp6       0      0 :::8006                 :::*                    LISTEN      1763/pveproxy 

Nothing is using port 80.

Renewal file:

version = 5.4.0
archive_dir = /etc/letsencrypt/archive/proxmox.hickshome.dev
cert = /etc/letsencrypt/live/proxmox.hickshome.dev/cert.pem
privkey = /etc/letsencrypt/live/proxmox.hickshome.dev/privkey.pem
chain = /etc/letsencrypt/live/proxmox.hickshome.dev/chain.pem
fullchain = /etc/letsencrypt/live/proxmox.hickshome.dev/fullchain.pem
[renewalparams]
account = 8d776bddb0ea0c7990c2b1e70be01577
authenticator = standalone
server = https://acme-v02.api.letsencrypt.org/directory
key_type = ecdsa
[acme_renewal_info]
ari_retry_after = 2026-06-02T15:50:51

There is only one file

Why not? What does this say: sudo systemctl status -l --no-pager nginx

Do you want nginx to be running? Or was that just something you tried instead of standalone. We need to decide which method you'll use before proceeding

Yes, nginx should be running, as I use it to serve my reverse proxies.
Ouput:

root@HICKS-SERVER01:~# systemctl status -l --no-pager nginx
× nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/usr/lib/systemd/system/nginx.service; enabled; preset: enabled)
     Active: failed (Result: exit-code) since Tue 2026-06-02 10:55:20 EDT; 1h 5min ago
 Invocation: df0a7b8e0eab4739bae4d3de8e8b1521
       Docs: man:nginx(8)
    Process: 1449 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)
   Mem peak: 5.3M
        CPU: 101ms

Jun 02 10:55:17 HICKS-SERVER01 systemd[1]: Starting nginx.service - A high performance web server and a reverse proxy server...
Jun 02 10:55:17 HICKS-SERVER01 nginx[1449]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:55:18 HICKS-SERVER01 nginx[1449]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:55:18 HICKS-SERVER01 nginx[1449]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:55:19 HICKS-SERVER01 nginx[1449]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:55:19 HICKS-SERVER01 nginx[1449]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jun 02 10:55:20 HICKS-SERVER01 nginx[1449]: nginx: [emerg] still could not bind()
Jun 02 10:55:20 HICKS-SERVER01 systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jun 02 10:55:20 HICKS-SERVER01 systemd[1]: nginx.service: Failed with result 'exit-code'.
Jun 02 10:55:20 HICKS-SERVER01 systemd[1]: Failed to start nginx.service - A high performance web server and a reverse proxy server.

I'm starting to think at this point it will be easier to just start over new.

Does that still show no one using port 80? Because nginx says something already has it.

Yep. It shows proxmox VE using 8006 and nothing else. I'm now fighting just getting nginx to start with no configuration. Something has gotten messed up.

If I put

    listen 80;
    listen [::]:80 ipv6only=off;

in my nginx config I get the same error: bind() to [::]:80 failed.

Try using this instead:

        listen      80;
        listen      [::]:80;

Hello All!

Little update, i just created an LXC and setup Nginx Proxy Manager and setup everything up with a dashboard. Lets Encrypt installed certs automatically without issues as well, so I'm going to mark this as closed. It's always fun to try to do things manually and learn more, but at the end of the day this is a hobby on a home server rack in my house and I don't want it to become a chore.