Nginx certfile does not exist

andrew-mitchell-721 · June 19, 2022, 10:21pm

My domain is: api.avondale.io
I ran this command: supervisorctl restart all
It produced this output: myappname: ERROR (spawn error)
My web server is (include version): Nginx / Gunicorn via supervisord
The operating system my web server runs on is (include version): Debian 10
My hosting provider, if applicable, is: Linode
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 0.31.0

Further Context
I'm running FastAPI using nginx/supervisor on a Debian server. All was working well, but when I upgraded my server, suddenly supervisor failed to start my app, throwing the following error (replaced domain with mydomainhere):

ValueError: certfile "/etc/letsencrypt/live/mydomainhere/fullchain.pem" does not exist

I have verified that the fullchain.pem file does in fact exist at the path described in the error, and has chmod 777.

Nginx Site Config (within /etc/nginx/sites-available):

upstream myproxy {
    server localhost:8000;
}

server {
    listen 80;
    server_name api.avondale.io www.api.avondale.io;
    root /var/www/html;

    location / {
        proxy_pass http://myproxy;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/api.avondale.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/api.avondale.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Supervisor Config (within /var/www/supervisor):

[program:myappname]
user=root
directory=/var/www
command=/var/www/app/bin/gunicorn -k uvicorn.workers.UvicornWorker app.main:app --bind 0.0.0.0:8000 --certfile=/etc/letsencrypt/live/mydomainhere/fullchain.pem --keyfile=/etc/letsencrypt/live/mydomainhere/privkey.pem
autostart=true
autorestart=true
stdout_logfile=/var/log/myappname/myappname.stdout.log
stdout_logfile_maxbytes=10mb
stdout_logfile_backups=5
stderr_logfile=/var/log/myappname/myappname.stderr.log
stderr_logfile_maxbytes=10mb
stderr_logfile_backups=5
environment = PYTHONUNBUFFERED=1

What am I missing?

MikeMcQ · June 19, 2022, 10:58pm

The file in /live/ folder should be a symlink to a file in /archive/ folder.

Does that target file exist and do its permissions allow it to be read?

andrew-mitchell-721 · June 19, 2022, 11:01pm

@MikeMcQ the .pem files in /archive/ all have the number 1 after them (fullchain1.pem). Will a simple mv do the trick to rename these to remove the 1's? Or will that mess with the symlink?

Edit: the files within /archive/ are 644 so I think we're good on the permissions front.

MikeMcQ · June 19, 2022, 11:05pm

what does an ls -l for the live folder show

andrew-mitchell-721 · June 19, 2022, 11:06pm

@MikeMcQ see below

total 4
lrwxrwxrwx 1 root root  39 Jun 19 21:58 cert.pem -> ../../archive/api.avondale.io/cert1.pem
lrwxrwxrwx 1 root root  40 Jun 19 21:58 chain.pem -> ../../archive/api.avondale.io/chain1.pem
lrwxrwxrwx 1 root root  44 Jun 19 21:58 fullchain.pem -> ../../archive/api.avondale.io/fullchain1.pem
lrwxrwxrwx 1 root root  42 Jun 19 21:58 privkey.pem -> ../../archive/api.avondale.io/privkey1.pem
-rw-r--r-- 1 root root 692 Jun 19 21:58 README

Looks like the links are all active still?

andrew-mitchell-721 · June 19, 2022, 11:19pm

@MikeMcQ I also updated my original post with a few changes to the niginx .conf file that I made after making this post. Did not fix this problem, but making sure you and others see the change in case it impacts something...

MikeMcQ · June 19, 2022, 11:46pm

In your supervisor config you have the path as mydomainhere

Is that actually the spelling? I had assumed it was redacted but ...

andrew-mitchell-721 · June 19, 2022, 11:51pm

Ha, yes, it's a placeholder. Anywhere you see mydomainhere it actually reads api.avondale.io

MikeMcQ · June 20, 2022, 1:11am

Well, that's all the easy stuff. Does your supervisor have access to the volume the certbot cert files are?

I mean, it's not magic. You can see the cert files. The symlinks seem fine. But your supervisor can't see them. The nginx error for missing cert files is different than that.

andrew-mitchell-721 · June 20, 2022, 1:15am

@MikeMcQ they're all on the same volume, so I can't imagine why supervisor wouldn't have access to them. This was all brought on by a server upgrade, though. I bumped up my Linode to the next tier, and when it finished booting, this error was present. Could something have gone awry during the upgrade that would have caused this?

It was an on-device upgrade... my IP and credentials remained the same, they just increased my provisioning.

The only other thing I can think of that would have caused the issue is that in a previous version of the config files, I was running supervisor using a non-root user. But, if anything, running as root would increase visibility into other directories/files, not decrease....

andrew-mitchell-721 · June 20, 2022, 1:18am

Is it worth trying to remove/reinstall some combination of nginx, supervisor, and certbot/certificates? I'm not sure what the best order/process would be for this, but that's the next thing I can think of, unless you've got another idea...

rg305 · June 20, 2022, 1:27am

Is there a reason why you are only using one server block for both 80 and 443 ?

andrew-mitchell-721 · June 20, 2022, 1:29am

@rg305 pretty sure that's the way it was configured, but I could be way off base there. Would this be the fix?

upstream myproxy {
    server localhost:8000;
}

server {
    listen 80;
    server_name api.avondale.io www.api.avondale.io;
    root /var/www/html;

    location / {
        proxy_pass http://myproxy;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

}

server {
    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/api.avondale.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/api.avondale.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

Do I need to re-list the server_name, root, and location/ lines in the second server block?

rg305 · June 20, 2022, 1:31am

The 443 would also need:
server_name xxx;
root /somewhere;
And maybe the proxy stuff too

andrew-mitchell-721 · June 20, 2022, 1:31am

But no need for the location / block?

rg305 · June 20, 2022, 1:33am

I can't say I know what you want it to do.

Most people redirect HTTP to HTTPS then handle things there.

If the final outcome is for it to be proxied, then I'd move that to 443

andrew-mitchell-721 · June 20, 2022, 1:37am

Got it, thanks @rg305! Here's what I have now in my nginx conf file:

upstream myproxy {
    server localhost:8000;
}

server {
    listen 80;
    server_name api.avondale.io www.api.avondale.io;
    root /var/www/html;

}

server {
    listen 443 ssl; # managed by Certbot
    server_name api.avondale.io www.api.avondale.io;
    root /var/www/html;

    location / {
        proxy_pass http://myproxy;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header Host $host;
        proxy_redirect off;
    }

    ssl_certificate /etc/letsencrypt/live/api.avondale.io/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/api.avondale.io/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

After running:
systemctl restart nginx
supervisorctl reread
supervisorctl restart all
...I'm still getting the same error I had before...

rg305 · June 20, 2022, 1:39am

This doesn't error?:

andrew-mitchell-721 · June 20, 2022, 1:40am

@rg305 nope, nginx is running fine. Here's the output of systemctl status nginx:

● nginx.service - A high performance web server and a reverse proxy server
   Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
   Active: active (running) since Mon 2022-06-20 01:39:28 UTC; 19s ago
     Docs: man:nginx(8)
  Process: 5304 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
  Process: 5305 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
 Main PID: 5307 (nginx)
    Tasks: 2 (limit: 2359)
   Memory: 2.8M
   CGroup: /system.slice/nginx.service
           ├─5307 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
           └─5308 nginx: worker process

Jun 20 01:39:28 localhost systemd[1]: Starting A high performance web server and a reverse proxy server...
Jun 20 01:39:28 localhost systemd[1]: nginx.service: Failed to parse PID from file /run/nginx.pid: Invalid argument
Jun 20 01:39:28 localhost systemd[1]: Started A high performance web server and a reverse proxy server.
root@localhost:/etc/supervisor#

The issue appears to be entirely with supervisord which fails to start up.

rg305 · June 20, 2022, 1:41am

That isn't wonderful though...
But I see it starts successfully.

What is the exact error? - please don't change anything within it.