I am using Ubuntu 20.04.
I use supervisord to control nginx. This means that nginx is not controlled via systemd.
To stop nginx I use supervisorctl stop nginx.
Unfortunately Certbot does not know about this, and seems to try and stop nginx using the usual service. It then fails to bind because nginx is already running. I deal with this by manually killing the service certbot started.
However this is a problem when it performs an SSL cert renewal. It can't bind after the restart, and my website went down for 18 hrs before I got a compliant. I rebuild the website every year so if the cert was valid for a year it would be fine, however certbot renews the certs every 3 months.
How does one tell certbot how to stop and start nginx via supervisorctl?
Certbot's nginx plugin calls the nginx bin directly to perform reloads (or to start nginx in the first place):
nginx -c /path/to/nginx.conf -s reload
and
nginx -c /path/to/nginx.conf
This happens to work with systemd supervision, for the most part, thanks to the PID file being coordinated between the nginx.conf file and systemd units that ship with most distro packages for nginx.
You could perhaps do the same thing with supervisord.
Otherwise, you will probably need to use certbot certonly in combination with --pre-hook, --post-hook, --deploy-hook, and to avoid using --nginx. The plugin is written to rely on the above commands working, and if they don't, it's probably not going to be something that's going to change.
Why would you need to stop a working web server (nginx) only to start a temporary web server (certbot --standalone) to serve a simple challenge request?
@rg305, the --nginx method also uses nginx restart commands after making temporary changes to satisfy challenges, and, as an installer, once the certificates are obtained. So Certbot has to be able to get nginx to reload its configuration when using --nginx.
Is there a way to leave nginx running, and have certbot perform the renewal with no interruption? I would think that would use the ip address, and hence maybe not succeed in the renewal.
That is the "long way" involving the nginx plugin.
I prefer the "short way" involving --webroot and simply pointcertbot to the webroot for each domain.
[which doesn't alter any web server configs - which, in turn, doesn't require a reload and then undo]
Is it documented the process that certbot does, piecing together the answers I am guessing this is what happens:
Web server is running as normal
Certbot alters nginx config somehow (does it edit the config files?)
Certbot reloads nginx config
Certbot performs renewal
Certbot undoes config changes.
Certbot reloads nginx config
Does that sound right to you?
Yes, that's right. (The "renewal" step includes communicating with the Let's Encrypt API between steps 1 and 2 as well, in order to find out exactly which changes to make.)
If you only want to handle renewals - and not have Certbot manage the nginx configuration - this is what I do:
In nginx, I proxy the /.well-known/acme-challenge directory to a higher port (e.g. 8086)
run Certbot in standalone mode with --http-01-port={the higher port}
run Certbot with a --deploy-hook to reload nginx
This is essentially what @_az recommends, except I proxy traffic to Certbot's internal webserver instead of using the webroot. For my needs, I have found this to be much easier than dealing with webroot.
adding: As someone who uses supervisord, personally, I would NOT use supervisord to run nginx. Supervisord is a bit of a pain to manage and has some odd nuances. One of the more annoying things to work with is how it acts on startup and restarts, which can keep your services offline longer than needed. In my experience, I think it's best to let systemctl manage nginx and then proxy traffic to applications run via supervisord.
From my research, one still requires to restart nginx afterwards so it uses the new certs (not the ones in memory). So the benefit is one restart as opposed to two?
Nginx supports a "graceful" restart in at least two ways (there may be more):
kill -HUP {PID}
nginx reload
The restart command on most systemctl/systemd scripts implements the restart as a graceful restart, not a "stop start".
A graceful restart has the main process re-read the configuration files, stops routing traffic to the child processes, and instructs those processes to exit and respawn after serving their current requests.
Apache2 supports this via the graceful command.
The restart issue I had with running processes under supervisord is that on a system boot (or restart), Systemctl will start Supervisord which then starts Nginx. You end up waiting for two services to boot up, instead of one, before nginx is started.
Beyond that, I find the way supervisord manages processes and logs errors to be a lot more delicate and specialized than systemctl. It is a great system for running first-party applications, but core OS services (i.e. those on privileged ports) and anything to do with routing traffic (including proxying in nginx) is easier managed on systemctl. Personally, I really dislike how it handles configuration changes and it's abilities to start/stop/enable/disable processes are inferior to systemctl -- which can become a big issue if you are dealing with scheduled downtime, surprise reboots, or autoscaling.
Supervisord is also fairly specialized knowledge, while systemctl is a core competency in linux, so you are complicating your devops and system management.