Certbot control Nginx which is being controlled via supervisord

Help
#1

Hi All,

I am using Ubuntu 20.04.
I use supervisord to control nginx. This means that nginx is not controlled via systemd.
To stop nginx I use supervisorctl stop nginx.

Unfortunately Certbot does not know about this, and seems to try and stop nginx using the usual service. It then fails to bind because nginx is already running. I deal with this by manually killing the service certbot started.

However this is a problem when it performs an SSL cert renewal. It can't bind after the restart, and my website went down for 18 hrs before I got a compliant. I rebuild the website every year so if the cert was valid for a year it would be fine, however certbot renews the certs every 3 months.

How does one tell certbot how to stop and start nginx via supervisorctl?

#2

Certbot's nginx plugin calls the nginx bin directly to perform reloads (or to start nginx in the first place):

nginx -c /path/to/nginx.conf -s reload

and

nginx -c /path/to/nginx.conf

This happens to work with systemd supervision, for the most part, thanks to the PID file being coordinated between the nginx.conf file and systemd units that ship with most distro packages for nginx.

You could perhaps do the same thing with supervisord.

Otherwise, you will probably need to use certbot certonly in combination with --pre-hook, --post-hook, --deploy-hook, and to avoid using --nginx. The plugin is written to rely on the above commands working, and if they don't, it's probably not going to be something that's going to change.

3 Likes
#3

Why would you need to stop a working web server (nginx) only to start a temporary web server (certbot --standalone) to serve a simple challenge request?

1 Like
#4

@rg305, the --nginx method also uses nginx restart commands after making temporary changes to satisfy challenges, and, as an installer, once the certificates are obtained. So Certbot has to be able to get nginx to reload its configuration when using --nginx.

1 Like
#5

Is there a way to leave nginx running, and have certbot perform the renewal with no interruption? I would think that would use the ip address, and hence maybe not succeed in the renewal.

#6

Sounds like I should just take nginx out of supervisorctl, and just use the normal setup.

#7

That is the default.

2 Likes
#8

I wouldn't change that part at all.

2 Likes
#9

I think the straightforward way to do this, if you want to keep using supervisord, would be something like:

certbot certonly -d example.com \
--webroot -w /path/to/webroot \
--deploy-hook "supervisorctl reload nginx"

but you would need to make sure that your nginx virtual host is serving up content from the webroot path you provide to Certbot.

4 Likes
#10

Is it documented the process that certbot does, piecing together the answers I am guessing this is what happens:

  1. Web server is running as normal
  2. Certbot alters nginx config somehow (does it edit the config files?)
  3. Certbot reloads nginx config
  4. Certbot performs renewal
  5. Certbot undoes config changes.
  6. Certbot reloads nginx config

Does that sound right to you?

#11

That is the "long way" involving the nginx plugin.
I prefer the "short way" involving --webroot and simply point certbot to the webroot for each domain.
[which doesn't alter any web server configs - which, in turn, doesn't require a reload and then undo]

As shown by @_az

2 Likes
#12

Thanks for that explaination of what @_az's solution does. That sounds like a great solution!

1 Like
#13

Is it documented the process that certbot does, piecing together the answers I am guessing this is what happens:

  1. Web server is running as normal
  2. Certbot alters nginx config somehow (does it edit the config files?)
  3. Certbot reloads nginx config
  4. Certbot performs renewal
  5. Certbot undoes config changes.
  6. Certbot reloads nginx config

Does that sound right to you?

Yes, that's right. (The "renewal" step includes communicating with the Let's Encrypt API between steps 1 and 2 as well, in order to find out exactly which changes to make.)

Some portions of this are described in

https://eff-certbot.readthedocs.io/en/stable/using.html

but not very precisely and not all in one place.

2 Likes
#14

If you only want to handle renewals - and not have Certbot manage the nginx configuration - this is what I do:

  • In nginx, I proxy the /.well-known/acme-challenge directory to a higher port (e.g. 8086)
  • run Certbot in standalone mode with --http-01-port={the higher port}
  • run Certbot with a --deploy-hook to reload nginx

This is essentially what @_az recommends, except I proxy traffic to Certbot's internal webserver instead of using the webroot. For my needs, I have found this to be much easier than dealing with webroot.

adding: As someone who uses supervisord, personally, I would NOT use supervisord to run nginx. Supervisord is a bit of a pain to manage and has some odd nuances. One of the more annoying things to work with is how it acts on startup and restarts, which can keep your services offline longer than needed. In my experience, I think it's best to let systemctl manage nginx and then proxy traffic to applications run via supervisord.

2 Likes
#15

From my research, one still requires to restart nginx afterwards so it uses the new certs (not the ones in memory). So the benefit is one restart as opposed to two?

#16

Actually just a reload which is less disruptive. You could even reload daily which would pick up the latest certs if / whenever they were created.

3 Likes
#17

Restart and reload are different operations.

Only reloads are needed, to load a new config.

1 Like
#18

Nginx supports a "graceful" restart in at least two ways (there may be more):

  • kill -HUP {PID}
  • nginx reload

The restart command on most systemctl/systemd scripts implements the restart as a graceful restart, not a "stop start".

A graceful restart has the main process re-read the configuration files, stops routing traffic to the child processes, and instructs those processes to exit and respawn after serving their current requests.

Apache2 supports this via the graceful command.

The restart issue I had with running processes under supervisord is that on a system boot (or restart), Systemctl will start Supervisord which then starts Nginx. You end up waiting for two services to boot up, instead of one, before nginx is started.

Beyond that, I find the way supervisord manages processes and logs errors to be a lot more delicate and specialized than systemctl. It is a great system for running first-party applications, but core OS services (i.e. those on privileged ports) and anything to do with routing traffic (including proxying in nginx) is easier managed on systemctl. Personally, I really dislike how it handles configuration changes and it's abilities to start/stop/enable/disable processes are inferior to systemctl -- which can become a big issue if you are dealing with scheduled downtime, surprise reboots, or autoscaling.

Supervisord is also fairly specialized knowledge, while systemctl is a core competency in linux, so you are complicating your devops and system management.

1 Like
#19

Yup. They are very seldom used, tho.

https://nginx.org/en/docs/control.html

1 Like
#20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.