/home/jerdvo/.rbenv/plugins/ruby-build/bin:/home/jerdvo/.rbenv/shims:/home/jerdvo/.rbenv/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin
$ which -a nginx
/usr/sbin/nginx
/sbin/nginx
Thank you, the situation is clarified.
Until we know what certbot touches improperly and that gets fixed, certbot --webroot is the only useful possibility in production mode.
rebooting could be acceptable in development - which also provides the syntax for .conf files.
We know that certbot might start nginx directly which causes problems in systemd distros like yours (and mine :)). And, _az promised to look at that.
But, this "feels" to me like an inherent issue related to certbot modifying your nginx.conf files while nginx is running. Does passenger or any other monitoring system auto-restart nginx when it detects a change to nginx.conf? Something like that would explain all the facts - especially the troubling unknown pid in the systemd segv error message.
If such an auto-restart is happening certbot --webroot (or a different acme client) is your only option as certbot --nginx will always update the nginx.conf files - for new issuance and renew.
I am not aware of such passenger behaviour; its role is related to the associated application (thus as sub-component of the nginx.conf file). However I am light years away from being an expert.
With the attached file though, the case should be replicable for capable hands.
The nginx segv fault is not easily reproduced. We would be seeing that a vast number of times per day if it was common.
Maybe look in /var/log/dmesg for clues? If you don't see something upload that and maybe we will see something helpful. (look for nginx and/or segfault)
To round that off (so even I can put this in a lab):
Which version of Ubuntu?
Were both nginx and certbot installed from apt?
OR was certbot installed via snap?
[OR other... like either or both were compiled from source]
Ubuntu 20.04.
nginx installed via apt
certbot installed via snap (freshly; I even ensured that sudo apt-get remove certbot ran beforehand - it drew a blank)
Do you the have Perl module enabled in nginx? i.e. Is /etc/nginx/modules-enabled/50-mod-http-perl.conf present?
nginx's master process segfaulting would explain some things. We've had numerous other reports of that module causing segfaults on reload, on Ubuntu servers.
Try disable it, if it's there.
If that doesn't help, it would be handy if you could gdb attach to the nginx master process before it crashes, and provide a backtrace of the segfault.
Yes, left the file in, but disabling its only line: # load_module modules/ngx_http_perl_module.so;
# added new server name
$ sudo service nginx restart
~$
$ ls -l /run/nginx.pid
-rw-r--r-- 1 root root 5 Nov 2 06:11 /run/nginx.pid
sudo ps -eF | grep -E "nginx|PID"
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
root 1137 1 0 26306 4668 0 06:11 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 1138 1137 0 26448 8828 0 06:11 ? 00:00:00 nginx: worker process
jerdvo 1159 1041 0 2039 2520 0 06:13 pts/0 00:00:00 grep --color=auto -E nginx|PID
$ sudo certbot --nginx -d testthree.fidely.club
# [...] Successfully received certificate. [...] Deploying certificate
$ ls -l /run/nginx.pid
-rw-r--r-- 1 root root 5 Nov 2 06:17 /run/nginx.pid
$ sudo ps -eF | grep -E "nginx|PID"
UID PID PPID C SZ RSS PSR STIME TTY TIME CMD
root 1419 1 0 26506 14428 0 06:17 ? 00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 1746 1419 0 26566 9316 0 06:24 ? 00:00:00 nginx: worker process
jerdvo 1825 1041 0 2039 2516 0 06:25 pts/0 00:00:00 grep --color=auto -E nginx|PID
$ sudo service nginx restart
~$
huzzah
So the letsencrypt.log does indicate a backup, but of the disabled load_module command
2021-11-02 06:24:37,789:DEBUG:certbot.reverter:Creating backup of /etc/nginx/modules-enabled/50-mod-http-cache-purge.conf
2021-11-02 06:24:37,789:DEBUG:certbot.reverter:Creating backup of /etc/nginx/modules-enabled/50-mod-http-perl.conf
2021-11-02 06:24:37,789:DEBUG:certbot.reverter:Creating backup of /etc/nginx/modules-enabled/50-mod-http-xslt-filter.conf
Further changes to conf files pass nginx tests and the service restarts.