I am using kernel 3.10 on CentOS 7. I have successfully installed CertBot 1.10.1 using alternative installation instructions as certbot-auto . I have manually added certbot-auto to autorun via systemd as:
/etc/systemd/system/certbot-renewal.service :
[Unit]
Description=Certbot Renewal
[Service]
ExecStart=/usr/local/bin/certbot-auto renew --pre-hook "service nginx stop" --post-hook "service nginx start" --quiet --agree-tos
/etc/systemd/system/certbot-renewal.timer :
[Unit]
Description=Timer for Certbot Renewal
[Timer]
OnBootSec=1h
OnUnitActiveSec=1d
[Install]
WantedBy=multi-user.target
Now, certbot-auto successfully refreshes SSL certificates when it is needed. However, the problem is that certbot-auto fails to start nginx .
For example, if certbot-auto updates certificates - my web-site is down. If I connect via SSH, I see this:
[root@somedomain ~]# sudo systemctl status nginx
● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
Active: inactive (dead) since Wed 2021-04-14 16:40:56 UTC; 3min 14s ago
Docs: man:systemd-sysv-generator(8)
Process: 5745 ExecStop=/etc/rc.d/init.d/nginx stop (code=exited, status=0/SUCCESS)
Process: 5737 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=0/SUCCESS)
Main PID: 5708 (code=exited, status=0/SUCCESS)
Apr 14 16:40:56 somedomain.com systemd[1]: Starting SYSV: Nginx is an HTTP(S)....
Apr 14 16:40:56 somedomain.com systemd[1]: Started SYSV: Nginx is an HTTP(S) ....
Hint: Some lines were ellipsized, use -l to show in full.
[root@somedomain ~]# sudo systemctl start nginx
[root@somedomain ~]# sudo systemctl status nginx
● nginx.service - SYSV: Nginx is an HTTP(S) server, HTTP(S) reverse proxy and IMAP/POP3 proxy server
Loaded: loaded (/etc/rc.d/init.d/nginx; bad; vendor preset: disabled)
Active: active (running) since Wed 2021-04-14 16:44:45 UTC; 3s ago
Docs: man:systemd-sysv-generator(8)
Process: 5745 ExecStop=/etc/rc.d/init.d/nginx stop (code=exited, status=0/SUCCESS)
Process: 5809 ExecStart=/etc/rc.d/init.d/nginx start (code=exited, status=0/SUCCESS)
Main PID: 5822 (nginx)
CGroup: /system.slice/nginx.service
├─5822 nginx: master process /usr/sbin/nginx -c /etc/nginx/nginx.c...
├─5823 nginx: worker process
├─5824 nginx: worker process
├─5826 nginx: cache manager process
└─5827 nginx: cache loader process
Apr 14 16:44:45 somedomain.com systemd[1]: Starting SYSV: Nginx is an HTTP(S)....
Apr 14 16:44:45 somedomain.com nginx[5809]: Starting nginx: [ OK ]
Apr 14 16:44:45 somedomain.com systemd[1]: Started SYSV: Nginx is an HTTP(S) ....
Hint: Some lines were ellipsized, use -l to show in full.
Looking at certbot logs does not show anything suspicious:
...
2021-04-14 16:40:46,329:INFO:certbot.compat.misc:Running pre-hook command: service nginx stop
2021-04-14 16:40:46,488:INFO:certbot.compat.misc:Output from pre-hook command service:
Stopping nginx (via systemctl): [ OK ]
2021-04-14 16:40:46,492:DEBUG:certbot.display.util:Notifying user: Renewing an existing certificate for somedomain.com and 4 more domains
...
2021-04-14 16:40:48,149:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/conf.d/somefile.conf:
...
2021-04-14 16:40:48,221:DEBUG:certbot_nginx._internal.configurator:nginx reload failed:
nginx: [error] open() "/run/nginx.pid" failed (2: No such file or directory)
...
2021-04-14 16:40:52,071:DEBUG:acme.client:Storing nonce: ...
2021-04-14 16:40:52,072:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-04-14 16:40:52,072:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-04-14 16:40:55,267:DEBUG:certbot._internal.storage:Writing new config /etc/letsencrypt/renewal/somedomain.com.conf.new.
2021-04-14 16:40:56,316:DEBUG:certbot.display.util:Notifying user: new certificate deployed with reload of nginx server; fullchain is
/etc/letsencrypt/live/somedomain.com/fullchain.pem
2021-04-14 16:40:56,322:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2021-04-14 16:40:56,324:DEBUG:certbot._internal.plugins.selection:Selecting plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7f75ee7ab250>
Prep: True
2021-04-14 16:40:56,325:DEBUG:certbot.display.util:Notifying user:
Congratulations, all renewals succeeded. The following certs have been renewed:
/etc/letsencrypt/live/somedomain.com/fullchain.pem (success)
2021-04-14 16:40:56,326:DEBUG:certbot._internal.renewal:no renewal failures
2021-04-14 16:40:56,326:INFO:certbot.compat.misc:Running post-hook command: service nginx start
2021-04-14 16:40:56,455:INFO:certbot.compat.misc:Output from post-hook command service:
Starting nginx (via systemctl): [ OK ]
As you can see - logs indicate that certbot was able to run nginx.
Looking at nginx logs:
... unrelated old entries
2021/04/14 16:40:46 [alert] 5188#0: *1395650 open socket #18 left in connection 10
2021/04/14 16:40:46 [alert] 5188#0: *1395649 open socket #13 left in connection 17
2021/04/14 16:40:46 [alert] 5188#0: aborting
2021/04/14 16:40:48 [notice] 5706#0: signal process started
2021/04/14 16:40:48 [error] 5706#0: open() "/run/nginx.pid" failed (2: No such file or directory)
2021/04/14 16:40:52 [notice] 5715#0: signal process started
2021/04/14 16:40:55 [notice] 5720#0: signal process started
Nothing suspicious to me as well. nginx seems to be started.
Any idea what could be wrong? Or what could I check?
Good catch. I removed it from the renewal configuration file (by deleting the file itself), but I missed that it was elsewhere. No wonder things are working strangely.
