Error while running apachectl graceful. Job for httpd.service invalid

Dear friends,

Please help me to resolve it. I have tried many ways but with no success :frowning:

I ran this command: sudo certbot --apache -d my domain -d www.my domain

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn’t close to expiry.
(ref: /etc/letsencrypt/renewal/av.gds.vn-0001.conf)

What would you like to do?


1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (may be subject to CA rate limits)


Select the appropriate number [1-2] then [enter] (press ‘c’ to cancel): 1
Keeping the existing certificate
Deploying Certificate to VirtualHost /etc/httpd/conf.d/nextcloud-le-ssl.conf
Error while running apachectl graceful.

Job for httpd.service invalid.

Unable to restart apache using [‘apachectl’, ‘graceful’]
Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

Rolling back to previous server configuration…
Error while running apachectl graceful.

Job for httpd.service invalid.

Unable to restart apache using [‘apachectl’, ‘graceful’]
Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

Encountered exception during recovery:
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/certbot/_internal/error_handler.py”, line 125, in _call_registered
self.funcs-1
File “/usr/lib/python2.7/site-packages/certbot/_internal/client.py”, line 630, in _rollback_and_restart
self.installer.restart()
File “/usr/lib/python2.7/site-packages/certbot_apache/_internal/configurator.py”, line 2410, in restart
self._reload()
File “/usr/lib/python2.7/site-packages/certbot_apache/_internal/configurator.py”, line 2437, in _reload
raise errors.MisconfigurationError(error)
MisconfigurationError: Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

IMPORTANT NOTES:

  • An error occurred and we failed to restore your config and restart
    your server. Please post to
    https://community.letsencrypt.org/c/help with details about your
    configuration and this error you received.
  • Congratulations! Your certificate and chain have been saved at:
    /etc/letsencrypt/live/av.gds.vn-0001/fullchain.pem
    Your key file has been saved at:
    /etc/letsencrypt/live/av.gds.vn-0001/privkey.pem
    Your cert will expire on 2020-12-15. To obtain a new or tweaked
    version of this certificate in the future, simply run certbot again
    with the “certonly” option. To non-interactively renew all of
    your certificates, run “certbot renew”

What’s the output of:

sudo apachectl configtest

Hi az,

[root@av ~]# sudo apachectl configtest
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:358
Syntax OK

That’s strange. Seems like Apache is in a half-crashed state, which results in Certbot being unable to reload it?

systemctl stop httpd
killall -9 httpd
systemctl start httpd
systemctl status httpd

Hi _az,

Thanks for your help. As your comment, Apache has problem but i don’t know why.

From the beginning, Apache still working normally.

● httpd.service - The Apache HTTP Server
** Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)**
** Active: active (running) since Wed 2020-09-16 18:52:06 +07; 4s ago**
** Docs: man:httpd(8)**
** man:apachectl(8)**
** Main PID: 15109 (httpd)**
** Status: “Processing requests…”**
** CGroup: /system.slice/httpd.service**
** ├─15109 /usr/sbin/httpd -DFOREGROUND**
** ├─15110 /usr/sbin/httpd -DFOREGROUND**
** ├─15111 /usr/sbin/httpd -DFOREGROUND**
** ├─15112 /usr/sbin/httpd -DFOREGROUND**
** ├─15113 /usr/sbin/httpd -DFOREGROUND**
** └─15114 /usr/sbin/httpd -DFOREGROUND**

Sep 16 18:52:06 av.gds.com.vn systemd[1]: Starting The Apache HTTP Server…
Sep 16 18:52:06 av.gds.com.vn httpd[15109]: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:357
Sep 16 18:52:06 av.gds.com.vn systemd[1]: Started The Apache HTTP Server.

But after i run this command, Apache will give an error:

[root@av ~]# sudo certbot --apache -d av.gds.vn -d www.av.gds.vn

● httpd.service - The Apache HTTP Server
** Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)**
** Active: failed (Result: exit-code) since Wed 2020-09-16 18:52:37 +07; 11s ago**
** Docs: man:httpd(8)**
** man:apachectl(8)**
** Process: 15234 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)**
** Process: 15171 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)**
** Process: 15232 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)**
** Main PID: 15232 (code=exited, status=1/FAILURE)**

Sep 16 18:52:37 av.gds.com.vn httpd[15232]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:443
Sep 16 18:52:37 av.gds.com.vn httpd[15232]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443
Sep 16 18:52:37 av.gds.com.vn httpd[15232]: no listening sockets available, shutting down
Sep 16 18:52:37 av.gds.com.vn httpd[15232]: AH00015: Unable to open logs
Sep 16 18:52:37 av.gds.com.vn systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Sep 16 18:52:37 av.gds.com.vn kill[15234]: kill: cannot find process ""
Sep 16 18:52:37 av.gds.com.vn systemd[1]: httpd.service: control process exited, code=exited status=1
Sep 16 18:52:37 av.gds.com.vn systemd[1]: Failed to start The Apache HTTP Server.
Sep 16 18:52:37 av.gds.com.vn systemd[1]: Unit httpd.service entered failed state.
Sep 16 18:52:37 av.gds.com.vn systemd[1]: httpd.service failed.

The reason is that the file httpd.conf has been changed.

when I delete “Listen 443”, Apache is working properly again.

Sorry for uploading the code, I intend to upload the image for easy to understand but new user only allowed to upload 1 picture.

Please help me to resolve it :frowning:

I think you might be hitting this Certbot bug: https://github.com/certbot/certbot/issues/5517

Listen 443 should appear exactly once in your configuration.

When it appears more than once, you end up with that Address already in use error and Apache can’t start.

Problem is, Certbot is apparently adding it unnecessarily, and that breaks your Apache configuration.

What I would try is the following:

First, remove the <IfModule mod_ssl.c> and </IfModule> lines that surrounds the Listen line. That may be breaking Certbot’s detection of whether the Listen 443 is already done.

Next, check whether anywhere else in your configuration contains that listen:

grep -Ri "Listen 443" /etc/httpd

Hi _az,

Thanks for your recommendations.

I already removed the <IfModule mod_ssl.c> and </IfModule> lines but Apache still adds "Listen 443" to httpd.conf file when i renew Cerbot.

[root@av ~]# grep -Ri “Listen 443” /etc/httpd
/etc/httpd/conf/httpd.conf.rtFvS5:Listen 443
/etc/httpd/conf/httpd.conf.rtFvS5:#Listen 443
/etc/httpd/conf/httpd.conf:#Listen 443

Hi _az,

Although I removed them before but actually Apache adds <IfModule mod_ssl.c> and </IfModule> lines + “Listen 443” to httpd.conf file when i run command.

sudo certbot --apache -d av.gds.vn -d www.av.gds.vn

Oh, you mean that Apache only works when there are no Listen 443 lines?

I’m not sure how that can be. Is it actually the same PIDs bound to 80 and 443?

ss -tlpn | grep -E ":(80|443)"

Hi _az,

Yes, Apache only works when there are no Listen 443 lines.

ss -tlpn | grep -E ":(80|443)"

No results are displayed.

[root@av ~]#
[root@av ~]# ss -tlpn | grep -E “: (80 | 443)”
[root@av ~]#

Could you please run the command exactly as I wrote it? Without the extra spaces?

Copy-pasting it might help.

Hi _az,

Here is the result:

[root@av ~]# ss -tlpn | grep -E “:(80|443)”
LISTEN 0 128 :::80 :::* users:((“httpd”,pid=32340,fd=4),(“httpd”,pid=32338,fd=4),(“httpd”,pid=32337,fd=4),(“httpd”,pid=32336,fd=4),(“httpd”,pid=32335,fd=4),(“httpd”,pid=31682,fd=4))
LISTEN 0 128 :::443 :::* users:((“httpd”,pid=5297,fd=4),(“httpd”,pid=4677,fd=4),(“httpd”,pid=4676,fd=4),(“httpd”,pid=4675,fd=4),(“httpd”,pid=1547,fd=4))

Thanks. It’s what I suspected earlier - you have two different Apache servers running. That’s why I asked you to do the killall -9.

Certbot is correctly adding the Listen 443, the problem is that Apache is starting up twice, possibly from different configurations.

This is not a Bitnami server, is it?

Hi _az,

thanks for your help.

Yes, this is not a Bitnami server.

How can i resolve it? :slight_smile:

I’m not really sure. You have to identify why there’s two different Apaches running.

If the systemctl stop httpd + killall -9 httpd didn’t help, and it’s not a Bitnami server, I’m stumped.

Something that might help would be to trace where each process comes from.

yum -y install psmisc

then

pstree -scp 32340
pstree -scp 5297

and compare the output of each one.

Hi _az,

I already disabled 1 Apache server and and successfully installed Cerbot.

but Unfortunately, Apache failed to start :frowning:

Ah, so your actual Apache server is ac_httpd. I see.

I think the main thing to understand is that Certbot’s Apache plugin really only works with the Apache server that comes with CentOS. It’s not built to work with ac_httpd.

This means that you would need to manually configure the certificate for ac_httpd, rather than relying on Certbot to autoconfigure it.

You will first want to make sure that unwanted Apache server is completely disabled. You can do so by running:

systemctl stop httpd
systemctl mask httpd

Are you able to restart ac_httpd after doing that?

If so, you should be able to symlink ac_httpd's server.key and server.crt to the private key and certificate, respectively, at /etc/letsencrypt/live/$DOMAIN/{privkey.pem,fullchain.pem}.

Then restart ac_httpd once again.

That should get you up and running with SSL.

Hi _az,

I really appreciate your help with my problem :slight_smile:

I have fixed the error and successfully installed Certbot :smiley: