Error while running apachectl graceful. Job for httpd.service invalid

DucTM · September 16, 2020, 9:41am

Dear friends,

Please help me to resolve it. I have tried many ways but with no success

I ran this command: sudo certbot --apache -d my domain -d www.my domain

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator apache, Installer apache
Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
Cert not yet due for renewal

You have an existing certificate that has exactly the same domains or certificate name you requested and isn’t close to expiry.
(ref: /etc/letsencrypt/renewal/av.gds.vn-0001.conf)

What would you like to do?

1: Attempt to reinstall this existing certificate
2: Renew & replace the cert (may be subject to CA rate limits)

Select the appropriate number [1-2] then [enter] (press ‘c’ to cancel): 1
Keeping the existing certificate
Deploying Certificate to VirtualHost /etc/httpd/conf.d/nextcloud-le-ssl.conf
Error while running apachectl graceful.

Job for httpd.service invalid.

Unable to restart apache using [‘apachectl’, ‘graceful’]
Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

Rolling back to previous server configuration…
Error while running apachectl graceful.

Job for httpd.service invalid.

Unable to restart apache using [‘apachectl’, ‘graceful’]
Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

Encountered exception during recovery:
Traceback (most recent call last):
File “/usr/lib/python2.7/site-packages/certbot/_internal/error_handler.py”, line 125, in _call_registered
self.funcs-1
File “/usr/lib/python2.7/site-packages/certbot/_internal/client.py”, line 630, in _rollback_and_restart
self.installer.restart()
File “/usr/lib/python2.7/site-packages/certbot_apache/_internal/configurator.py”, line 2410, in restart
self._reload()
File “/usr/lib/python2.7/site-packages/certbot_apache/_internal/configurator.py”, line 2437, in _reload
raise errors.MisconfigurationError(error)
MisconfigurationError: Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

Error while running apachectl restart.

Job for httpd.service failed because the control process exited with error code. See “systemctl status httpd.service” and “journalctl -xe” for details.

IMPORTANT NOTES:

An error occurred and we failed to restore your config and restart
your server. Please post to
https://community.letsencrypt.org/c/help with details about your
configuration and this error you received.
Congratulations! Your certificate and chain have been saved at:
/etc/letsencrypt/live/av.gds.vn-0001/fullchain.pem
Your key file has been saved at:
/etc/letsencrypt/live/av.gds.vn-0001/privkey.pem
Your cert will expire on 2020-12-15. To obtain a new or tweaked
version of this certificate in the future, simply run certbot again
with the “certonly” option. To non-interactively renew all of
your certificates, run “certbot renew”

_az · September 16, 2020, 9:43am

What’s the output of:

sudo apachectl configtest

DucTM · September 16, 2020, 9:44am

Hi az,

[root@av ~]# sudo apachectl configtest
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:358
Syntax OK

_az · September 16, 2020, 10:02am

That’s strange. Seems like Apache is in a half-crashed state, which results in Certbot being unable to reload it?

systemctl stop httpd
killall -9 httpd
systemctl start httpd
systemctl status httpd

DucTM · September 16, 2020, 12:21pm

Hi _az,

Thanks for your help. As your comment, Apache has problem but i don't know why.

From the beginning, Apache still working normally.

● httpd.service - The Apache HTTP Server
** Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)**
** Active: active (running) since Wed 2020-09-16 18:52:06 +07; 4s ago**
** Docs: man:httpd(8)**
** man:apachectl(8)**
** Main PID: 15109 (httpd)**
** Status: "Processing requests..."**
** CGroup: /system.slice/httpd.service**
** ├─15109 /usr/sbin/httpd -DFOREGROUND**
** ├─15110 /usr/sbin/httpd -DFOREGROUND**
** ├─15111 /usr/sbin/httpd -DFOREGROUND**
** ├─15112 /usr/sbin/httpd -DFOREGROUND**
** ├─15113 /usr/sbin/httpd -DFOREGROUND**
** └─15114 /usr/sbin/httpd -DFOREGROUND**

Sep 16 18:52:06 av.gds.com.vn systemd[1]: Starting The Apache HTTP Server...
Sep 16 18:52:06 av.gds.com.vn httpd[15109]: AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/httpd/conf/httpd.conf:357
Sep 16 18:52:06 av.gds.com.vn systemd[1]: Started The Apache HTTP Server.

But after i run this command, Apache will give an error:

[root@av ~]# sudo certbot --apache -d av.gds.vn -d www.av.gds.vn

● httpd.service - The Apache HTTP Server
** Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)**
** Active: failed (Result: exit-code) since Wed 2020-09-16 18:52:37 +07; 11s ago**
** Docs: man:httpd(8)**
** man:apachectl(8)**
** Process: 15234 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=1/FAILURE)**
** Process: 15171 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)**
** Process: 15232 ExecStart=/usr/sbin/httpd $OPTIONS -DFOREGROUND (code=exited, status=1/FAILURE)**
** Main PID: 15232 (code=exited, status=1/FAILURE)**

Sep 16 18:52:37 av.gds.com.vn httpd[15232]: (98)Address already in use: AH00072: make_sock: could not bind to address [::]:443
Sep 16 18:52:37 av.gds.com.vn httpd[15232]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:443
Sep 16 18:52:37 av.gds.com.vn httpd[15232]: no listening sockets available, shutting down
Sep 16 18:52:37 av.gds.com.vn httpd[15232]: AH00015: Unable to open logs
Sep 16 18:52:37 av.gds.com.vn systemd[1]: httpd.service: main process exited, code=exited, status=1/FAILURE
Sep 16 18:52:37 av.gds.com.vn kill[15234]: kill: cannot find process ""
Sep 16 18:52:37 av.gds.com.vn systemd[1]: httpd.service: control process exited, code=exited status=1
Sep 16 18:52:37 av.gds.com.vn systemd[1]: Failed to start The Apache HTTP Server.
Sep 16 18:52:37 av.gds.com.vn systemd[1]: Unit httpd.service entered failed state.
Sep 16 18:52:37 av.gds.com.vn systemd[1]: httpd.service failed.

The reason is that the file httpd.conf has been changed.

when I delete "Listen 443", Apache is working properly again.

Sorry for uploading the code, I intend to upload the image for easy to understand but new user only allowed to upload 1 picture.

Please help me to resolve it

_az · September 16, 2020, 8:59pm

I think you might be hitting this Certbot bug: https://github.com/certbot/certbot/issues/5517

Listen 443 should appear exactly once in your configuration.

When it appears more than once, you end up with that Address already in use error and Apache can’t start.

Problem is, Certbot is apparently adding it unnecessarily, and that breaks your Apache configuration.

What I would try is the following:

First, remove the <IfModule mod_ssl.c> and </IfModule> lines that surrounds the Listen line. That may be breaking Certbot’s detection of whether the Listen 443 is already done.

Next, check whether anywhere else in your configuration contains that listen:

grep -Ri "Listen 443" /etc/httpd

DucTM · September 17, 2020, 4:34am

Hi _az,

Thanks for your recommendations.

I already removed the <IfModule mod_ssl.c> and </IfModule> lines but Apache still adds "Listen 443" to httpd.conf file when i renew Cerbot.

[root@av ~]# grep -Ri “Listen 443” /etc/httpd
/etc/httpd/conf/httpd.conf.rtFvS5:Listen 443
/etc/httpd/conf/httpd.conf.rtFvS5:#Listen 443
/etc/httpd/conf/httpd.conf:#Listen 443

DucTM · September 17, 2020, 4:53am

Hi _az,

Although I removed them before but actually Apache adds <IfModule mod_ssl.c> and </IfModule> lines + “Listen 443” to httpd.conf file when i run command.

sudo certbot --apache -d av.gds.vn -d www.av.gds.vn

_az · September 17, 2020, 5:26am

Oh, you mean that Apache only works when there are no Listen 443 lines?

I’m not sure how that can be. Is it actually the same PIDs bound to 80 and 443?

ss -tlpn | grep -E ":(80|443)"

DucTM · September 17, 2020, 6:00am

Hi _az,

Yes, Apache only works when there are no Listen 443 lines.

ss -tlpn | grep -E ":(80|443)"

No results are displayed.

[root@av ~]#
[root@av ~]# ss -tlpn | grep -E “: (80 | 443)”
[root@av ~]#

_az · September 17, 2020, 6:03am

Could you please run the command exactly as I wrote it? Without the extra spaces?

Copy-pasting it might help.

DucTM · September 17, 2020, 6:17am

Hi _az,

Here is the result:

[root@av ~]# ss -tlpn | grep -E “:(80|443)”
LISTEN 0 128 :::80 :::* users:((“httpd”,pid=32340,fd=4),(“httpd”,pid=32338,fd=4),(“httpd”,pid=32337,fd=4),(“httpd”,pid=32336,fd=4),(“httpd”,pid=32335,fd=4),(“httpd”,pid=31682,fd=4))
LISTEN 0 128 :::443 :::* users:((“httpd”,pid=5297,fd=4),(“httpd”,pid=4677,fd=4),(“httpd”,pid=4676,fd=4),(“httpd”,pid=4675,fd=4),(“httpd”,pid=1547,fd=4))

_az · September 17, 2020, 6:32am

Thanks. It’s what I suspected earlier - you have two different Apache servers running. That’s why I asked you to do the killall -9.

Certbot is correctly adding the Listen 443, the problem is that Apache is starting up twice, possibly from different configurations.

This is not a Bitnami server, is it?

DucTM · September 17, 2020, 7:16am

Hi _az,

thanks for your help.

Yes, this is not a Bitnami server.

How can i resolve it?

_az · September 17, 2020, 7:28am

I’m not really sure. You have to identify why there’s two different Apaches running.

If the systemctl stop httpd + killall -9 httpd didn’t help, and it’s not a Bitnami server, I’m stumped.

Something that might help would be to trace where each process comes from.

yum -y install psmisc

then

pstree -scp 32340
pstree -scp 5297

and compare the output of each one.

DucTM · September 17, 2020, 7:54am

Hi _az,

I already disabled 1 Apache server and and successfully installed Cerbot.

but Unfortunately, Apache failed to start

_az · September 17, 2020, 8:03am

Ah, so your actual Apache server is ac_httpd. I see.

I think the main thing to understand is that Certbot’s Apache plugin really only works with the Apache server that comes with CentOS. It’s not built to work with ac_httpd.

This means that you would need to manually configure the certificate for ac_httpd, rather than relying on Certbot to autoconfigure it.

You will first want to make sure that unwanted Apache server is completely disabled. You can do so by running:

systemctl stop httpd
systemctl mask httpd

Are you able to restart ac_httpd after doing that?

If so, you should be able to symlink ac_httpd's server.key and server.crt to the private key and certificate, respectively, at /etc/letsencrypt/live/$DOMAIN/{privkey.pem,fullchain.pem}.

Then restart ac_httpd once again.

That should get you up and running with SSL.

DucTM · September 17, 2020, 8:34am

Hi _az,

I really appreciate your help with my problem

I have fixed the error and successfully installed Certbot

system · October 17, 2020, 8:34am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Error while running apachectl graceful Server	3	3257	October 27, 2018
Error while running apachectl graceful - Unable to find a virtual host listening on port 80 Help	2	1275	October 30, 2020
Certificate generation failure because of Job for httpd.service invalid error Help	13	1679	May 18, 2023
Certbot on wordpress docker shows error Unable to restart apache using ['apachectl', 'graceful'] Help	13	1403	November 1, 2022
Let's Encrypt messed up my apache Help	2	1181	February 27, 2018

Error while running apachectl graceful. Job for httpd.service invalid

Related topics