Unable to restart apache after removing domain

Help
#1

I am running a nnumber of VirtualHosts - all was running reasonably fine.

My domain is: canstream.co.uk

I ran these commands to delete a VirtualHost:

  • rm -Rf /etc/letsencrypt/live/podcasts.commedia.org.uk
  • rm /etc/letsencrypt/renewal/podcasts.commedia.org.uk.conf
  • rm -Rf /etc/letsencrypt/archive/podcasts.commedia.org.uk

It produced this output:

[root@machine~]# /etc/init.d/httpd restart
Stopping httpd: [ OK ]
Starting httpd: [FAILED]

My web server is (include version): Server version: Apache/2.2.15 (Unix)

The operating system my web server runs on is (include version): CentOS release 6.6 (Final)

My hosting provider, if applicable, is: myself

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.37.1

Logs:

/var/log/letsencrypt.log.1

2019-08-10 00:11:02,027:DEBUG:acme.client:Storing nonce: S-yp5c7fbdkGQrnPA7zi89i6dSAJjXzCe_5BXEgRgM8
2019-08-10 00:11:02,028:DEBUG:certbot.renewal:Dry run: skipping updating lineage at /etc/letsencrypt/live/canstream.co.uk
2019-08-10 00:11:02,029:DEBUG:certbot.updater:Skipping renewal deployer in dry-run mode.
2019-08-10 00:11:02,231:ERROR:certbot.util:Error while running apachectl graceful.
httpd not running, trying to start

2019-08-10 00:11:02,232:INFO:certbot_apache.configurator:Unable to restart apache using [‘apachectl’, ‘graceful’]
2019-08-10 00:11:02,232:DEBUG:certbot_apache.configurator:Trying alternative restart command: [‘apachectl’, ‘restart’]
2019-08-10 00:11:02,360:ERROR:certbot.util:Error while running apachectl restart.
httpd not running, trying to start

2019-08-10 00:11:02,361:WARNING:certbot.renewal:Attempting to renew cert (canstream.co.uk) from /etc/letsencrypt/renewal/canstream.co.uk.conf produced an unexpected error: Error while running apachectl restart.
httpd not running, trying to start

. Skipping.
2019-08-10 00:11:02,363:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot_apache/configurator.py”, line 2212, in _reload
util.run_script(self.option(“restart_cmd”))
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/util.py”, line 84, in run_script
raise errors.SubprocessError(msg)
certbot.errors.SubprocessError: Error while running apachectl graceful.
httpd not running, trying to start

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/renewal.py”, line 449, in handle_renewal_request
main.renew_cert(lineage_config, plugins, renewal_candidate)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 1220, in renew_cert
installer.restart()
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot_apache/configurator.py”, line 2203, in restart
self._reload()
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot_apache/configurator.py”, line 2230, in _reload
raise errors.MisconfigurationError(error)
certbot.errors.MisconfigurationError: Error while running apachectl restart.
httpd not running, trying to start

Nothing in apache error logs as apache will not start.

Any ideas.

Thanks in advance.

#2

Before restaring Apache I tested the config thus:

[root@machine~]# httpd -t
Syntax OK

And it still checks out ok.

#3

So if you run:

httpd -k restart

Nothing at all appears in /var/log/httpd/error_log?

1 Like
#4

Thanks for coming back.

No.

[root@machine ~]# httpd -k restart
httpd not running, trying to start
[root@machine ~]# service httpd status
httpd is stopped

Nothing in the apache logs to help.

Error log:

[Fri Aug 09 23:29:36 2019] [notice] Digest: generating secret for digest authentication …
[Fri Aug 09 23:29:36 2019] [notice] Digest: done
[Fri Aug 09 23:29:37 2019] [notice] Apache/2.2.15 (Unix) mod_ssl/2.2.15 OpenSSL/1.0.1e-fips DAV/2 configured – resuming normal operations
[Fri Aug 09 23:34:44 2019] [notice] caught SIGTERM, shutting down

That output at 23:34 was when I rebooted and Apache hasn’t worked since.

#5

How about

httpd -k stop
httpd -X -e debug

The second command starts Apache in the foreground with a single worker. It should hopefully error out with the cause of your issue.

1 Like
#6

Hi.

No, nothing.

[root@machine conf.d]# httpd -k stop
httpd (no pid file) not running
[root@machine conf.d]# httpd -X
[root@machine conf.d]#

I’m trying to track down the log at the precise time of failure at 23:34.

#7

Inconclusive.

[root@machine conf.d]# httpd -k stop
httpd (no pid file) not running
[root@machine conf.d]# httpd -X -e debug
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module ssl_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module authz_host_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module auth_basic_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module authn_file_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module authz_user_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module authz_default_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module authz_groupfile_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module auth_digest_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module ldap_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module include_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module log_config_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module logio_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module env_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module mime_magic_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module cern_meta_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module expires_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module deflate_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module headers_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module usertrack_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module setenvif_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module mime_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module dav_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module status_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module autoindex_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module asis_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module info_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module dav_fs_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module vhost_alias_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module negotiation_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module dir_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module actions_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module speling_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module userdir_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module alias_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module rewrite_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module cache_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module suexec_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module disk_cache_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module file_cache_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module mem_cache_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module cgi_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module evasive20_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module geoip_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module security2_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module unique_id_module
[Sat Aug 10 01:08:51 2019] [debug] mod_so.c(246): loaded module php5_module
[root@machine conf.d]#

#8

But if you don’t Ctrl-c it, your webserver is running, right? Can you curl localhost in a second terminal while httpd -X runs?

The reason I ask is that it sounds like your init scripts or /var/run/httpd might be busted, but it depends on whether the server on its own serves traffic or not.

1 Like
#9

No.

After I issue httpd -X -e debug and I am just returned to the command line where I can do this:

[root@listenagain conf.d]# service httpd status
httpd is stopped

The server is remote and I am ssh’ed on to it so maybe an X session won’t start?

#10

I don’t really care what service httpd status thinks - it’s just looking at the pidfile in /var/run.

I want to know whether you can open a second SSH window and run curl localhost with httpd -X running in the first SSH window.

1 Like
#11

Thanks.

Ok, when I open a second ssh window and run curl localhost with httpd -X running in the first SSH window I just get this:

[root@listenagain conf.d]# curl localhost
curl: (7) couldn’t connect to host

The letsencrypt log nearest to the failure I can find so far is this:

2019-08-09 23:36:25,942:DEBUG:acme.client:Storing nonce: EqB6YZUw_rcWlF7BGkNTIkobfBY70F9QTRezr5LK3Fs
2019-08-09 23:36:25,943:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/bin/letsencrypt”, line 11, in
load_entry_point(‘letsencrypt==0.7.0’, ‘console_scripts’, ‘letsencrypt’)()
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 1378, in main
return config.func(config, plugins)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 1133, in run
certname, lineage)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/main.py”, line 121, in _get_and_save_cert
lineage = le_client.obtain_and_enroll_certificate(domains, certname)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/client.py”, line 422, in obtain_and_enroll_certificate
self.config)
File “/opt/eff.org/certbot/venv/lib64/python3.4/site-packages/certbot/storage.py”, line 1005, in new_lineage
"archive directory exists for " + lineagename)
certbot.errors.CertStorageError: archive directory exists for podcasts.commedia.org.uk

And then I manually deleted that directory.

#12

Ah, I have got apache going now…

I went back into an apache.conf file and commented out this:

Include /etc/letsencrypt/options-ssl-apache.conf

Looks like SSL is working again.

Thank you for your intervention @_az.

#13

Hi @_az, any advice on how I might safely revoke certificates and start again.

Everything had been working reasonably well and I wanted to improve the set-up.

I renewed certs using this sort of set-up:

certbot-auto --apache -d example.com -d www.example.com -d example1.com -d www.example1.com -d example2.com -d www.example2.com […]

and I thought it would be a good idea to set-up the certs individually:

certbot-auto --apache -d example.com -d www.example.com
certbot-auto example1.com -d www.example1.com
certbot-auto -d example2.com -d www.example2.com […]

But now I’m getting Certificate name mismatch errors which I didn’t have before.

I’m apprehensive about making more changes which might break things as I need to sleep.

#14

In general, you do not want to revoke certificates. The only reason to do that is if you believe that your certificate private key has been compromised. You can delete certificates without revoking them using

certbot-auto delete --cert-name <certificate name>

and you can find the certificate name by running:

certbot-auto certificates

Before deleting certificates, make sure that your httpd config does not rely on them. Otherwise you will cause httpd to be unable to start.

Your approach of separating your domains across different certificates is correct (though each command should include --apache). If you experience certificate mismatch errors, it’s probably because of your Apache virtualhost setup and how your new certificates map to them. Naturally, it’s not possible to give you any advice on fixing that with redacted domains.

2 Likes
#15

Thank you @_az.

I’ll continue to separate out the domains and try again. This is what I was doing when it all went pear-shaped.

Thanks again for your help.

#16

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.