Certbot kills apache. Need help to restart httpd

Good day to all:

I had certbot renew scheduled to run. It ran fine for a few months, even renewed the certificates twice. Last night it ran successfully, and the certificates were generated, but the website does not load.

When I load the backup image, I can see one root httpd server and 4 user processes. This is the command I use: sudo lsof -i tcp:80 -s tcp:listen

On the latest updated version, it shows one root and 3 user httpd processes.

Environment: Centos 7
The certbot is 0.27.1
The systemctl status shows httpd is Active. But the website does not load.

regular systemctl stop/start httpd do not do any good.
Logs do not show any errors (checked /var/log/letsencrypt).

My certificate expired a few hours back.
I would like at least to properly restart the httpd service, so I can have the next 90 days putting in a proper fix. Please help.

Thank you.

Hi @genrkha

please answer the following questions. That’s the standard template of #help


Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

Hi, Juergen:

No secrets from you. Just did not know how much info is required. Thank you for guiding me.

The domain is: back2rent.com
The web server is: apache 2.4.29
Operation System: Centos 7
Provider: Linode
I use command line. Have full admin.
I do not use any control panel (besides Linode DNS Manager)
The certbot version is 0.27.1
The command that caused the issue is certbot renew --apache

Now that I think of it, during the last “not up for renewal” run, apache also went down. I just went back to prior backup image without doing any investigation.

– I did some troubleshooting yesterday. --dry-run option also brings the server down.

Thank you for your time.

Looks like your https doesn't work ( https://check-your-website.server-daten.de/?q=back2rent.com ):

You have ipv4 and ipv6:

Host T IP-Address is auth. ∑ Queries ∑ Timeout
back2rent.com A 50.116.33.194 yes 1 0
AAAA 2600:3c02::f03c:91ff:fecc:210a yes
www.back2rent.com A 50.116.33.194 yes 1 0
AAAA 2600:3c02::f03c:91ff:fecc:210a yes

But the list with the url-checks ends in a blocked answer:

Domainname Http-Status redirect Sec. G
http://back2rent.com/
50.116.33.194 301 https://www.back2rent.com/ 0.240 E
http://back2rent.com/
2600:3c02::f03c:91ff:fecc:210a 301 https://www.back2rent.com/ 0.267 E
http://www.back2rent.com/
50.116.33.194 301 https://www.back2rent.com/ 0.240 A
http://www.back2rent.com/
2600:3c02::f03c:91ff:fecc:210a 301 https://www.back2rent.com/ 0.270 A
https://back2rent.com/
50.116.33.194 -2 1.350 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 50.116.33.194:443
https://back2rent.com/
2600:3c02::f03c:91ff:fecc:210a -2 1.393 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it [2600:3c02::f03c:91ff:fecc:210a]:443
https://www.back2rent.com/
50.116.33.194 -2 1.357 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 50.116.33.194:443
https://www.back2rent.com/
2600:3c02::f03c:91ff:fecc:210a -2 1.397 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it [2600:3c02::f03c:91ff:fecc:210a]:443
http://back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
50.116.33.194 301 https://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 0.243 E
Visible Content: Moved Permanently The document has moved here .
http://back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
2600:3c02::f03c:91ff:fecc:210a 301 https://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 0.244 E
Visible Content: Moved Permanently The document has moved here .
http://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
50.116.33.194 301 https://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 0.240 A
Visible Content: Moved Permanently The document has moved here .
http://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
2600:3c02::f03c:91ff:fecc:210a 301 https://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de 0.240 A
Visible Content: Moved Permanently The document has moved here .
https://www.back2rent.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de -2 2.840 V
ConnectFailure - Unable to connect to the remote server No connection could be made because the target machine actively refused it 50.116.33.194:443

Redirects http -> https are ok, Letsencrypt follows these redirects. But if the destination doesn't answer, it's bad.

So

  • remove the redirects to create a certificate (or)
  • use dns-01 validation + --manual, that should always work (or)
  • first install a self signed certificate (your installation should have one), so your port 443 is running

Sorry for silly questions. I am not sysadmin by trade, used google and forums to install everything.

I already have a certificate generated by letsencrypt last night. I validated it with openssl command, and it showed that it is valid and expires in June. I thought I could use it, and the only issue is https connection.

Thank you.

I don't see a https connection.

I see a blocking / active firewall.

So I don't know: Is it a firewall problem? Or is no webserver running port 443?

Does https work internal?

Did Certbot created a vHost?

What says

Apachectl -S

There is another website that I am hosting: floridamakler.com

Both of these websites ran together on the same box and their certificates were issued and renewed with the same process.

Yesterday I moved floridamakler.com to a different box. Both boxes are built from the same backup image. So, before the “certbot renew” runs, both servers are identical. The certificate expired today in the morning, so I do not know if querying the floridamakler.com will provide any clues.

I compared both, apache and letsencrypt folders on both nodes, and they are identical even after the certbot run. With exception of a newly created certificate.

The initial httpd-ssl.conf was initially created by the certbot when I first created the certifcates back in November. I also had issues with redirecting 80 to 443, so I disconnected the redirect.

Here is the output from apachectl -S:

VirtualHost configuration:
*:443 www.floridamakler.net (/etc/httpd/conf.d/ssl.conf:56)
ServerRoot: “/etc/httpd”
Main DocumentRoot: “/var/www/html”
Main ErrorLog: “/etc/httpd/logs/error_log”
Mutex mpm-accept: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex proxy-balancer-shm: using_defaults
Mutex rewrite-map: using_defaults
Mutex authdigest-client: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex authn-socache: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/run/httpd/" mechanism=default
PidFile: “/run/httpd/httpd.pid”
Define: _RH_HAS_HTTPPROTOCOLOPTIONS
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name=“apache” id=48
Group: name=“apache” id=48

Really?

Then you don't have a vHost with back2rent.com.

ServerName back2rent.com
ServerAlias www.back2rent.com

is required.

My bad. I ran as sudo. I do have 2 apache installations. 2.4.6 came from the default storage. And I cannot get rid of it. Anyway, I ran as root:

[root@www ~]# apachectl -S
VirtualHost configuration:
*:80 is a NameVirtualHost
default server default (/apache/conf/extra/http_vhosts.conf:1)
port 80 namevhost default (/apache/conf/extra/http_vhosts.conf:1)
port 80 namevhost www.floridamakler.net (/apache/conf/extra/http_vhosts.conf:9)
port 80 namevhost floridamakler.net (/apache/conf/extra/http_vhosts.conf:14)
port 80 namevhost www.floridamakler.com (/apache/conf/extra/http_vhosts.conf:20)
port 80 namevhost floridamakler.com (/apache/conf/extra/http_vhosts.conf:25)
port 80 namevhost www.back2rent.com (/apache/conf/extra/http_vhosts.conf:47)
port 80 namevhost back2rent.com (/apache/conf/extra/http_vhosts.conf:52)
port 80 namevhost default (/apache/conf/extra/http_vhosts.conf:1)
port 80 namevhost www.floridamakler.net (/apache/conf/extra/http_vhosts.conf:9)
port 80 namevhost floridamakler.net (/apache/conf/extra/http_vhosts.conf:14)
port 80 namevhost www.floridamakler.com (/apache/conf/extra/http_vhosts.conf:20)
port 80 namevhost floridamakler.com (/apache/conf/extra/http_vhosts.conf:25)
port 80 namevhost www.back2rent.com (/apache/conf/extra/http_vhosts.conf:47)
port 80 namevhost back2rent.com (/apache/conf/extra/http_vhosts.conf:52)
*:443 is a NameVirtualHost
default server back2rent.com (/apache/conf/extra/httpd-ssl.conf:122)
port 443 namevhost back2rent.com (/apache/conf/extra/httpd-ssl.conf:122)
alias www.back2rent.com
port 443 namevhost floridamakler.com (/apache/conf/extra/httpd-ssl.conf:294)
alias www.floridamakler.com
ServerRoot: “/apache”
Main DocumentRoot: “/opt/apache-2.4.29/htdocs”
Main ErrorLog: “/apache/logs/error_log”
Mutex default: dir="/apache/logs/" mechanism=default
Mutex rewrite-map: using_defaults
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
PidFile: “/apache/logs/httpd.pid”
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: MODSEC_2.5
Define: MODSEC_2.9
User: name=“apache” id=48
Group: name=“apache” id=48

Uh.

That’s terrible.

Every combination of port and domain name should be unique, only one time.

Then it’s completely unclear which vHost is used.

Something happens after the certbot runs. The only thing that I saw promising is the number of apache instances. Before the run I run the command to see what is listening on port 80. The result is 1 process run as root and 4 processes run as apache (user). After the run I see 1 as root and 3 as user. I reloaded the image multiple times and always the same thing.

I created the initial certificate with the certbot parameters which point to apache.2.24.29

That's also terrible. Perhaps Certbot has changed the wrong Apache.

This might be the case. Will compare files before and after.

Good morning:

I fixed the problem. But still do not know the root cause.

The certificates were generated successfully, so I copied them to the other server to the /etc/letsencrypt/archive, modified links in the /etc/letsencrypt/live/back2rent.com to point to the new certificates. And rebooted the server (probably restart httpd would have also worked).

Juergen, thank you very much for your help.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.