Trying to install certbot I broke Apache

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: safeandtacticalfirearmstraining.com

I ran this command: Many - log attached.

It produced this output: Multiple errors - Log attached

My web server is (include version): Apache2 2.4.61

The operating system my web server runs on is (include version): Opensuse LEAP 15.4

My hosting provider, if applicable, is: Me

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 154.2.1

I haven't needed my webserver for over a year, so the SSH certificate expired. I had it set up to auto-renew using CertBot but the renewal failed. It's been so long since I set it up I have forgotten most of the setup, so the online research began starting with the error codes. I tried multiple fixes I found on online forums but nothing accomplished a new certificate installed.

My last command has broken Apache. It's time for some serious help.

First priority: Unmuck Apache.
Next priority: Get Certbot operational.

I have attached the apache log since I started tinkering.

[Thu Mar 14 03:53:39.245378 2024] [core:error] [pid 7256] [client 141.98.11.96:46442] AH10244: invalid URI path (/../../mnt/mtd/Config/Account1)
[Thu Mar 14 04:38:33.949184 2024] [cgi:error] [pid 22538] [client 80.94.92.60:37290] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Thu Mar 14 04:38:33.951559 2024] [cgi:error] [pid 22538] [client 80.94.92.60:37290] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Thu Mar 14 04:41:17.089589 2024] [core:error] [pid 19498] [client 141.98.11.96:49350] AH10244: invalid URI path (/../../mnt/mtd/Config/Account2)
[Thu Mar 14 07:25:29.707159 2024] [cgi:error] [pid 19488] [client 80.94.92.60:52856] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Thu Mar 14 07:25:29.708044 2024] [cgi:error] [pid 19488] [client 80.94.92.60:52856] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Thu Mar 14 10:59:39.538135 2024] [negotiation:error] [pid 1669] [client 47.74.90.127:36604] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Thu Mar 14 11:07:05.499495 2024] [negotiation:error] [pid 19491] [client 47.74.90.127:46282] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Thu Mar 14 12:18:56.866664 2024] [php7:error] [pid 19491] [client 134.209.163.92:38016] script '/srv/www/htdocs/upl.php' not found or unable to stat
[Thu Mar 14 12:18:57.529559 2024] [php7:error] [pid 7253] [client 134.209.163.92:38042] script '/srv/www/htdocs/1.php' not found or unable to stat
[Thu Mar 14 12:18:58.207919 2024] [php7:error] [pid 1669] [client 134.209.163.92:38080] script '/srv/www/htdocs/password.php' not found or unable to stat
[Thu Mar 14 12:18:58.382160 2024] [php7:error] [pid 22538] [client 134.209.163.92:38086] script '/srv/www/htdocs/info.php' not found or unable to stat
[Thu Mar 14 16:51:38.140804 2024] [negotiation:error] [pid 7255] [client 8.222.253.90:38656] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Thu Mar 14 17:28:09.024809 2024] [negotiation:error] [pid 1669] [client 8.222.253.90:52966] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 00:08:52.045402 2024] [cgi:error] [pid 7255] [client 80.94.92.60:49430] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Fri Mar 15 00:08:52.047254 2024] [cgi:error] [pid 7255] [client 80.94.92.60:49430] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Fri Mar 15 01:06:09.568834 2024] [php7:error] [pid 19490] [client 138.197.102.212:49748] script '/srv/www/htdocs/upl.php' not found or unable to stat
[Fri Mar 15 01:06:10.251181 2024] [php7:error] [pid 22538] [client 138.197.102.212:49780] script '/srv/www/htdocs/1.php' not found or unable to stat
[Fri Mar 15 01:06:10.941721 2024] [php7:error] [pid 7256] [client 138.197.102.212:49802] script '/srv/www/htdocs/password.php' not found or unable to stat
[Fri Mar 15 01:06:11.114907 2024] [php7:error] [pid 19494] [client 138.197.102.212:49812] script '/srv/www/htdocs/info.php' not found or unable to stat
[Fri Mar 15 02:22:53.786352 2024] [negotiation:error] [pid 19498] [client 8.217.149.64:42240] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 02:22:54.034896 2024] [negotiation:error] [pid 7256] [client 8.217.149.64:42242] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 06:19:26.463931 2024] [cgi:error] [pid 19490] [client 80.94.92.60:33600] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Fri Mar 15 06:19:26.465642 2024] [cgi:error] [pid 19490] [client 80.94.92.60:33600] AH02811: script not found or unable to stat: /srv/www/cgi-bin/luci
[Fri Mar 15 06:34:35.552329 2024] [negotiation:error] [pid 19491] [client 8.209.68.21:42174] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 06:34:35.694549 2024] [negotiation:error] [pid 7253] [client 8.209.68.21:42176] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 06:37:30.573630 2024] [php7:error] [pid 7256] [client 143.198.173.69:33316] script '/srv/www/htdocs/phpMyAdmin/index.php' not found or unable to stat
[Fri Mar 15 09:30:11.805682 2024] [negotiation:error] [pid 7255] [client 47.90.254.226:42236] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 09:31:52.058467 2024] [negotiation:error] [pid 22538] [client 47.90.254.226:56184] AH00690: no acceptable variant: /usr/share/apache2/error/HTTP_NOT_FOUND.html.var
[Fri Mar 15 10:42:41.331835 2024] [mpm_prefork:notice] [pid 1498] AH00171: Graceful restart requested, doing restart
[Fri Mar 15 10:42:41.430819 2024] [ssl:error] [pid 1498] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=safeandtacticalfirearmstraining.com / issuer: CN=ZeroSSL RSA Domain Secure Site CA,O=ZeroSSL,C=AT / serial: 150CA3705634E48B5862A9A98A11679B / notbefore: Feb  7 00:00:00 2022 GMT / notafter: May  8 23:59:59 2022 GMT]
[Fri Mar 15 10:42:41.430849 2024] [ssl:error] [pid 1498] AH02604: Unable to configure certificate www.safeandtacticalfirearmstraining.com:443:0 for stapling
[Fri Mar 15 10:42:41.431482 2024] [ssl:error] [pid 1498] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=safeandtacticalfirearmstraining.com / issuer: CN=ZeroSSL RSA Domain Secure Site CA,O=ZeroSSL,C=AT / serial: 150CA3705634E48B5862A9A98A11679B / notbefore: Feb  7 00:00:00 2022 GMT / notafter: May  8 23:59:59 2022 GMT]
[Fri Mar 15 10:42:41.431496 2024] [ssl:error] [pid 1498] AH02604: Unable to configure certificate www.safeandtacticalfirearmstraining.com:443:0 for stapling
[Fri Mar 15 10:42:41.431643 2024] [mpm_prefork:notice] [pid 1498] AH00163: Apache/2.4.51 (Linux/SUSE) PHP/7.4.33 OpenSSL/1.1.1l configured -- resuming normal operations
[Fri Mar 15 10:42:41.431653 2024] [core:notice] [pid 1498] AH00094: Command line: '/usr/sbin/httpd-prefork -D SYSCONFIG -D SSL -D phpMyAdmin -C PidFile /run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -D SYSTEMD -D FOREGROUND'
[Fri Mar 15 11:51:39.791797 2024] [mpm_prefork:notice] [pid 1498] AH00171: Graceful restart requested, doing restart
[Fri Mar 15 11:51:39.863822 2024] [ssl:error] [pid 1498] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=safeandtacticalfirearmstraining.com / issuer: CN=ZeroSSL RSA Domain Secure Site CA,O=ZeroSSL,C=AT / serial: 150CA3705634E48B5862A9A98A11679B / notbefore: Feb  7 00:00:00 2022 GMT / notafter: May  8 23:59:59 2022 GMT]
[Fri Mar 15 11:51:39.863853 2024] [ssl:error] [pid 1498] AH02604: Unable to configure certificate www.safeandtacticalfirearmstraining.com:443:0 for stapling
[Fri Mar 15 11:51:39.864378 2024] [ssl:error] [pid 1498] AH02217: ssl_stapling_init_cert: can't retrieve issuer certificate! [subject: CN=safeandtacticalfirearmstraining.com / issuer: CN=ZeroSSL RSA Domain Secure Site CA,O=ZeroSSL,C=AT / serial: 150CA3705634E48B5862A9A98A11679B / notbefore: Feb  7 00:00:00 2022 GMT / notafter: May  8 23:59:59 2022 GMT]
[Fri Mar 15 11:51:39.864386 2024] [ssl:error] [pid 1498] AH02604: Unable to configure certificate www.safeandtacticalfirearmstraining.com:443:0 for stapling
[Fri Mar 15 11:51:39.864521 2024] [mpm_prefork:notice] [pid 1498] AH00163: Apache/2.4.51 (Linux/SUSE) PHP/7.4.33 OpenSSL/1.1.1l configured -- resuming normal operations
[Fri Mar 15 11:51:39.864530 2024] [core:notice] [pid 1498] AH00094: Command line: '/usr/sbin/httpd-prefork -D SYSCONFIG -D SSL -D phpMyAdmin -C PidFile /run/httpd.pid -C Include /etc/apache2/sysconfig.d//loadmodule.conf -C Include /etc/apache2/sysconfig.d//global.conf -f /etc/apache2/httpd.conf -c Include /etc/apache2/sysconfig.d//include.conf -D SYSTEMD -D FOREGROUND'
2

That's not a valid Certbot version number.

And it also seems you're using ZeroSSL certificates and not Let's Encrypt certificates?

4 Likes
3

Using certbot --version it says 1.22.0.

I previously used ZeroSSL - thought I was now using Let's Encrypt certificates. Another opportunity to learn about this process. I'd like to know how to configure certbot properly.

4

You are using GoDaddy as your DNS provider and it looks like you have its URL Redirect (or URL Forward) service activated. Your first step to support HTTPS is to disable that and add an A record pointing to the public IP of your server. And an AAAA record if you support IPv6.

Your Apache is not the one replying to HTTP requests. The GoDaddy redirect service is doing that instead. That won't work for HTTPS support.

After that is resolved (see GoDaddy docs) show result of below. We'll see what kind of state your local certs are in.

sudo certbot certificates
4 Likes
5

Nevermind that you dont even need to use certbot with modern Apache thanks to the inclusion of mod_md in the core.

https://httpd.apache.org/docs/2.4/mod/mod_md.html

5 Likes
6

I removed site forwarding at GoDaddy and added an A-record pointing to my public IP address.

However, 1) my site was working before I tried to configure certbot, and 2) Apache2 will not start.

systemctl start apache2
Job for apache2.service failed because the control process exited with error code.
See "systemctl status apache2.service" and "journalctl -xeu apache2.service" for details.

That result in apache2.log should be reflected in the log file posted above.

Here is journalctl -xeu apache2.service:

░░ Subject: A start job for unit apache2.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit apache2.service has begun execution.
░░ 
░░ The job identifier is 81000.
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: (98)Address already in use: AH0007>
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: no listening sockets available, sh>
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: AH00015: Unable to open logs
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Main process exited, code=>
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ An ExecStart= process belonging to unit apache2.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Failed with result 'exit-c>
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit apache2.service has entered the 'failed' state with result 'exit-code'.
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: Failed to start The Apache Webserver.
░░ Subject: A start job for unit apache2.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ A start job for unit apache2.service has finished with a failure.
░░ 
░░ The job identifier is 81000 and the job result is failed.
lines 47-96/96 (END)
░░ An ExecStart= process belonging to unit apache2.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
Mar 15 19:07:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Failed with result 'exit-co>
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░ 
░░ The unit apache2.service has entered the 'failed' state with result 'exit-code'.

This section repeat many times. It appears to be the same info so I selected one iteration of the error message.

7

Still are pointing at a URL forwarding service. Below are your current A records from your authoritive DNS servers

;; ANSWER SECTION:
safeandtacticalfirearmstraining.com.	0	IN	A	15.197.148.33
safeandtacticalfirearmstraining.com.	0	IN	A	3.33.130.190

As for this

Was it working with HTTPS?

What does this show?

sudo apache2ctl -t

And this

sudo systemctl --no-pager -l status apache2
2 Likes
8

Also please show

sudo netstat -pant | grep -i listen | grep -E ':80|:443'
2 Likes
9

  1. Godaddy lists both entries but indicates neither can be edited or deleted: You can't modify records that have been applied by a product or service connected to your domain.

  2. I expected HTTPS would be default and didn't look. Didn't know it was important.

  3. sudo apache2ctl -t
    Syntax OK

  4. sudo systemctl --no-pager -l status apache2
    × apache2.service - The Apache Webserver
    Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
    Active: failed (Result: exit-code) since Sun 2024-03-17 19:36:08 PDT; 57min ago
    Process: 9632 ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start (code=exited, status=1/FAILURE)
    Main PID: 9632 (code=exited, status=1/FAILURE)
    Status: "Reading configuration..."

Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: Starting The Apache Webserver...
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: no listening sockets available, shutting down
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: AH00015: Unable to open logs
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Failed with result 'exit-code'.
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: Failed to start The Apache Webserver.

  1. sudo systemctl --no-pager -l status apache2
    × apache2.service - The Apache Webserver
    Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
    Active: failed (Result: exit-code) since Sun 2024-03-17 19:36:08 PDT; 57min ago
    Process: 9632 ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start (code=exited, status=1/FAILURE)
    Main PID: 9632 (code=exited, status=1/FAILURE)
    Status: "Reading configuration..."

Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: Starting The Apache Webserver...
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: no listening sockets available, shutting down
Mar 17 19:36:08 safeandtacticalfirearmstraining.com start_apache2[9632]: AH00015: Unable to open logs
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Failed with result 'exit-code'.
Mar 17 19:36:08 safeandtacticalfirearmstraining.com systemd[1]: Failed to start The Apache Webserver.
safeandtacticalfirearmstraining:/etc/letsencrypt # sudo netstat -pant | grep -i listen | grep -E ':80|:443'
tcp 0 0 0.0.0.0:80 0.0.0.0:* LISTEN 2600/nginx: worker

10

You must first disable the URL Forwarding service then you can change the DNS entries. Please see your GoDaddy docs

It is what certs are used for so essential for this discussion

You have an nginx server using port 80. That is preventing Apache from starting as it also wants port 80. Apache could not possibly have worked alongside this nginx even just for HTTP.

What is nginx used for?

Did you try doing certbot --nginx at one time even though you are not using nginx?

Here is the essential part of the Apache error message and why nginx is a conflict

systemd[1]: Starting The Apache Webserver...
start_apache2[9632]: (98)Address already in use: 
AH00072: make_sock: could not bind to address 0.0.0.0:80
start_apache2[9632]: no listening sockets available, shutting down
2 Likes
11

I read thru the Apache Docs. Not sure what I should use at this point. I installed the Apache Mod and later Nginx at the direction of a forum contributor. Often the situations presented are similar but not identical, different distros and worse, the contributors just give code without an explanation. Makes it difficult to navigate all the advice out there.

I do think that native utilities provided by the source (Apache) is a good idea. I'll evaluate it once I'm up again. Thx for your input.

2 Likes
12

Ok, of course your HTTPS comments make sense. ...long day.

GoDaddy forwarding has been removed.

I uninstalled Nginx, but Apache still will not start. Here is the output:

systemctl status apache2
× apache2.service - The Apache Webserver
Loaded: loaded (/usr/lib/systemd/system/apache2.service; enabled; vendor preset: disabled)
Active: failed (Result: exit-code) since Sun 2024-03-17 21:01:26 PDT; 12s ago
Process: 13652 ExecStart=/usr/sbin/start_apache2 -DSYSTEMD -DFOREGROUND -k start (code=exited, status=1/FAILURE)
Main PID: 13652 (code=exited, status=1/FAILURE)
Status: "Reading configuration..."

Mar 17 21:01:26 safeandtacticalfirearmstraining.com systemd[1]: Starting The Apache Webserver...
Mar 17 21:01:26 safeandtacticalfirearmstraining.com start_apache2[13652]: (98)Address already in use: AH00072: make_sock: could not bind to address 0.0.0.0:80
Mar 17 21:01:26 safeandtacticalfirearmstraining.com start_apache2[13652]: no listening sockets available, shutting down
Mar 17 21:01:26 safeandtacticalfirearmstraining.com start_apache2[13652]: AH00015: Unable to open logs
Mar 17 21:01:26 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Main process exited, code=exited, status=1/FAILURE
Mar 17 21:01:26 safeandtacticalfirearmstraining.com systemd[1]: apache2.service: Failed with result 'exit-code'.
Mar 17 21:01:26 safeandtacticalfirearmstraining.com systemd[1]: Failed to start The Apache Webserver.

13

My guess is that nginx is still running even though you uninstalled it. You could try checking the netstat command again To see what is listening on port 80 or just reboot your server that should clean it up.

And you are not gonna like hearing this but your dns is still not correct. Try using a tool like https://unboundtest.com to check the IP address in your A record.

3 Likes
14

Making some headway! After rebooting Apache did start. Now I can reach my webpage by ip address but not by the text URL. Apparently there is a DNS issue. I checked GoDaddy and confirmed the A-record URL & IP address are correct. They say it could take up to 48 hours for DNS to process but that has not been my experience - usually an hour.

15

Good to see that progress.

Maybe disconnecting their forwarding service long. But, I have seen this issue many times on this forum and 48 hours seems a long time.

Use https://unboundtest.com to monitor the A record. It checks the authoritive servers directly and so not affected by any further TTL propagation delay.

3 Likes
16

Hi Mike, I just logged into GoDaddy and discovered a "Parked" A-record on the domain. A GoDaddy rep told me they automatically add that record whenever there is a DNS change. So I again have to wait for propogation.

If you're available, let's continue with the security certificate using CertBot or mod_md. I note that a security warning appears when I connect to my website using the IP address.

17

I don't see an A record there at all right now (using https://unboundtest.com)

So, you might try adding one if you can. I was just signing off but might have time tomorrow to check on your progress. Other volunteers could help too.

2 Likes
18

A-record is already there - Again they say 24 to 48 hrs. Is DNS required to add a certificate?

19

Let's Encrypt queries the domain's authoritative namseververs, so the 24-48 hour propagation timeline does not apply.

2 Likes
20

Ok. Guess I'll have to be patient then. Thx.

1 Like