Certbot Apache renewal stopped working (rpi)

Help
1

The certificate automatic renewal has stopped working.The certificate is still valid and I am able to access my webserver at http://krewik.net/ which redirects to and https://krewik.net/.

But the automatic renewal has not gone thru and when I attempt to trigger it manually I get an error message, see below. In an attempt to solve the issue I followed these instructions Certbot Instructions | Certbot and installed snapd and replaced the certbot with it's version but the issue persits - connection refused.

Could it be that certbot is not creating the challenge files in the correct place or that the webserver is unable to read them?

Any help is very much appreciated.

My domain is: krewik.net

I ran this command: sudo certbot renew

It produced this output:
matias@rpi4-lamp:~$ sudo certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/krewik.net.conf

Renewing an existing certificate for krewik.net and 2 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
Domain: krewik.net
Type: connection
Detail: 188.149.197.26: Fetching http://krewik.net/.well-known/acme-challenge/TB68WccEicn2RFOFI7HCrp7zGu7mvI9k1CvZ4wu6CIc: Connection refused

Domain: wishlist.krewik.net
Type: connection
Detail: 188.149.197.26: Fetching http://wishlist.krewik.net/.well-known/acme-challenge/z2qXrBeMJzfa7PVGonsjhPqNpbg5CqKx9byKMmrJVA0: Connection refused

Domain: www.krewik.net
Type: connection
Detail: 188.149.197.26: Fetching http://www.krewik.net/.well-known/acme-challenge/Scx_wVfcvQ-ZCx_9bpx7kaueBDQpi4BtlnZ4QZyNVRI: Connection refused

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate krewik.net with error: Some challenges have failed.

All renewals failed. The following certificates could not be renewed:
/etc/letsencrypt/live/krewik.net/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):
Server version: Apache/2.4.41 (Ubuntu)
Server built: 2022-06-14T13:30:55

The operating system my web server runs on is (include version):
Linux rpi4-lamp 5.4.0-1066-raspi #76-Ubuntu SMP PREEMPT Mon Jun 27 11:02:52 UTC 2022 aarch64 aarch64 aarch64 GNU/Linux

My hosting provider, if applicable, is:
Not applicable, hosting in my own raspberry pi

I can login to a root shell on my machine:
Yes

I'm using a control panel to manage my site:
No

The version of my client is:
certbot 1.29.0

2

Welcome to the community @matkr265

It looks like your port 80 for http requests is closed. You have port 443 open but port 80 needs to be open for the ACME HTTP challenge to work.

Check your firewalls and/or router to see if they are blocking that. Check that your port 80 VirtualHost in Apache is still enabled.

5 Likes
3

Thanks for the reply @MikeMcQ, but I think port 80 is open. The webserver responds to requests on http with a 301 and points to https instead. Is that not allowed or am I reading the devtools in chrome incorrectly?

Request

GET / HTTP/1.1
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9
Accept-Encoding: gzip, deflate
Accept-Language: en,sv;q=0.9
Cache-Control: no-cache
Connection: keep-alive
Host: krewik.net
Pragma: no-cache
Upgrade-Insecure-Requests: 1
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/103.0.0.0 Safari/537.36

Response

HTTP/1.1 301 Moved Permanently
Date: Fri, 22 Jul 2022 13:41:01 GMT
Server: Apache/2.4.41 (Ubuntu)
Location: https://krewik.net/
Content-Length: 303
Keep-Alive: timeout=5, max=100
Connection: Keep-Alive
Content-Type: text/html; charset=iso-8859-1
1 Like
4

Is that just from your local network? Because I get this

curl -I krewik.net
curl: (7) Failed to connect to krewik.net port 80 after 420 ms: Connection refused

And so does the Let's Debug test site

4 Likes
5

Not from where I sit:

 ✘ dan@Dan-MacBook-Pro-2013  ~  curl www.krewik.net
curl: (7) Failed to connect to www.krewik.net port 80: Connection refused
5 Likes
6 
PORT     STATE  SERVICE
80/tcp   closed http
222/tcp  open   rsh-spx
443/tcp  open   https
5061/tcp closed sip-tls
5 Likes
7

@MikeMcQ yes you are correct, only from my internal network, sorry. I am out of my depth, but too me it looks like Apache is listening on 80 and 443 and that the ports are open in the firewall as well as forwarded in my router.

However, as you all stated, requests to port 80 outside my internal network does not yield a response. As I think is proved by the iptables logs.

I m beginning to think my ISP has started to block port 80, or is it possible that my LAMP server is configured to only allow port 80 internally?

Netstat

matias@rpi4-lamp:~$ sudo netstat -tulpn | grep ':80\|:443'
tcp6       0      0 :::80                   :::*                    LISTEN      885/apache2
tcp6       0      0 :::443                  :::*                    LISTEN      885/apache2

Iptables

matias@rpi4-lamp:~$ sudo iptables-save|grep "spt:\|dpt:\|dports\|sports"
-A ufw-user-input -p tcp -m multiport --dports 80,443 -m comment --comment "\'dapp_Apache%20Full\'" -j ACCEPT

Iptables log rules

matias@rpi4-lamp:~$ sudo iptables -L INPUT  --line-numbers
Chain INPUT (policy DROP)
num  target     prot opt source               destination
1    LOG        tcp  --  anywhere             anywhere             tcp dpt:https flags:FIN,SYN,RST,ACK/SYN LOG level warning prefix "HTTPS SYN: "
2    LOG        tcp  --  anywhere             anywhere             tcp dpt:http flags:FIN,SYN,RST,ACK/SYN LOG level warning prefix "HTTP SYN: "

Iptables log output

Jul 22 23:44:31 rpi4-lamp kernel: [37764.748665] HTTP SYN: IN=wlan0 OUT= MAC=e4:5f:01:27:b5:e6:44:d4:53:3b:e7:17:08:00 SRC=188.149.197.26 DST=192.168.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=5272 DF PROTO=TCP SPT=55649 DPT=80 WINDOW=64240 RES=0x00 SYN URGP=0
Jul 22 23:44:31 rpi4-lamp kernel: [37764.748789] HTTPS SYN: IN=wlan0 OUT= MAC=e4:5f:01:27:b5:e6:44:d4:53:3b:e7:17:08:00 SRC=188.149.197.26 DST=192.168.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=127 ID=5273 DF PROTO=TCP SPT=55650 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0
Jul 22 23:45:18 rpi4-lamp kernel: [37811.230945] HTTPS SYN: IN=wlan0 OUT= MAC=e4:5f:01:27:b5:e6:44:d4:53:3b:e7:17:08:00 SRC=178.174.161.170 DST=192.168.0.29 LEN=52 TOS=0x00 PREC=0x00 TTL=120 ID=52713 DF PROTO=TCP SPT=53876 DPT=443 WINDOW=64240 RES=0x00 SYN URGP=0
8

Is there a NAT device?
If so, please show that table.

3 Likes
9

@rg305 I think you mean port forwarding settings in my router, image below.

router_portforward
router_portforward972×1228 61.1 KB

10

Can you show the firewall tab in your router too. Thanks

4 Likes
11

Hello @MikeMcQ, firewall tab below.

ISP support is adamant that they do not block port 80. I will attempt to verify this later tonight by bypassing the ISP router and connecting directly to the eth interface on the fiber media converter. In the mean time I will move my LAMP server to the DMZ too see if that helps.

router_fw2
router_fw2931×871 36.5 KB

12

Please post the output of the command:

sudo netstat -nap | grep :80

On your server.

4 Likes
13

@matkr265 Would you also show the Firewall Advanced tab. And, the result of this command

sudo apachectl -t -D DUMP_VHOSTS
4 Likes
14

@MikeMcQ Cmd output and screen below.

I moved my LAMP server to the DMZ, still not able to connect to port 80. Must wait 6 hours until i can disconnect the router and connect my PC directly to WAN.

matias@rpi4-lamp:~$ sudo apachectl -t -D DUMP_VHOSTS
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.0.29. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server krewik.net (/etc/apache2/sites-enabled/krewik_net-le-ssl.conf:2)
         port 443 namevhost krewik.net (/etc/apache2/sites-enabled/krewik_net-le-ssl.conf:2)
                 alias www.krewik.net
         port 443 namevhost krewik.net (/etc/apache2/sites-enabled/wishlist_krewik_net-le-ssl.conf:2)
                 alias wishlist.krewik.net
*:80                   is a NameVirtualHost
         default server krewik.net (/etc/apache2/sites-enabled/krewik_net.conf:1)
         port 80 namevhost krewik.net (/etc/apache2/sites-enabled/krewik_net.conf:1)
                 alias www.krewik.net
         port 80 namevhost krewik.net (/etc/apache2/sites-enabled/wishlist_krewik_net.conf:1)
                 alias wishlist.krewik.net

router_fw_adv
router_fw_adv662×837 35.8 KB

15

Well, you have name/port overlap on both ports. The name krewik.net must only appear once for each port.

As a test, try changing the ServerName in both the wishlist VirtualHosts to be Xkrewik.net. Restart Apache and see if that helps

4 Likes
17

@MikeMcQ Changed the server name in virtual host conf files, same error message from certbot and the sites are not available on port 80.

matias@rpi4-lamp:~$ sudo apachectl -t -D DUMP_VHOSTS
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 192.168.0.29. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:443                  is a NameVirtualHost
         default server krewik.net (/etc/apache2/sites-enabled/krewik_net-le-ssl.conf:2)
         port 443 namevhost krewik.net (/etc/apache2/sites-enabled/krewik_net-le-ssl.conf:2)
                 alias www.krewik.net
         port 443 namevhost Xkrewik.net (/etc/apache2/sites-enabled/wishlist_krewik_net-le-ssl.conf:2)
                 alias wishlist.krewik.net
*:80                   is a NameVirtualHost
         default server krewik.net (/etc/apache2/sites-enabled/krewik_net.conf:1)
         port 80 namevhost krewik.net (/etc/apache2/sites-enabled/krewik_net.conf:1)
                 alias www.krewik.net
         port 80 namevhost Xkrewik.net (/etc/apache2/sites-enabled/wishlist_krewik_net.conf:1)
                 alias wishlist.krewik.net

matias@rpi4-lamp:~$ sudo /etc/init.d/apache2 restart
Restarting apache2 (via systemctl): apache2.service.
1 Like
18

Would you show contents of this:

Please use 3 backticks before and after the output like this
```
output
```

4 Likes
19

@MikeMcQ VirtualHost config output below.

matias@rpi4-lamp:~$ sudo cat /etc/apache2/sites-available/krewik_net.conf
<VirtualHost *:80>
    ServerName krewik.net
    ServerAlias www.krewik.net
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/krewik_net
    ErrorLog ${APACHE_LOG_DIR}/error.log
    CustomLog ${APACHE_LOG_DIR}/access.log combined

    RewriteEngine on
    RewriteCond %{SERVER_NAME} =krewik.net [OR]
    RewriteCond %{SERVER_NAME} =www.krewik.net
    RewriteRule ^ https://%{SERVER_NAME}%{REQUEST_URI} [END,NE,R=permanent]
</VirtualHost>
20

Please retry:

3 Likes
21

@rg305 The error persists, unfortunately.

matias@rpi4-lamp:~$ sudo certbot renew
[sudo] password for matias:
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/krewik.net.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Renewing an existing certificate for krewik.net and 2 more domains

Certbot failed to authenticate some domains (authenticator: apache). The Certificate Authority reported these problems:
  Domain: krewik.net
  Type:   connection
  Detail: 188.149.197.26: Fetching http://krewik.net/.well-known/acme-challenge/69al8avUtacDeveumze7ClGo9HPA97HSGdE-yd_4HBk: Connection refused

  Domain: www.krewik.net
  Type:   connection
  Detail: 188.149.197.26: Fetching http://www.krewik.net/.well-known/acme-challenge/IwSJLJwfYoOr0ePnBKJOU55SwpVDEbrzdPMM8gm5Y1A: Connection refused

  Domain: wishlist.krewik.net
  Type:   connection
  Detail: 188.149.197.26: Fetching http://wishlist.krewik.net/.well-known/acme-challenge/rndXPkaf7mYGtGNIz88ORJORd8hqJKOBoapn0kb018E: Connection refused

Hint: The Certificate Authority failed to verify the temporary Apache configuration changes made by Certbot. Ensure that the listed domains point to this Apache server and that it is accessible from the internet.

Failed to renew certificate krewik.net with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/krewik.net/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.