Can't renew certificates on subdomain server

nen777w · August 6, 2021, 8:38am

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: community.codijy.com

I ran this command: sudo certbot renew

It produced this output:

root@community:~# certbot renew
Saving debug log to /var/log/letsencrypt/letsencrypt.log

Processing /etc/letsencrypt/renewal/community.codijy.com.conf

Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for community.codijy.com
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (community.codijy.com) from /etc/letsencrypt/renewal/community.codijy.com.conf produced an unexpected error: Failed authorization procedure. community.codijy.com (http-01): urn:ietf:params:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://community.codijy.com/.well-known/acme-challenge/wwZJ_JsZr0SvyUoi8p4XFDUxWqfp_MvZz-hIGKzj_6k: Timeout during connect (likely firewall problem). Skipping.
All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/community.codijy.com/fullchain.pem (failure)

All renewal attempts failed. The following certs could not be renewed:
/etc/letsencrypt/live/community.codijy.com/fullchain.pem (failure)

1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: community.codijy.com
Type: connection
Detail: Fetching
http://community.codijy.com/.well-known/acme-challenge/wwZJ_JsZr0SvyUoi8p4XFDUxWqfp_MvZz-hIGKzj_6k:
Timeout during connect (likely firewall problem)

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address. Additionally, please check that
your computer has a publicly routable IP address and that no
firewalls are preventing the server from communicating with the
client. If you're using the webroot plugin, you should also verify
that you are serving files from the webroot path you provided.

My web server is (include version):

root@community:~# apache2 --version
[Fri Aug 06 04:31:46.971468 2021] [core:warn] [pid 27815] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot

The operating system my web server runs on is (include version):

root@community:~# uname -a
Linux community.codijy.com 5.4.0-80-generic #90~18.04.1-Ubuntu SMP Tue Jul 13 19:40:02 UTC 2021 x86_64 x86_64 x86_64 GNU/Linu

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

ISPConfig

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

root@community:~# certbot --version
certbot 0.27.0

Additional information:
I have a main server codijy.com which is located on 162.243.0.185
And subdomain community.codijy.com which is located on 45.140.121.163

Probably I should open a specific port's on 45.140.121.163 ?

jens_hb · August 6, 2021, 11:21am

yes, i think you have to open port 80 and 443, i get a timeout on port 80

rg305 · August 6, 2021, 9:36pm

Hi @nen777w, and welcome to the LE community forum

I see that your cert expired recently.
Has anything changed since your last cert was issued?
Did your ISP start blocking port 80?
Did you change the Apache configuration?
On port 443, I see:
Server: Apache/2.4.29 (Ubuntu)
Please show the outputs of:
sudo netstat -pant | grep -Ei 'apache|80|443'
sudo apachectl -S

nen777w · August 7, 2021, 9:27pm

No, I do not make any changes in the apache configuration.
And of course, my ISP does not block 80 and 443 ports.

Here is the commands output:

root@community:~# sudo netstat -pant | grep -Ei 'apache|80|443'
tcp6 0 0 :::8080 :::* LISTEN 2336/apache2
tcp6 0 0 :::80 :::* LISTEN 2336/apache2
tcp6 0 0 :::8081 :::* LISTEN 2336/apache2
tcp6 0 0 :::443 :::* LISTEN 2336/apache2
tcp6 0 0 192.168.1.176:443 192.241.216.156:48740 SYN_RECV -
tcp6 0 0 192.168.1.176:8080 192.168.1.229:31809 ESTABLISHED 2340/apache2
tcp6 0 0 192.168.1.176:443 192.168.1.1:28232 TIME_WAIT -
root@community:~# sudo apachectl -S
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig.conf:73
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using community.codijy.com. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:8081 community.codijy.com (/etc/apache2/sites-enabled/000-apps.vhost:9)
*:8080 community.codijy.com (/etc/apache2/sites-enabled/000-ispconfig.vhost:9)
*:80 is a NameVirtualHost
default server community.codijy.com (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost community.codijy.com (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost community.codijy.com (/etc/apache2/sites-enabled/100-community.codijy.com.vhost:7)
*:443 community.codijy.com (/etc/apache2/sites-enabled/100-community.codijy.com.vhost:199)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex fcgid-pipe: using_defaults
Mutex authdigest-opaque: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex fcgid-proctbl: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33

nen777w · August 7, 2021, 9:31pm

Hm...
I just checked here: https://portchecker.co/ port 80 and found that it's really closed.
I do not understand how it happens because in my router configuration this port is open and forwarded to the community.codijy.com server. Also as a 443.

rg305 · August 7, 2021, 9:38pm

There is a name:port overlap seen in the configuration:

nen777w:

port 80 namevhost [community.codijy.com](http://community.codijy.com) (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost [community.codijy.com](http://community.codijy.com) (/etc/apache2/sites-enabled/100-community.codijy.com.vhost:7)

nen777w · August 8, 2021, 8:19am

I change the port for the default host but it does not help...

root@community:~# sudo apachectl -S
AH00548: NameVirtualHost has no effect and will be removed in the next release /etc/apache2/sites-enabled/000-ispconfig.conf:73
AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using community.codijy.com. Set the 'ServerName' directive globally to suppress this message
VirtualHost configuration:
*:8081 community.codijy.com (/etc/apache2/sites-enabled/000-apps.vhost:9)
*:81 community.codijy.com (/etc/apache2/sites-enabled/000-default.conf:1)
*:8080 community.codijy.com (/etc/apache2/sites-enabled/000-ispconfig.vhost:9)
*:80 community.codijy.com (/etc/apache2/sites-enabled/100-community.codijy.com.vhost:7)
*:443 community.codijy.com (/etc/apache2/sites-enabled/100-community.codijy.com.vhost:199)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex authdigest-opaque: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
Mutex authdigest-client: using_defaults
Mutex fcgid-proctbl: using_defaults
Mutex ssl-stapling: using_defaults
Mutex proxy: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex fcgid-pipe: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
Define: ENABLE_USR_LIB_CGI_BIN
User: name="www-data" id=33
Group: name="www-data" id=33

jens_hb · August 8, 2021, 11:53am

I think the error is in the apache configuration.
I don't see any obvious error, but virtualhost are name-based or IP-based. no different ports are required.
revise the configuration so that it can be reached via port 80 community.codijy.com.
Example from me for ssl only port 80:
<VirtualHost *: 80>
ServerName community.codijy.com
Permanent redirect / https://community.codijy.com

rg305 · August 8, 2021, 8:07pm

Ok so the syntax appears to have been corrected.
Let's dot every eye and cross every tea.
Starting with...

check the current Internet IP(s) with:
curl -4 ifconfig.co
curl -6 ifconfig.co

nen777w · August 13, 2021, 9:17am

curl -4 ifconfig.co
45.140.121.163
curl -6 ifconfig.co
curl: (7) Couldn't connect to server

rg305 · August 13, 2021, 2:36pm

You're definitely in the right place:

Name:    community.codijy.com
Address: 45.140.121.163

Is there Router/NAT device that port forwards incoming connections?

You even mentioned:

Has that been verified?

nen777w · August 13, 2021, 2:59pm

Yes.

[edit: reduced output for security]

rg305 · August 13, 2021, 3:01pm

Ok let's verify that even further, with the output of:
sudo ifconfig | grep -Ei 'add|inet'

nen777w · August 13, 2021, 4:06pm

root@community:~# sudo ifconfig | grep -Ei 'add|inet'
inet 192.168.1.176 netmask 255.255.255.0 broadcast 192.168.1.255
inet6 fe80::c5ae:9994:df31:7045 prefixlen 64 scopeid 0x20
inet 127.0.0.1 netmask 255.0.0.0
inet6 ::1 prefixlen 128 scopeid 0x10

BTW, Thank you for trying to help me understand the problem!

rg305 · August 13, 2021, 4:10pm

Well that seems to match the Port Forwarding.

Let's see if Apache is actually connected and listening there, with:
netstat -pant | grep -Ei 'apache|:80|:443'

Port 80 fails:

curl -Iki http://community.codijy.com
curl: (56) Recv failure: Connection reset by peer

Port 443 works:

curl -Iki https://community.codijy.com
HTTP/1.1 200 OK
Date: Fri, 13 Aug 2021 16:13:35 GMT
Server: Apache/2.4.29 (Ubuntu)

Does your router use port 80 (for itself)?
When I telnet to 45.140.121.163 on port 80, it connects.
It isn't "speaking" HTTP, nor HTTPS, but something is there and it "picks up the call".

nen777w · August 13, 2021, 6:20pm

root@community:~# netstat -pant | grep -Ei 'apache|:80|:443'
tcp6       0      0 :::443                  :::*                    LISTEN      2281/apache2
tcp6       0      0 :::8080                 :::*                    LISTEN      2281/apache2
tcp6       0      0 :::80                   :::*                    LISTEN      2281/apache2
tcp6       0      0 :::8081                 :::*                    LISTEN      2281/apache2
tcp6       0      0 192.168.1.176:443       146.88.240.4:49086      SYN_RECV    -
tcp6       0      0 192.168.1.176:443       192.168.1.1:5943        ESTABLISHED 2284/apache2
tcp6       0      0 192.168.1.176:443       192.168.1.1:1027        TIME_WAIT   -

Does your router use port 80 (for itself)?

It was used for local access, but I reassigned it in settings before I trying to update the certificate.

If I understand correctly the problem in the following:

if scan port 80 from the internet it looks closed
but again, if connect from the internet to this port for example:
telnet 45.140.121.163 80 - some application is answered

From my local network the command: telnet 45.140.121.163 80
connect to a server, but not answered.
But If send some symbols to it, after some period in answered with disconnection:

HTTP/1.1 301 Moved Permanently
Date: Fri, 13 Aug 2021 18:14:17 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https:///error/400.html
Content-Length: 317
Connection: close
Content-Type: text/html; charset=iso-8859-1

... and so on...

rg305 · August 13, 2021, 6:42pm

45.140.121.163:80 should be directly connected to your internal system at 192.168.1.176:80
[443 is working - we won't worry about that anymore]

Whatever you see locally at: http://192.168.1.176
Should match exactly what is seen externally at: http://45.140.121.163

The problem is highly likely to be within the router.

Review the documentation for it and ensure it is configured to allow port 80 in to your server.
Reboot the router if necessary (ensure all setting and IPs return as expected after reboot).
If problem persists, speak with, or email, your ISP about the router and this port problem.

system · September 12, 2021, 6:42pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.