Sudo certbot renew failed

This is really weird the server is not busy at all. Is it possible to somehow get banned on Lets encrypt or something? I have no idea why it doesn't work anymore either.

If you were banned, you'd see another error (429 or just a timeout). This is something interfering with requests from LE to you. So, maybe your ISP is playing dirty.

1 Like

I agree. @EDV you should ask your ISP if they are blocking port 80. Some ISPs do that.

I can reach your site fine with HTTPS (port 443) but not with HTTP (port 80)

curl -I -m10 http://inet.bbs-ahaus.de/.well-known/acme-challenge/ChallengeFile
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received

curl -I -m10 https://inet.bbs-ahaus.de/.well-known/acme-challenge/ChallengeFile
HTTP/1.1 404 Not Found
Date: Thu, 14 Apr 2022 12:50:19 GMT
Server: Apache/2.4.29 (Ubuntu)
Content-Type: text/html; charset=iso-8859-1
2 Likes

That could be theoretical. The isp currently has a few problems here. The internet is sometimes there and sometimes gone. But the internet is currently stable and the error keeps coming and that was the case yesterday. In addition, the website can also be reached, only Lets encrypt has problems

It looks like the story of the firewall from Michigan, but this time we actually get a 502 error (who the hell is issuing that error? @EDV check /var/log/apache2/error.log maybe?)

% curl -I -m10 http://inet.bbs-ahaus.de/.well-known/acme-challenge
HTTP/1.1 301 Moved Permanently
Date: Thu, 14 Apr 2022 12:56:26 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://inet.bbs-ahaus.de/.well-known/acme-challenge
Content-Type: text/html; charset=iso-8859-1
Connection: keep-alive

% curl -I -m10 http://inet.bbs-ahaus.de/.well-known/acme-challenge
HTTP/1.1 301 Moved Permanently
Date: Thu, 14 Apr 2022 12:56:29 GMT
Server: Apache/2.4.29 (Ubuntu)
Location: https://inet.bbs-ahaus.de/.well-known/acme-challenge
Content-Type: text/html; charset=iso-8859-1
Connection: keep-alive

% curl -I -m10 http://inet.bbs-ahaus.de/.well-known/acme-challenge/
HTTP/1.1 502 Connection timed out
Date: Thu, 14 Apr 2022 12:56:41 GMT
Connection: close
Cache-Control: no-store
Content-Type: text/html
Content-Language: en
Content-Length: 219

% curl -I -m10 http://inet.bbs-ahaus.de/.well-known/acme-challenge/
HTTP/1.1 502 Connection timed out
Date: Thu, 14 Apr 2022 12:56:50 GMT
Connection: close
Cache-Control: no-store
Content-Type: text/html
Content-Language: en
Content-Length: 219

%
2 Likes

Yeah, I was just going to add to my post that "index page" requests reached fine so not ISP.

I don't get a 502 but that can happen when a proxy fails. But, I don't get a 502 just a normal curl timeout even with 120s timeout.

Maybe the hosting site is blocking inbound requests to that path?

@EDV We are making curl requests to your site. Let's Encrypt is not involved. We are using a URL path like LE would but any path to your site should give some response. It is just this certain path that does not respond right.

2 Likes

It almost looks like this. The website works without any problems, only when calling the command certbot renew there is a timeout
Actually it can almost only be the firewall or the ISP because another server shows the same error.

Maybe it is the firewall. The logs /var/log/apache2/error.log look like that:

[Wed Apr 13 06:25:01.430562 2022] [mpm_prefork:notice] [pid 106026] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Apr 13 06:25:01.430575 2022] [core:notice] [pid 106026] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 13 06:58:15.017856 2022] [mpm_prefork:notice] [pid 106026] AH00171: Graceful restart requested, doing restart
[Wed Apr 13 06:58:15.053405 2022] [mpm_prefork:notice] [pid 106026] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Apr 13 06:58:15.053416 2022] [core:notice] [pid 106026] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 13 06:59:50.079824 2022] [mpm_prefork:notice] [pid 106026] AH00171: Graceful restart requested, doing restart
[Wed Apr 13 06:59:50.109109 2022] [mpm_prefork:notice] [pid 106026] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Apr 13 06:59:50.109121 2022] [core:notice] [pid 106026] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 13 07:07:54.550139 2022] [mpm_prefork:notice] [pid 106026] AH00169: caught SIGTERM, shutting down
[Wed Apr 13 07:08:11.230219 2022] [mpm_prefork:notice] [pid 1292] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Apr 13 07:08:11.232310 2022] [core:notice] [pid 1292] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 13 07:08:39.156816 2022] [mpm_prefork:notice] [pid 1292] AH00171: Graceful restart requested, doing restart
[Wed Apr 13 07:08:39.270117 2022] [mpm_prefork:notice] [pid 1292] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Apr 13 07:08:39.270129 2022] [core:notice] [pid 1292] AH00094: Command line: '/usr/sbin/apache2'
[Wed Apr 13 07:10:14.106822 2022] [mpm_prefork:notice] [pid 1292] AH00171: Graceful restart requested, doing restart
[Wed Apr 13 07:10:14.133349 2022] [mpm_prefork:notice] [pid 1292] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Wed Apr 13 07:10:14.133360 2022] [core:notice] [pid 1292] AH00094: Command line: '/usr/sbin/apache2'
[Thu Apr 14 11:28:57.120620 2022] [mpm_prefork:notice] [pid 1308] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Thu Apr 14 11:28:57.123634 2022] [core:notice] [pid 1308] AH00094: Command line: '/usr/sbin/apache2'
[Thu Apr 14 11:30:00.137235 2022] [mpm_prefork:notice] [pid 1308] AH00171: Graceful restart requested, doing restart
[Thu Apr 14 11:30:00.173431 2022] [mpm_prefork:notice] [pid 1308] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Thu Apr 14 11:30:00.173444 2022] [core:notice] [pid 1308] AH00094: Command line: '/usr/sbin/apache2'
[Thu Apr 14 11:31:35.468925 2022] [mpm_prefork:notice] [pid 1308] AH00171: Graceful restart requested, doing restart
[Thu Apr 14 11:31:35.506033 2022] [mpm_prefork:notice] [pid 1308] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Thu Apr 14 11:31:35.506046 2022] [core:notice] [pid 1308] AH00094: Command line: '/usr/sbin/apache2'
[Thu Apr 14 12:47:33.596834 2022] [mpm_prefork:notice] [pid 1308] AH00171: Graceful restart requested, doing restart
[Thu Apr 14 12:47:33.637631 2022] [mpm_prefork:notice] [pid 1308] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Thu Apr 14 12:47:33.637643 2022] [core:notice] [pid 1308] AH00094: Command line: '/usr/sbin/apache2'
[Thu Apr 14 12:49:08.528482 2022] [mpm_prefork:notice] [pid 1308] AH00171: Graceful restart requested, doing restart
[Thu Apr 14 12:49:08.563343 2022] [mpm_prefork:notice] [pid 1308] AH00163: Apache/2.4.29 (Ubuntu) OpenSSL/1.1.1 configured -- resuming normal operations
[Thu Apr 14 12:49:08.563354 2022] [core:notice] [pid 1308] AH00094: Command line: '/usr/sbin/apache2'

Well, I read a 502 as "something, somewhere" is responding to that http request.

I don't get why, though.

502 is "bad gateway"

504 is "gateway timeout"

1 Like

@9peppe How long of a timeout do you tolerate to see a 502? I am up to 300s and don't get one.

Maybe that's a clue it's a geography based firewall since you are Italy and I am US? Agree a well-behaved proxy would respond properly but we don't know what it is yet.

Agree someone must send 502. @edv can you tell us more about your hosting service? Are you shared hosting. Do you have any kind of CDN feature in it?

Can you show us result of this command:

sudo apachectl -S
2 Likes

It could be geography indeed.

1 Like

Hello @EDV,
I belong to the Hochschule of Music in Nürnberg and I am having the same problem as you, the creation of certificates with CertBot, which has always worked for all servers, has suddenly stopped working. Do you also have your servers hosted at the Technische Hochshule in Nürnberg? Because it seems to be a repeating pattern that didn't happen before.

Hello @EDV,

Ask where you host the servers to create a rule for that server in the acme-protocol application firewall. This has worked for me :).

Best regards.

2 Likes

Im here in Germany
The results of the Command:
VirtualHost configuration:
*:443 inet.bbs-ahaus.de (/etc/apache2/sites-enabled/000-default-le-ssl.conf:2)
*:80 is a NameVirtualHost
default server inet.bbs-ahaus.de (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost inet.bbs-ahaus.de (/etc/apache2/sites-enabled/000-default.conf:1)
port 80 namevhost inet.bbs-ahaus.de (/etc/apache2/sites-enabled/wordpress.conf:1)
ServerRoot: "/etc/apache2"
Main DocumentRoot: "/var/www/html"
Main ErrorLog: "/var/log/apache2/error.log"
Mutex ssl-stapling: using_defaults
Mutex ssl-cache: using_defaults
Mutex default: dir="/var/run/apache2/" mechanism=default
Mutex mpm-accept: using_defaults
Mutex watchdog-callback: using_defaults
Mutex rewrite-map: using_defaults
Mutex ssl-stapling-refresh: using_defaults
PidFile: "/var/run/apache2/apache2.pid"
Define: DUMP_VHOSTS
Define: DUMP_RUN_CFG
User: name="www-data" id=33
Group: name="www-data" id=33

Hello,
the servers are hosted here by us and nothing has changed behind our firewall there.
I don't find any blocked packets or the like in our firewall. Do you know exactly what kind of rule was created?

You have the same name defined in two places. That should not be.
Can you show the content of each of those files?
Please put 3 backticks before and after the output of each file like this:
Name 000-default.conf
```
contents of /etc/apache2/sites-enabled/000-default.conf
```
Name wordpress.conf
```
contents /etc/apache2/sites-enabled/wordpress.conf
```

3 Likes

Holy moly you saved my weekend. It was actually a firewall rule. I had everything correct from the Lan side but the acme protocol was missing on the Wan side. this application was then blocked
Thank you all for the support and sorry for the stupid mistake :grinning:

5 Likes

Have a nice weekend everyone and thanks again for the support :blush:

3 Likes
3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.