Timeout during connect (likely firewall problem) but request present in access.log

Hi,
I'm having some new issues with renewing an old certificate that I did not notice had expired.
First I had a problem with my DNS provider but after I updated the acme.sh client I use to issue the certificate the DNS part worked. But then it comes back to validating with a http response, but here it fails with a Timeout, the odd part is that I see the request in my nginx logs returning 200.

I tried checking with Let's Debug and that shows the same main connection issue, but the weird part is that the debug HTTP check passes.

@0ms: Making a request to http://noshi.se/.well-known/acme-challenge/letsdebug-test (using initial IP 89.160.9.126)
@0ms: Dialing 89.160.9.126
@201ms: Server response: HTTP 200 OK 

I can also access the port myself from a totally different network

$ curl -Ii http://noshi.se/.well-known/acme-challenge/letsdebug-test
HTTP/1.1 200 OK
Server: nginx/1.17.7
Date: Wed, 27 Apr 2022 17:38:34 GMT
Content-Type: text/plain
Content-Length: 6
Last-Modified: Wed, 27 Apr 2022 17:02:53 GMT
Connection: keep-alive
ETag: "6269773d-6"
X-Frame-Options: DENY
X-Content-Type-Options: nosniff
X-XSS-Protection: 1; mode=block
Accept-Ranges: bytes

My domain is:
noshi.se, *.noshi.se

I ran this command:
"/root/.acme.sh"/acme.sh --cron --home "/root/.acme.sh"

It produced this output:

[Wed Apr 27 18:16:19 CEST 2022] ===Starting cron===
[Wed Apr 27 18:16:19 CEST 2022] Renew: 'noshi.se'
[Wed Apr 27 18:16:20 CEST 2022] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Wed Apr 27 18:16:20 CEST 2022] Multi domain='DNS:noshi.se,DNS:*.noshi.se'
[Wed Apr 27 18:16:20 CEST 2022] Getting domain auth token for each domain
[Wed Apr 27 18:16:23 CEST 2022] Getting webroot for domain='noshi.se'
[Wed Apr 27 18:16:23 CEST 2022] Getting webroot for domain='*.noshi.se'
[Wed Apr 27 18:16:24 CEST 2022] Verifying: noshi.se
[Wed Apr 27 18:16:24 CEST 2022] Pending, The CA is processing your order, please just wait. (1/30)
[Wed Apr 27 18:16:27 CEST 2022] Pending, The CA is processing your order, please just wait. (2/30)
[Wed Apr 27 18:16:30 CEST 2022] Pending, The CA is processing your order, please just wait. (3/30)
[Wed Apr 27 18:16:33 CEST 2022] Pending, The CA is processing your order, please just wait. (4/30)
[Wed Apr 27 18:16:35 CEST 2022] noshi.se:Verify error:89.160.9.126: Fetching http://noshi.se/.well-known/acme-challenge/Ij1Wid5mwXR10IoTElCPgWorBdW1wCXtiS4WBY7PAIs: Timeout during connect (likely firewall problem)
[Wed Apr 27 18:16:36 CEST 2022] Please check log file for more details: /root/.acme.sh/acme.sh.log
[Wed Apr 27 18:16:37 CEST 2022] Error renew noshi.se_ecc.
[Wed Apr 27 18:16:37 CEST 2022] ===End cron===

My web server is (include version):
nginx version: nginx/1.17.7 (x86_64-pc-linux-gnu)

nginx access logs with matching challange id:

# cat /var/log/nginx/access.log | grep Ij1Wid5mwXR10IoTElCPgWorBdW1
wCXtiS4WBY7PAIs
3.69.241.162 - - [27/Apr/2022:18:16:25 +0200] "GET /.well-known/acme-challenge/Ij1Wid5mwXR10IoTElCPgWorBdW1wCXtiS4WBY7PAIs HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
3.144.218.183 - - [27/Apr/2022:18:16:25 +0200] "GET /.well-known/acme-challenge/Ij1Wid5mwXR10IoTElCPgWorBdW1wCXtiS4WBY7PAIs HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"
54.191.192.47 - - [27/Apr/2022:18:16:26 +0200] "GET /.well-known/acme-challenge/Ij1Wid5mwXR10IoTElCPgWorBdW1wCXtiS4WBY7PAIs HTTP/1.1" 200 87 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

The operating system my web server runs on is (include version):
Turris OS version 5.3.8 (OpenWRT)

I can login to a root shell on my machine (yes or no, or I don't know):
yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

# ./acme.sh --version
https://github.com/acmesh-official/acme.sh
v3.0.3

I'm a bit unsure on how to debug this further, any suggestions on what more I could try?

You have 3 requests in your access log. As of today, you should be seeing 4 requests. The primary validation server is being prevented from connecting. Check that 66.133.109.36 isn't blocked in some firewall.

4 Likes

Thank you, this was indeed the issue, the 66.133.109.36 hint was what I needed.

I have the Turris Sentinel dynamic firewall activated and not really at home with iptables but

# iptables -S | grep DROP
-P FORWARD DROP
-N zone_wan_src_DROP
-A syn_flood -m comment --comment "!fw3" -j DROP
-A zone_wan_dest_ACCEPT -o eth2 -m conntrack --ctstate INVALID -m comment --comment "!fw3: Prevent NAT leakage" -j DROP
-A zone_wan_forward -m set --match-set turris-sn-dynfw-block src -m conntrack --ctstate NEW -m comment --comment "!sentinel: dynamic firewall block" -j zone_wan_src_DROP
-A zone_wan_input -m set --match-set turris-sn-dynfw-block src -m mark ! --mark 0x10/0x10 -m conntrack --ctstate NEW -m comment --comment "!sentinel: dynamic firewall block" -j zone_wan_src_DROP
-A zone_wan_src_DROP -m comment --comment "!sentinel: fwlogs" -j NFLOG --nflog-group 1914 --nflog-threshold 32
-A zone_wan_src_DROP -m comment --comment "!sentinel" -j DROP

and the ipset list turris-sn-dynfw-blockdid contain 66.133.109.36

ipset list turris-sn-dynfw-block | grep 66.133.109.
66.133.109.36

After manually deleting it with

ipset del turris-sn-dynfw-block 66.133.109.36

The request was let through and seems to work, Let's Debug

This post on the Turris forum, that I had missed..., also helped Dynamic firewall and Let's encrypt - #3 by Skippi - General discussion - Turris forum

Hopefully this helps someone else too, but what is left is why the Sentinel firewall blocks the request, but that is not for this forum and I changed to DNS challenges so hopefully not an issue for me anymore.

Thanks again @_az

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.