Timeout during connect (likely firewall problem)

Help
1

Hi guys, I got a similar problem. Although, I use --standalone only. The server connects to my server and certbot replies properly. The result is Timeout during connect (likely firewall problem) , though.

Here is the tcpdump from my server. Hope it helps.

34.213.106.112.45538 > 95.216.177.125.80: Flags [P.], cksum 0x7c5c (correct), seq 4273540643:4273540914, ack 3099952526, win 211, options [nop,nop,TS val 4068796029 ecr 1314244930], length 271: HTTP, length: 271
	GET /.well-known/acme-challenge/pTjzRh9TuLlyLwYocXfnEMMG_XHZXtG0jSbP2Qzsxsc HTTP/1.1
	Host: console.prokams.com
	User-Agent: Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)
	Accept: */*
	Accept-Encoding: gzip
	Connection: close
	
	0x0000:  4500 0143 ebc0 4000 2606 c959 22d5 6a70  E..C..@.&..Y".jp
	0x0010:  5fd8 b17d b1e2 0050 feb9 0e23 b8c5 858e  _..}...P...#....
	0x0020:  8018 00d3 7c5c 0000 0101 080a f284 e67d  ....|\.........}
	0x0030:  4e55 c942 4745 5420 2f2e 7765 6c6c 2d6b  NU.BGET./.well-k
	0x0040:  6e6f 776e 2f61 636d 652d 6368 616c 6c65  nown/acme-challe
	0x0050:  6e67 652f 7054 6a7a 5268 3954 754c 6c79  nge/pTjzRh9TuLly
	0x0060:  4c77 596f 6358 666e 454d 4d47 5f58 485a  LwYocXfnEMMG_XHZ
	0x0070:  5874 4730 6a53 6250 3251 7a73 7873 6320  XtG0jSbP2Qzsxsc.
	0x0080:  4854 5450 2f31 2e31 0d0a 486f 7374 3a20  HTTP/1.1..Host:.
	0x0090:  636f 6e73 6f6c 652e 7072 6f6b 616d 732e  console.prokams.
	0x00a0:  636f 6d0d 0a55 7365 722d 4167 656e 743a  com..User-Agent:
	0x00b0:  204d 6f7a 696c 6c61 2f35 2e30 2028 636f  .Mozilla/5.0.(co
	0x00c0:  6d70 6174 6962 6c65 3b20 4c65 7427 7320  mpatible;.Let's.
	0x00d0:  456e 6372 7970 7420 7661 6c69 6461 7469  Encrypt.validati
	0x00e0:  6f6e 2073 6572 7665 723b 202b 6874 7470  on.server;.+http
	0x00f0:  733a 2f2f 7777 772e 6c65 7473 656e 6372  s://www.letsencr
	0x0100:  7970 742e 6f72 6729 0d0a 4163 6365 7074  ypt.org)..Accept
	0x0110:  3a20 2a2f 2a0d 0a41 6363 6570 742d 456e  :.*/*..Accept-En
	0x0120:  636f 6469 6e67 3a20 677a 6970 0d0a 436f  coding:.gzip..Co
	0x0130:  6e6e 6563 7469 6f6e 3a20 636c 6f73 650d  nnection:.close.
	0x0140:  0a0d 0a                                  ...
13:48:56.608060 IP (tos 0x0, ttl 63, id 19700, offset 0, flags [DF], proto TCP (6), length 52)
    95.216.177.125.80 > 34.213.106.112.45538: Flags [.], cksum 0xe7f1 (correct), seq 3099952526, ack 4273540914, win 235, options [nop,nop,TS val 1314244986 ecr 4068796029], length 0
	0x0000:  4500 0034 4cf4 4000 3f06 5035 5fd8 b17d  E..4L.@.?.P5_..}
	0x0010:  22d5 6a70 0050 b1e2 b8c5 858e feb9 0f32  ".jp.P.........2
	0x0020:  8010 00eb e7f1 0000 0101 080a 4e55 c97a  ............NU.z
	0x0030:  f284 e67d                                ...}
13:48:56.612100 IP (tos 0x0, ttl 63, id 19701, offset 0, flags [DF], proto TCP (6), length 231)
    95.216.177.125.80 > 34.213.106.112.45538: Flags [P.], cksum 0xb864 (correct), seq 3099952526:3099952705, ack 4273540914, win 235, options [nop,nop,TS val 1314244987 ecr 4068796029], length 179: HTTP, length: 179
	HTTP/1.0 200 OK
	Server: BaseHTTP/0.3 Python/2.7.13
	Date: Mon, 31 Dec 2018 12:48:56 GMT
	
	pTjzRh9TuLlyLwYocXfnEMMG_XHZXtG0jSbP2Qzsxsc.05mAbe6c8pHNjCnk36P7twBKqy_qox2hiXbQO8MdPJ4[!http]
	0x0000:  4500 00e7 4cf5 4000 3f06 4f81 5fd8 b17d  E...L.@.?.O._..}
	0x0010:  22d5 6a70 0050 b1e2 b8c5 858e feb9 0f32  ".jp.P.........2
	0x0020:  8018 00eb b864 0000 0101 080a 4e55 c97b  .....d......NU.{
	0x0030:  f284 e67d 4854 5450 2f31 2e30 2032 3030  ...}HTTP/1.0.200
	0x0040:  204f 4b0d 0a53 6572 7665 723a 2042 6173  .OK..Server:.Bas
	0x0050:  6548 5454 502f 302e 3320 5079 7468 6f6e  eHTTP/0.3.Python
	0x0060:  2f32 2e37 2e31 330d 0a44 6174 653a 204d  /2.7.13..Date:.M
	0x0070:  6f6e 2c20 3331 2044 6563 2032 3031 3820  on,.31.Dec.2018.
	0x0080:  3132 3a34 383a 3536 2047 4d54 0d0a 0d0a  12:48:56.GMT....
	0x0090:  7054 6a7a 5268 3954 754c 6c79 4c77 596f  pTjzRh9TuLlyLwYo
	0x00a0:  6358 666e 454d 4d47 5f58 485a 5874 4730  cXfnEMMG_XHZXtG0
	0x00b0:  6a53 6250 3251 7a73 7873 632e 3035 6d41  jSbP2Qzsxsc.05mA
	0x00c0:  6265 3663 3870 484e 6a43 6e6b 3336 5037  be6c8pHNjCnk36P7
	0x00d0:  7477 424b 7179 5f71 6f78 3268 6958 6251  twBKqy_qox2hiXbQ
	0x00e0:  4f38 4d64 504a 34                        O8MdPJ4
2

Hi. Can you provide more information?

When using the staging environment, Let’s Encrypt makes multiple requests from different IP addresses. If you only got one, it’s evidence there is a problem.

(Plus, that request isn’t from the ISP used by the production environment.)

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

3

Thanks! First of all my answers.

My domain is: console.prokams.com

I ran this command: certbot certonly --standalone -d console.prokams.com --dry-run

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for console.prokams.com
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. console.prokams.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://console.prokams.com/.well-known/acme-challenge/4MnAQzgpkQho3GVAwHAOHjrDqRnc0FMgKfqz1futfV4: Timeout during connect (likely firewall problem)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: console.prokams.com
   Type:   connection
   Detail: Fetching
   http://console.prokams.com/.well-known/acme-challenge/4MnAQzgpkQho3GVAwHAOHjrDqRnc0FMgKfqz1futfV4:
   Timeout during connect (likely firewall problem)

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

My web server is (include version): custom forwarding proxy

The operating system my web server runs on is (include version): Debian 9

My hosting provider, if applicable, is: VMs on Hetzner

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

My server is a multiple-instances forwarding proxy. Two IPs connect to my server: 13.58.30.69 and 34.213.106.112. All this worked a month ago. I’ve got 9 working certificates into my server system and I made no changes within last 2 months. Just yesterday I tried to add a new FQDN and Timeout problem occurred. As I already mention, certbot gets the request and sends the response. The dump I provided was captured on the perimeter server. All domains sitting on my system work and are accessible. If you need some more information, please, don’t hesitate and ask for it.

Oh, certbot version is 0.10.2. Debian 9 doesn’t offer higher.

Thank you in advance!

4

There was at least one more IP.

5

All attempts contain 2 different IPs. I tried to run certbot renew, everything failed. Another IP 64.71.168.194. And there was another IP 77.7.9.23 but I guess it was you :slight_smile: it asked for favicon as well.

6

All attempts came from 3 IPs, but one of them timed out.

Can you get an mtr or traceroute to outbound1.letsencrypt.org or outbound2.letsencrypt.org?

7 
traceroute to outbound1.letsencrypt.org (66.133.109.36), 30 hops max, 60 byte packets
 1  172.31.1.1 (172.31.1.1)  0.433 ms  0.402 ms  0.385 ms
 2  10619.your-cloud.host (95.216.128.238)  0.369 ms  0.352 ms  0.345 ms
 3  * * *
 4  spine2.cloud1.hel1.hetzner.com (88.198.252.49)  0.887 ms  0.908 ms spine1.cloud1.hel1.hetzner.com (88.198.248.249)  0.840 ms
 5  core32.hel1.hetzner.com (88.198.249.93)  0.442 ms  0.416 ms  0.425 ms
 6  juniper4.dc1.hel1.hetzner.com (213.239.224.25)  0.583 ms juniper4.dc1.hel1.hetzner.com (213.239.224.37)  0.258 ms juniper4.dc1.hel1.hetzner.com (213.239.224.25)  0.289 ms
 7  hls-b1-link.telia.net (213.248.66.76)  1.359 ms  1.345 ms  1.315 ms
 8  ae-6.bar1.Helsinki1.Level3.net (4.68.74.141)  27.834 ms  27.862 ms  27.852 ms
 9  4.69.137.41 (4.69.137.41)  166.541 ms  166.509 ms  166.455 ms
10  VIAWEST-INT.bar2.SaltLakeCity1.Level3.net (4.35.172.34)  196.618 ms  196.488 ms  196.678 ms
11  be21.bbrt02.slc04.viawest.net (66.51.1.97)  197.387 ms  197.417 ms  197.462 ms
12  be151.crrt01.slc07.viawest.net (66.51.9.130)  196.011 ms  196.128 ms  196.016 ms
13  vl62.aggm01.slc07.viawest.net (66.133.111.218)  195.706 ms  195.663 ms  196.071 ms
14  * * *
15  * * *
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *


traceroute to outbound2.letsencrypt.org (64.78.149.164), 30 hops max, 60 byte packets
 1  172.31.1.1 (172.31.1.1)  0.266 ms  0.252 ms  0.243 ms
 2  10619.your-cloud.host (95.216.128.238)  0.234 ms  0.231 ms  0.222 ms
 3  * * *
 4  spine1.cloud1.hel1.hetzner.com (88.198.248.249)  0.697 ms spine2.cloud1.hel1.hetzner.com (88.198.252.49)  0.865 ms spine1.cloud1.hel1.hetzner.com (88.198.248.249)  1.091 ms
 5  core31.hel1.hetzner.com (88.198.249.89)  0.415 ms core32.hel1.hetzner.com (88.198.249.93)  14.943 ms  14.934 ms
 6  juniper4.dc1.hel1.hetzner.com (213.239.224.25)  0.300 ms juniper4.dc1.hel1.hetzner.com (213.239.224.37)  0.311 ms  0.345 ms
 7  hls-b1-link.telia.net (213.248.66.76)  1.396 ms  1.456 ms  1.421 ms
 8  ae-6.bar1.Helsinki1.Level3.net (4.68.74.141)  27.791 ms  27.823 ms  27.846 ms
 9  ae-2-3502.ear2.Denver1.Level3.net (4.69.207.53)  149.607 ms * *
10  4.34.59.186 (4.34.59.186)  148.677 ms  148.653 ms  148.643 ms
11  be29.crrt02.den03.viawest.net (66.51.2.125)  149.240 ms  149.233 ms  149.095 ms
12  be39.bbrt02.den03.viawest.net (66.51.5.69)  150.065 ms  149.820 ms  149.818 ms
13  be104.bbrt01.den05.viawest.net (66.51.5.101)  149.325 ms  149.449 ms  149.474 ms
14  be26.crrt01.den05.viawest.net (66.51.5.154)  150.169 ms  150.154 ms  150.122 ms
15  vl62.aggm01.den05.viawest.net (66.51.0.214)  149.215 ms  149.137 ms  149.344 ms
16  * * *
17  * * *
18  * * *
19  * * *
20  * * *
21  * * *
22  * * *
23  * * *
24  * * *
25  * * *
26  * * *
27  * * *
28  * * *
29  * * *
30  * * *
8

That looks correct. :confused: (The last hop doesn’t respond to UDP traceroute, though it does respond to ICMP echo.)

9 
traceroute to outbound1.letsencrypt.org (66.133.109.36), 30 hops max, 60 byte packets
 1  172.31.1.1 (172.31.1.1)  0.331 ms  0.320 ms  0.317 ms
 2  10619.your-cloud.host (95.216.128.238)  0.315 ms  0.324 ms  0.324 ms
 3  * * *
 4  spine1.cloud1.hel1.hetzner.com (88.198.248.249)  0.997 ms  0.997 ms  0.996 ms
 5  core31.hel1.hetzner.com (88.198.249.89)  0.630 ms  0.726 ms  0.740 ms
 6  juniper4.dc1.hel1.hetzner.com (213.239.224.37)  0.411 ms  0.346 ms  0.335 ms
 7  hls-b1-link.telia.net (213.248.66.76)  1.315 ms  1.346 ms  1.344 ms
 8  ae-6.bar1.Helsinki1.Level3.net (4.68.74.141)  27.929 ms  27.905 ms  27.903 ms
 9  4.69.137.41 (4.69.137.41)  165.652 ms  165.675 ms  165.674 ms
10  VIAWEST-INT.bar2.SaltLakeCity1.Level3.net (4.35.172.34)  196.719 ms  196.764 ms  196.684 ms
11  be21.bbrt02.slc04.viawest.net (66.51.1.97)  197.107 ms  197.107 ms  197.119 ms
12  be151.crrt01.slc07.viawest.net (66.51.9.130)  196.064 ms  196.048 ms  196.088 ms
13  vl62.aggm01.slc07.viawest.net (66.133.111.218)  195.716 ms  195.750 ms  195.805 ms
14  outbound1.letsencrypt.org (66.133.109.36)  196.089 ms  196.088 ms  196.086 ms
10

ICMP works. What would be the next step? I can try to launch certbot directly from some machine (skipping the forwarding proxy). It will take little longer to flush DNS caches out.

11

I don’t know.

Maybe your system or Hetzner have blocked Let’s Encrypt’s main IPs after identifying them as weird automated traffic?

12

No, that’s not very likely. What IPs are dedicated to this service? Maybe they were listed on Zen. I use Zen for the traffic filtering.

13

[local RBL] IP address 34.213.106.112 validated (good guy).
[local RBL] IP address 13.58.30.69 validated (good guy).
[local RBL] IP address 66.133.109.36 validated (bad guy).

Do you recognize these IPs, please? That is a log from my RBL checks.

Output from dig

; <<>> DiG 9.10.3-P4-Debian <<>> 36.109.133.66.zen.spamhaus.org
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 54914
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;36.109.133.66.zen.spamhaus.org.	IN	A

;; ANSWER SECTION:
36.109.133.66.zen.spamhaus.org.	577 IN	A	127.0.0.4

;; Query time: 0 msec
;; SERVER: 213.133.100.100#53(213.133.100.100)
;; WHEN: Mon Dec 31 16:56:04 CET 2018
;; MSG SIZE  rcvd: 75

127.0.0.4 means that this IP has been recognized as an exploit provider.

14

Yes, 66.133.109.36 is outbound1.letsencrypt.org. You have to unblock it.

Well, they're lying. (Probably.)

https://www.spamhaus.org/query/ip/66.133.109.36

https://www.abuseat.org/lookup.cgi?ip=66.133.109.36

It keeps getting listed because sometimes people try to get Let's Encrypt certificates for botnet C&C domains, so Let's Encrypt connects to the domain to validate it -- just like it connects to your site -- and they incorrectly say that everything that connects to the bad domain must be an infected victim.

15

that’s not good. Can you somehow explain to Spamhaus that you don’t provide any exploiting activities? Man, that can ruin a lot of services.

Well, at least, we found the problem. My perimeter server does not provide any whitelist feature, so it will take time I can test it and confirm if it works. Anyway, I’m pretty much sure this is a problem. I think you can close this issue.

Thank you very much and Happy New Year!
And don’t get too drunk tonight :slight_smile:

16

We could ask for an update on this issue from @lestaff but I hope nobody there replies until after the holiday. :slight_smile:

17

We’ve reached out to Spamhaus but because of their scale and automation, they may not be able to make an exception for us. Allowing any IP to access /.well-known/acme-challenge - or using the DNS-01 challenge instead - may be your best bet.

2 Likes
18

Thank you gentlemen. It looks like something we have to cope with. I understand Spamhaus, I’m not very happy as well for putting some exceptions into the mechanism which supposed to be secure.

Anyway, I’m glad that the cause of this issue was discovered and explained. I’ve created a temporary workaround (what means that it will last forever) and my system is proceeding to work as planned.

BTW, thousands thumbs up for your approach. You are very good at showing to the world how the support should work. Thank you again!

2 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.