Timeout during connect (likely firewall problem)

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: test.h43.ru

I ran this command: le.pl ???

It produced this output:

2022/10/30 21:26:00 [INFO] [ Crypt::LE client v0.38 started. ]
2022/10/30 21:26:00 [INFO] Loading an account key from /home/???/letsencrypt/var/account.key
2022/10/30 21:26:00 [INFO] Loading a CSR from /home/???/letsencrypt/tmp/domain.csr.20020
2022/10/30 21:26:01 [INFO] Registering the account key
2022/10/30 21:26:01 [INFO] The key is already registered. ID: 7694677
2022/10/30 21:26:02 [INFO] Successfully saved a challenge file '/home/???/letsencrypt/WWW/.well-known/acme-challenge/aBd5bkvdAtK2iXtS87SodwyIoYUF7xuFKTy-o4HA6CI' for domain 'test.h43.ru'
2022/10/30 21:27:33 [ERROR] Domain verification results for 'test.h43.ru': error. During secondary validation: 46.255.97.130: Fetching http://test.h43.ru/.well-known/acme-challenge/aBd5bkvdAtK2iXtS87SodwyIoYUF7xuFKTy-o4HA6CI: Timeout during connect (likely firewall problem)
2022/10/30 21:27:33 [INFO] Challenge file '/home/???/letsencrypt/WWW/.well-known/acme-challenge/aBd5bkvdAtK2iXtS87SodwyIoYUF7xuFKTy-o4HA6CI' has been deleted.
2022/10/30 21:27:33 [ERROR] All verifications failed

My web server is (include version): nginx 1.22.1

The operating system my web server runs on is (include version):FreeBSD 12.3

My hosting provider, if applicable, is: ihead.ru (I myself)

I can login to a root shell on my machine (yes or no, or I don't know):yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot).

Today 2022-10-30 our system can not get ssl certificates from Let's Encrypt (now we have 9 pending tasks for different domains and different IP-networks).

This means the primary validation (from the US) succeeded, but 2 or more validations from the secondary locations, I believe all from AWS, around the globe, have failed.

From my location in Europe I can connect to your site, but e.g. Let's Debug (Let's Debug) cannot. To me, this suggests there could be regional blockades in your firewall.

3 Likes

What can we do? If it is firewall, it is not on our side. My two locations (nets: 92.39.69.224/27, 46.255.97.128/27) have same result.

Are all the h43.ru domain names yours and on this same system? (cert history here)

Because I see you most recently got a cert 11 days ago. Has anything changed on your end since then? Any comms equipment? Software upgrades?

It does not look Let's Encrypt related. Because the first test Let's Debug does is just a simple http request from its own test server and it times out. I have a test server in an AWS East Coast region and I see your site fine. I'm not sure it's a geographic based firewall but maybe just IP based

2 Likes

I used domain test.h43.ru only for test purpose (to not show domains of our customers). Nothing changed on our side. Last successful issue was yesterday, 2022-10-29 09:01:08 (MSK).

Hmmm. I don't see a cert for test.h43.ru issued at that time. Sometimes the logs get delayed but that looks more than 24H ago and should for sure be in the logs by now.

All Let's Encrypt certs are logged in the public Certificate Transparency Logs (crt.sh is one way to view them). The censys.io search also does not see any certs with that name. Were you using the Let's Encrypt staging system for that? Or the LE production system?

2 Likes

Cert was issued for another domain.

Then I don't know how that info is helpful. In any case, this looks like a comms problem reaching your system.

Let's Debug, as I said, cannot reach you from its own test server with a test request.

I tried two different sites I use for connectivity testing and neither one can see your system. (site24x7.com free website availability test and reqbin.com/curl)

I don't have any other ideas and I don't see that it is unique to LE

2 Likes

I can ask our upstream ISP, but i need IPs, from which LE makes requests to us.

LE validation request IP's may change from request to request. The IP's are not published.

Since the Let's Debug test connect fails you could use that as an example failure. It's IPv4 address is 172.104.24.29 (from DNS). You can see in its verbose display the request URL that times out from this IP

Request to: test.h43.ru/46.255.97.130, Result: [Address=46.255.97.130,Address Type=IPv4,Server=,HTTP Status=0], Issue: ANotWorking
Trace:
@0ms: Making a request to http://test.h43.ru/.well-known/acme-challenge/letsdebug-test (using initial IP 46.255.97.130)
@0ms: Dialing 46.255.97.130
@10000ms: Experienced error: context deadline exceeded

That Let's Debug test, when run to my own test server, shows in the nginx access log like this

172.104.24.29 - - [30/Oct/2022:19:44:57 +0000] "GET /.well-known/acme-challenge/letsdebug-test HTTP/1.1" 404 134 "http://(redacted-domain)/.well-known/acme-challenge/letsdebug-test" "Mozilla/5.0 (compatible; Let's Debug emulating Let's Encrypt validation server; +https://letsdebug.net)"
2 Likes

I don't see in verbose display any IP, except mine. It seems, you are not related to LE, and so can't help.

This forum is first-line support for Let's Encrypt. It is staffed mostly by volunteers of which I am one. In my experience, the Let's Encrypt staff do not get involved debugging general http comms problems.

The Let's Debug system is an open-source project with various contributors (mostly az who is a certbot developer). I am not one of the contributors. It is a tool we routinely use on this forum. See the source here. The IP address of its website is readily found in the public DNS system. That is the IP I supplied.

In addition to that I also provided two other testing tools that cannot reach your site.

Something is preventing consistent http requests from working for that domain. These failed requests are not unique to Let's Encrypt. But, LE looks to be having the same problem as these other sites.

Resolving such comms problems is not easy. I've done all I care to do. Good luck

EDIT: For LE IP info see this FAQ topic and be sure to read its multi perspective validation link topic

2 Likes

DNS challenge works. But we use http by default. And it worked all these years.

Let's Debug works on Linode net 172.104.0.0/15. This network is blocked on our side.

Somehow certificate was successfully issued (time is MSK):
2022/10/31 00:00:00 [INFO] [ Crypt::LE client v0.38 started. ]
2022/10/31 00:00:00 [INFO] Loading an account key from /home/???/letsencrypt/var/account.key
2022/10/31 00:00:00 [INFO] Loading a CSR from /home/???/letsencrypt/tmp/domain.csr.20020
2022/10/31 00:00:01 [INFO] Registering the account key
2022/10/31 00:00:02 [INFO] The key is already registered. ID: 7694677
2022/10/31 00:00:02 [INFO] Successfully saved a challenge file '/home/???/letsencrypt/WWW/.well-known/acme-challenge/wWr9Gz12Dqa47fNRdfLsnK9VISmB9xCTbUotW-XI6OY' for domain 'test.h43.ru'
2022/10/31 00:01:33 [INFO] Domain verification results for 'test.h43.ru': success.
2022/10/31 00:01:33 [INFO] Challenge file '/home/???/letsencrypt/WWW/.well-known/acme-challenge/wWr9Gz12Dqa47fNRdfLsnK9VISmB9xCTbUotW-XI6OY' has been deleted.
2022/10/31 00:01:33 [INFO] Requesting domain certificate.
2022/10/31 00:01:34 [INFO] Requesting issuer's certificate.
2022/10/31 00:01:34 [INFO] Saving the full certificate chain to /home/???/letsencrypt/tmp/domain.crt.20020.
2022/10/31 00:01:34 [INFO] The job is done, enjoy your certificate!

So i think the problem in LE maintenance. The timing 00:00:02-00:01:33 also points to this.

Please don't start your requests at midnight.

2 Likes

I understand that servers are restarting in midnight. Task was in queue several hours.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.