Timeout during connect (likely firewall problem)

Hello,

I am experiencing the exact same issue with a couple of certs of mine. I did packet captures and validated from 2 different VPS sites that access to the HTTP challenge is working 100% but I am met with a similar message:

Timeout during connect (likely firewall problem)

My apache/nginx logs show me 200s, my packet captures are replying back to sources properly, and my VPS test curls while the certbot was running for the challenge to be verified worked(I was able to hit the exact URL and get the challenge from my EC2 instance and VPS instance).

At this point I am starting to believe that there is some issue with the Lets Encrypt servers.

Your belief is incorrect. Please open your own topic in the Help category and complete the template so that you may receive individual attention for your issue.

6 Likes

Hello @certdude, welcome to the Let's Encrypt community. :slightly_smiling_face:

As already stated, Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

Thank you for assisting us in helping YOU!

6 Likes

How many? Let's Encrypt validates from multiple vantage points around the world. See also: Let's Encrypt is adding two new remote perspectives for domain validation - #3 by lenaunderwood. (Note that 4 remotes equals to 5 requests total, including the primary "non-remote" location.)

5 Likes


Looks like 3, if what you are saying @Osiris it sounds like 2/5 are failing?

1 Like

Looks like it, indeed.

Your error message probably also includes "secondary" somewhere looking at the IP addresses.

Do you have some kind of geoblocking active perhaps?

3 Likes

Yes secondary failure indeed:

The server could not connect to the client to verify the domain :: During secondary validation:

My firewall does geo block a bunch of countries, is there a list a full list of CIDRs/servers that Let's Encrypt auths from? The three I see are all from AWS.

Nope, the countries/regions can change at any time, so you shouldn't count on anything even if there were a list.

For now that's the case, but could change at any time in the future.

3 Likes

That is an interesting log because those 3 are an http 200 which is a successful reponse by your server. Given the cert request failed it means one or more of those replies did not reach the Let's Encrypt server. This is not likely an inbound firewall failure. It is perhaps an outbound firewall or, more likely, some kind of comms routing problem. But, at least your server wanted to reply correctly.

The LE servers only need "enough" responses to determine success or failure. At the moment 3 successes is enough (likely to be 4 fairly soon).

3 Likes

@MikeMcQ Currently 4 remote validation points are in situ, if you include the primary there are 5 requests. If only 1 remote validation point is allowed to fail, 3 requests, 1 from the primary and 2 from remote locations, would fail the challenge.

So I'm not sure if this is an outbound problem, especially if OP says there's geoblocking in play.

2 Likes

But I thought they are still tolerating 2 failures of 5.

The IP starting with '23.' is not AWS it is the primary data center using the Cloudflare Magic Transit IP.

3 Likes

It was stated there were only a few days between steps, and the increment to 4 remote validation points was 5 days ago already. While I'm not sure of course, chances are the quorum has been increased already.

2 Likes

The HTTP challenge quorum change will definitely catch a lot of people off guard, there are tons of orgs out there that run firewalls which block specific countries. My suggestion to Let's Encrypt team would be to publish a whitelist of subnets(at minimum) to keep folks functional in their renewal mechanism.

I can likely work with my firewall to try and catch which IP addresses are being blocked during the HTTP challenge phase, to create my own whitelist, but I doubt many will have the patience or willingness to go that far.

That would defeat the purpose of multiple remote validation locations.

You could use the dns-01 challenge perhaps if http-01/tls-alpn-01 doesn't suit your situation.

4 Likes

@certdude Looks like you are only seeing US based server farms. Let's Encrypt has long validated from outside the US and is doing so even more now.

If the quorum is now 4 then I agree with @Osiris this is more likely you just blocking inbound non-US requests.

EDIT: I have since gotten confirmation the quorum is now 4 of the 5 validations.

5 Likes

7 posts were split to a new topic: Multiple validation perspectives and geolocation blocking