IP addresses seem to be blocked from getting new certicates

Help
1

I have a few servers at he.net with IP address range of

65.49.22.224 /27
216.218.249.160 /27
66.220.11.0 /27

Getting list of URLs for API Requesting new nonce for client communication Account already registered. Continuing. Sending registration to letsencrypt server Sending signed request to https://acme-v02.api.letsencrypt.org/acme/new-acct Account: https://acme-v02.api.letsencrypt.org/acme/acct/60823609 Starting certificate generation process for domains Requesting challenge for calibaja.ippbxsupport.com Sending signed request to https://acme-v02.api.letsencrypt.org/acme/new-order Sending signed request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/8005792078 Got challenge token for calibaja.ippbxsupport.com Token for calibaja.ippbxsupport.com saved at /var/www/html/.well-known/acme-challenge/MpCPaz41__NYaruOoB9Aj-mI9fW35ZAeV2oT20PJwAk and should be available at http://calibaja.ippbxsupport.com/.well-known/acme-challenge/MpCPaz41__NYaruOoB9Aj-mI9fW35ZAeV2oT20PJwAk Sending request to challenge Sending signed request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/8005792078/hXZv1Q Verification pending, sleeping 1s Sending signed request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/8005792078/hXZv1Q Verification pending, sleeping 1s Sending signed request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/8005792078/hXZv1Q Verification pending, sleeping 1s Sending signed request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/8005792078/hXZv1Q Verification pending, sleeping 1s Sending signed request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/8005792078/hXZv1Q Verification pending, sleeping 1s

There was an error updating the certificate: Verification timed out

This is happening to all servers which have gotten a certificate. This has been going on for about three months now

I can ping to acme-v02.api.letsencrypt.org just fine.

To test, I put up a similar type server on Digital Ocean and gets certificate fine.

One of the domain names is calibaja.ippbxsupport.com

Thanks!

2

I don't think we've seen enough of the logs to know exactly what is failing.
That said, I think your server is reaching acme-v02 and it may be the secondary HTTP challenges that are failing - most likely due to GeoLocation blocking or an IPS or IP block lists in place by your systems (or HE).

[&2* readers: Get involved; Be heard. It starts with: if you read something you like, then like it :heart:]

3

The validation server error is present in the /authz-v3/ URI from the log:

During secondary validation: Fetching http://calibaja.ippbxsupport.com/.well-known/acme-challenge/MpCPaz41__NYaruOoB9Aj-mI9fW35ZAeV2oT20PJwAk: Timeout during connect (likely firewall problem)

I can confirm that from my location: just a time out to IP 216.218.249.183. Nothing seems to be responding: no ping, no open port 21 or 22, no open HTTP or HTTPS.

So I'm with the error message: please check any and all firewalls and/or routers for correct behaviour and get port 80 open somehow, because it's closed at the moment.

4

There is a firewall which is blocking any traffic which is not permitted, however, the following URL's are exempted.

outbound1.letsencrypt.org
outbound2.letsencrypt.org

I have also disabled the firewall as a test to check if anything from there is blocking.
I can reach port 80 with, or without the firewall enabled.

As for logs, I am just not seeing any logs for Let's Encrypt.

Thanks for the responses

5

Whitelisting those hosts won't work. Let's Encrypt validates from AWS as well (which is the "secondary" part of the "During secondary validation" error): https://letsencrypt.org/docs/faq/#what-ip-addresses-does-let-s-encrypt-use-to-validate-my-web-server

6

I have the same problem. Http challenge, timeout during connect, likely firewall problem

I think your problem is due, letsencrypt is using AWS for verifying http-01 challenge, which means when there is problem from your network with AWS. Your domain cannot be verified

I've asked and confirmed to AWS support. That there is indeed a problem with connection between my networks and AWS which results in dropped tcp packet. And i'm still waiting the issue to resolve, even there is no ETA

From my perspective, i think the http-01 challenge bot should try verify the domain using more diversified cloud services, ex: Azure or Google Cloud, as the current implementation really depends on the stability of AWS network connection

I think the problem will become more and more common, due yesterday i have 3 ips that AWS have trouble with, and today i have 2 ips more that AWS have trouble with.

The problem maybe not be from AWS itself, as there is a lot of hops before reaching AWS. But my point is, it just take one bad network hop, to make letsencrypt unusable from some network.

Quoting from aws support:
"With the size and scale of AWS, traffic from different IPs can take different routes within the AWS network and a routing issue could cause a packet drop which looks very similar to a firewall that simply drops denied packets"

7

Just adding to what @_az posted....

About four months ago, LetsEncrypt removed the remaining exceptions and rolled out "unpredictable" verification to everyone. See: ACME v1/v2: Validating challenges from multiple network vantage points - #3 by cpu

8

I guess I will get with the network guy and see about shutting down the firewall just for a moment to see if we can get a successful response from the servers.

9

You don't have to shut it down completely.
[that sounds very extreme]
You only need to allow HTTP to reach your server to pass HTTP validation.

[&2* readers: Get involved; Be heard. It starts with: if you read something you like, then like it :heart:]

10

Just need to quickly verify if it is a firewall issue. If so, then that can be easily taken care of.