Verification fails - error 400 - Liquid Web and ESET

Updates

My hosting provider Liquid Web has confirmed that they are seeing the issue with other customers and they are checking further.

I can see that this particular domain is now secure, I guess LiquidWeb managed to get that working, but I don't know if the root cause was resolved or it's still a random issue.

Update

Liquid Web believes the issue is caused by ESET's Network Attack Protection module which they install by default on all of their servers, which is sometimes incorrectly banning the letsencrypt ip addresses.

They are able to issue a certificate by temporarily disabling the ESET Network Attack Protection module.

This is the cause of the Problem. Deactivate ESET Network Filter and it runs fine.

Any ideas why Letsencrypt makes 3 requests for each address? (6 including www). Perhaps that could be what is triggering ESET to think it's some kind of malicious behavior.

Let's Encrypt checks from multiple places around the Internet, to ensure that you actually control the name as seen from multiple perspectives.

Interesting, but not sure I understand.

Lets encrypt does DNS resolution on the name and resolves to an IP address, are they concerned that the same ip address will be under different control depending on where the request is coming from? I can't think of a scenario where I have control of the server behind an ip address only if the request comes from one part of the internet.

BTW and unrelated, if the name resolves to multiple addresses (e.g. load balancer, cloudflare, etc) do they check each one?

Yes, that's exactly the concern, for both the IP address of the DNS server and the IP address of the web server that's in the DNS response.

https://www.usenix.org/conference/usenixsecurity18/presentation/birge-lee

Thanks, I'm aware of the possibility of BGP hijacking, just never thought how that can be used for faking a ssl certificate.

So let's accept that the 3 requests are needed. Perhaps letsencrypt can insert a slight delay between the multiple requests to avoid triggering a firewall IDS systems.

Hi,
I have ESET installed on my web server and am facing the same problem. I spotted that LE IP addresses are blocklisted on the Spamhaus blocklist that's why the validations are failing. Eset checks any inbound IPs in the blocklist before allowing the connections in. How can I have the LE IP ranges for the inbound challenge connection so I can make an exception? I don't want to disable ESET security modules or filters to bypass the issue as this is not a good practice. Thanks

Interesting, I see a long discussion about letsencrypt servers being blocked by spamhaus here: Validation server is blacklisted

I would hope that spamhaus and letsencrypt can work together on a solution to this. Perhaps they can arrange for a secure private and automated method to let spamhaus know which addresses lets encrypt is using at any given time. Perhaps spamhaus can create a few domains, generate certificates on a daily basis and see where the verification request comes from.

Is ESET able to whitelist any URI with /.well-known/acme-challenge/ ?

Even make it optional for people choosing to use ACME cert acquisition?

I checked thru their interface, in the network protection module can whitelist ip addresses, but not urls.

Maybe ask them to add such a feature :slight_smile:

ESET and Letsencrypt are big organizations that can talk to each other, change is not going to come from the little guy caught in the middle.

And technically, the block is likely happening at a level before they even know the url (Not sure, just a guess)

I wouldn't expect a rapid solution in any case. I can't speak for ISRG (Let's Encrypt) officially I am just a guy on the internet.

But, last year we dealt with a lot of issues for people using Palo Alto Networks brand firewalls. Palo Alto added a new feature which, by default, blocked /.well-known/acme-challenge requests. We don't see that much anymore but we saw problems for months. I assume Palo Alto changed the default. Or, at least now better inform their customers what to do.

Always good to have actual customers complain to their vendor about problems.

Any product that blocks web traffic based on the inclusion of an IP in an email blocklist is not suitable for use in a production webserver environment. I'd put that defective ESET product in the trash and let them know why you are removing it.

I am not the customer, LiquidWeb is the customer, and I'm sure they are in touch with ESET.

Their software works and protects the server. Except for this issue I have not had any complaints.

Let's Encrypt is a lot smaller than you think it is.

Most servers without their software work just fine.

That makes little sense to me.
How can anyone control the rate at which the Internet makes requests onto your site.
Is the IDS set to accept only two requests per time slice of a certain type and then drop all others?
The multiple similar requests should be treated separately [from one another].
If ten people decide to click on a link to your site - all at the same time - would you NOT want to serve them all?

In short: ESET can't be blocking based on the request rate.

Good point, I agree with you.