Correct. I was hoping it might help isolate the region or origin IPs that were not getting through but it did not.
2 Likes
Of course I understand that I can not see in web servers logs requests from the let's encrypt server which could not connect to my server. This is actually the root problem!
It would be helpful if each one make let's debug request and show here own 3 web server log lines with IP addresses.
May be we could identify the the LS server which could not connect to mine.
The LE Server IPs vary. I don't get the same repeated IPs each time.
A consistent thing is you only see 1 or 2 requests before the rest are blocked. This seems very much like a DDoS firewall setting on your end. Sometimes called "adaptive" or "smart" firewalls.
If there was a general outbound comms problem from a pool of LE Servers we would be getting flooded with similar errors on this forum. The most likely culprit is a firewall of some kind in your hosting service or server that is not yet understood. Have you asked your hosting service about this yet? You said this had been working for some time so something must have changed.
3 Likes
I wish it was so.
But NO firewall blocking.
The real problem is that Lets Encrypt does not show in own log from which IP it failed to connect.
I do not understand why LE force to guess which their server is not able to connect (
Of course. They do not have known open cases concerning this issue. And they do not block any external connections.
Probably to not increase any attack vector which might compromise the challenge. The whole reason for multiple vantage points is to lower the possibility of a network attacker hijacking the challenge to begin with. See Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt for more info. It wouldn't be helpful if the validation server would spit out the IP addresses the hijacker would need to reroute.
5 Likes
If you post your IP and domain I can look at logs tell you what IPs we used that failed to validate, but you didn't include that in your original post so it's hard to help.
We routinely rotate the IPs especially for the secondary validation as they're from AWS.
5 Likes
I sent you PM with data.
Finally yesterday night I got successfull cert renew without any changes at the server and with all firewall blocking rules active.
But I do want to understand what it was to try prevent it in the future.
I'm glad the problem resolved itself. When the validators scaled up and back down, new instances with new IPs come online. It seems likely that one of our IPs was blocked somewhere, possibly by Hetzner. I'm sorry for the trouble this caused you.
3 Likes
Which IP was blocked?
Could you send me it?
I have confirmed we saw errors from validator instances in both AWS's us-east-2 and eu-central-1 regions to your IP address.
Sorry, I don't easily have the IP address of the instances and it would take a bit of work to correlate the different logs to find what the external IPs of the instances are.
3 Likes
Will you further investigate the connectivity issues between LE verification servers and Hetzner DE network?
It seems Hetzner is not so small EU hosting provider to ignore such issues.
The problem doesn't appear to have been anything on our end, so there's not much more for us to investigate.
The overall success rates of validation are high, so it seems unlikely this is widespread. I have looked at some Hetzner IP ranges and there are many successes from them.
Unfortunately we can't debug every intermittent network connection, as there is always some low levels of failures due to networks, misconfigured firewalls, etc.
4 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.