Our ip address was spoofed, letsencrypt servers were attacked for a month with the fake ip address. We did not make this attack, but letsencrypt sent us reply packets due to the fake tcp packets sent to letsencrypt. For this reason, we also received a high degree of reverse attack. Letsencrypt blocked our ip address, thinking that we did the attack.
After this incident last month, letsencrypt still has not unblocked the IP address. Since the IP address belongs to the shared host server, thousands of our customers cannot create SSL and are victimized. Although I forwarded the email@example.com address, there was no interest, no solution was offered.
In network pcap packet analysis, letsencrypt tcp syn request comes for HTTP verification and tcp syn+ack response goes from us. But since letsencrypt blocks the ip address, the tcp syn+ack response does not reach letsencrypt and the authentication challenge timeout error appears.
This is not a symptom of blocking on Let's Encrypts side. Let's Encrypt blocks generally mean that you cannot connect to the ACME endpoint.
This error message means that you are able to succesfully connect to the ACME endpoint, and create new orders (no block). The error complains that Let's Encrypts secondary validation servers are not able to connect to you. This is unlikely to have anything to do with Let's Encrypt.
LE's secondary validation is primarily run on AWS infrastructure, so you appear to have connectivity issues to them.
Let's Encrypt connects to you from more than a single vantage point. What you see in packet capture is likely the (succesful) request from LE's primary validation servers. The error message refers to secondary validation being unsuccesful.
The fact that primary validation is succesful is also a clear indicator that there is no blocking going on from LE's side.
Failure to connect indicates an interference. Timeout indicates a drop, not a rejection. This does not indicate where the blocking occurred. If you know Network TCP/IP Protocol flags, you can understand what I am saying. Since I have determined that this blocking is not on my side and that packets containing TCP flags are going out over my network, but letsencrypt does not respond to this packet, letsencrypt has blocked me or letsencrypt is a clear evidence that there is a blocking in the data center where letsencrypt is in the outgoing packets.
As I mentioned above, there is a drop in my detection. Letsencrypt is accessing me using 18.104.22.168/24 class for all site HTTP validations. I haven't received a request from a different class for months. On accessing 22.214.171.124 via this class, there is an incoming TCP (SYN), outgoing TCP (SYN+ACK), and there is no new TCP coming afterwards. So this means;
The outgoing access from the source IP address 126.96.36.199 to the destination IP address 188.8.131.52/24 is blocked (incoming is not blocked).
This determination is definitive and cannot be interpreted differently.
I am an advanced datacenter security/network administrator. We manage millions of ip addresses and provide DDoS protection services.
If I made a detection, letsencrypt should definitely check it.
I'm sorry, but I do not agree with your conclusion.
The reason is the "During secondary validation" part of the error message.
That piece of extra information is ONLY added to the error message when the primary endpoint succeeds, but the secondary endpoint fails.
If the primary endpoint fails, with or without failing secondary endpoints, the "During secondary validation" part is omitted.
I have checked this behaviour from the ACME validation server with a few simple iptables rules.
This leads me to believe that in your case, even with all your pcap analysis, the problem does NOT lie with the 184.108.40.206 IP address, which is the primary validation location, but with one or more of the 3 secondary validation locations.
How many validation attempts from an AWS IP address space do you see in your webserver logs?
@lestaff Can you confirm features.MultiVAFullResults is enabled in Boulder, as my above statement is corroborated with numerous (staging) validation attempts?
As stated above by @Osiris , you'll also receive up to three additional connections from (currently) random AWS addresses for secondary validation. Enough of those aren't succeeding, per your error message.
So that confirms my suspicion that the mentioning of the secondary validation failure means that the validation attempt from 220.127.116.11/24 (primary) as actually successful, despite all the pcap investigations and the problem lies with at least two of the three attempts from one of the secondary AWS locations.
I see that your announcement of 18.104.22.168/24 is via AS50673 (Serverius), a DDoS protection provider. Have you checked their flow logs to make sure they aren't blocking inbound requests to port 80 from our AWS instances (in multiple regions; the IPs vary) at the times you're attempting validation?