You are using at least one validation endpoint in an Amazon datacenter. Since these are often misused for automated and massive expoitation of websites, we have decided to limit or block connections from known amazon ip ranges. To get a better chance of excluding your endpoint(s) from these limits, it could be very helpful to track back the endpoint which connection timed out. Could this be included in the authorization object on failure?
EDIT: This is meant for http-01 authorization only.
I’m not sure exactly what you’re asking. That when a validation from a remote validation authority fails that its marked as being from a remote VA?
We are not likely to include any information about the source IP of the validation. We explicitly do not advertise the IP space we perform validations from because it may change at any point
At what level are you performing this block of AWS IP space? Since you’re specifically interested in HTTP-01 is it possible that you could make an exception for requests to the the /.well-known/acme-challenge path?
I ask for another field in the challenge structure that clearly states which remote endpoint (address) failed to do the test.
The problem is, that we sometimes get an invalid authorization due to a timeout fetching the challenge-token (urn:acme:error:connection). After re-initializing the authentication, these errors go away. There are some related posts which have to do with ipv6 preference, but this can clearly be ruled out.
If the origin for the specific request would be included in the object, one may exactly check for offending firewall rules.
The block of the known AWS addresses is done on the ip layer, so excluding these based on specific http requests is not possible.
Yes, I mean the validation authority. I don't ask for sharing any address, I just ask to add another field to the authorization object (that can be queried via the auth-uri) that declares which VA failed to do the test.
My follow-up question would be how you would want the failing VA to be identified if not by source IP? E.g. We can update the authorization with a problem that adds a note that it was "VA #2" that wasn't satisfied by the response, but how does that help you troubleshoot? Isn't it sufficient to know that the VA failed and there is a problem to address?
Of course, the VA has to be given by source ip address. I cannot understand why this information should be kept secret at all, because these addresses can also be recorded in access logs, if the challenge request hits the web server.
Ok, I think my confusion here stems from your earlier comment, "I don’t ask for sharing any address". It sounds like you are asking for us to share the source address.
That's true but in the future its possible not every VA will contact your server, we may use a subset. More importantly I feel like publishing the addresses explicitly has the air of official process and will encourage a dependence on that information. Folks like yourself will use it to write ingress firewall rules. If we change in the future those rules will break, we'll have to take on a more burdensome announcement/change management process. Leaving this undocumented gives us freedom to change things at will. We want to encourage server operators to make their challenge response servers accessible to Let's Encrypt from wherever we decide to validate.
@bytecamp Have you looked into the DNS-01 challenge instead of HTTP-01/TLS-SNI-01? The DNS-01 challenge doesn't require an inbound HTTP/TLS connection and would work much better within the constraints of your firewall design.
I just asked for adding the address to the failed object, not publishing a list of all VA addresses officially. I fully agree with you that these addresses don't have to be documented.
Lets close this now, if we run into another issue with failed validations, I will open another ticket.