I was referring to the LE [source] DNS systems - being on AWS.
2 Likes
I see. I didn't know that. (Net that I'm surprised.)
Does .mil have something against AWS?
Having dealt with similar issues in the past ... sometimes list of "attacking" IP addresses are distributed DoD-wide and are put into place across large swaths of the DoD, and low-level grunts like me aren't aware until issues like these come up. That is just a guess as to what the problem is here.
I would love to hear any suggestions as to how to progress further. If we could get unboundtest.com like results from any of the LE validation servers we could at least figure out where the problem lies.
2 Likes
That is a very good guess.
That OR some sort of DNS DoS device that is tightened a bit tooooo tightly - and is seeing the multiple/simultaneous requests from an IP as an attack.
4 Likes
I am going to tack on to this thread, I am seeing issues with multiple .army.mil sites
We are currently running into issues pulling certs that seem related. Is there any way to know which DNS endpoint you are using to validate so we can investigate? I have verified the hostname is valid from as many DNS servers as i can find, and the same process works for other domains on the same set of hosts.
IMPORTANT NOTES:
The following errors were reported by the server:
Domain: army.mil
Type: dns
Detail: DNS problem: query timed out looking up A for
army.mil; DNS problem: query timed out looking up
AAAA for army.mil
1 Like
To chime in here ... I have been contacted by people at DISA who are looking into this, but they told me that a HUGE obstacle is not having any idea as to the source IP address of the queries. I don't really understand why anyone from Let's Encrypt is willing to provide this information (and I do understand that this is volunteer support, but I know that Let's Encrypt staff members do read these forums), but that seems to be the reality.
HOWEVER ... I decided to investigate this issue a different way. I looked in our DNS server's logs back a few weeks ago when renewals worked. I see a query for one of our ACME records from the IP address 23.178.112.102, which reverse-resolves to the name outbound1c.letsencrypt.org. Seems likely that this is one of the addresses used by the Let's Encrypt validation service.
(To followup to myself ... I didn't realize those queries were in mixed case, so no wonder my original 'grep' didn't find anything. I also see queries from 18.226.34.243 and 18.216.48.63 which are inside AWS)
1 Like
Hi @kenh1
What IP addresses does Let’s Encrypt use to validate my web server?
Let’s Encrypt does not publish a list of IP addresses we use to validate,
and these IP addresses may change at any time.
And yes the Let’s Encrypt staff does read and participate in this community. 
It's supposed to be like that. You're not supposed to respond differently to Let's Encrypt than you do to the rest of the internet. Validation gets less reliable if you do that.
2 Likes
Hi @philipdumont, @kenh1, @rthomas, et al. -- we're actively looking into this. We have a few hypotheses, but no confirmed root cause (neither within nor outside our own infrastructure) or prospective fix yet.
I'm declaring this thread to be the One True Thread for this issue; we'll provide updates here if and when we have them. Please point others to this thread if they are also encountering errors with domains that use .mil's nameservers.
3 Likes
Also https://www.abetterinternet.org/ which https://letsencrypt.org/ is part of is a small nonprofit organization with limited staff and other resources. They do a tremendous job with what they have.
2 Likes
To that I say, Amen!
2 Likes
How do we get some of the let's encrypt staff on with some of the .mil DNS operators? We're getting slammed on this issue over here so maybe a conference call and some targeted testing would be helpful?
3 Likes
Hi @philipdumont, @kenh1, @rthomas, et al. -- we're actively looking into this. We have a few hypotheses, but no confirmed root cause (neither within nor outside our own infrastructure) or prospective fix yet.
Appreciate the heads up. If you need to arrange any contact with networking folks at DISA please drop me a private message and I will facilitate it.
2 Likes
If people are looking for workarounds, it may be worth trying some other free CAs that support ACME. It should be pretty easy to switch to them, though I don't know what policies might be in place on the military/government side for what CAs you're allowed to use. (I'm kind of surprised that they'd be using Let's Encrypt honestly; doesn't the US government run its own CAs?)
1 Like
Not a publicly trusted one, I hope. Do you want do you want the NSA to have carte blanche for TLS MITM attacks?
1 Like
No not really. Maybe you aren't familiar with the absolute crush of DDoS volume on the DNS infrastructure but you absolutely change responses based on certain behaviors. Knowing sources, even for a short time gives you the ability to debug a problem, rather than blanket assuming everything is fine or the same. If that wasn't the case AWS, Cloudflare, etc wouldn't provide API methods to retrieve their source IPs, regions, and other pertinent information used for debugging. Plus, regardless of what an enterprise does internally there are many external factors that could come into play and impact traffic before it reaches the boundary. Source IP information is necessary to debug this. Or at the least highly preferred. And BTW, if a trusted source publishes it's IPs (maybe with SSL encrypted even!?!) then we can trust that source and allow that traffic to be un-affected if it is being treated as malicious.
Keeping DNS online and working is hard. No doubt about that.
Filters based in behaviour shouldn't catch Let's Encrypt. Filters based on source literally break every assumption we're making.
1 Like
Yeah, apparently I was mistaken. Though the Federal PKI is trusted in some systems, apparently it's not enough for widespread use.
3 Likes
This is getting a lot of heat within DoD. Need to get any information and updates you have.
Let's Encrypt engineering is looking into this. We'd appreciate it if anyone not involved avoids posting in this thread. Thank you.
6 Likes