.mil engineers and operators are available to assist from our end if needs be. We can verify with .mil email addresses to confirm identity. Please feel free to reach out to us.
2 Likes
We currently believe that DNS requests from our primary vantage point are being blocked by the .mil nameservers.
From external perspectives (such as our AWS-based multi-perspective validation hosts), we can query the .mil nameservers successfully:
$ dig AAAA RAMSTEIN-NS2.AFNOC.af.mil. @132.3.48.148 +norecurse +bufsize=512
; <<>> DiG 9.16.1-Ubuntu <<>> AAAA RAMSTEIN-NS2.AFNOC.af.mil. @132.3.48.148 +norecurse +bufsize=512
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 36735
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 1, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;RAMSTEIN-NS2.AFNOC.af.mil. IN AAAA
;; AUTHORITY SECTION:
AFNOC.af.mil. 14 IN SOA gunter-ns2.AFNOC.af.mil. dnsman.us.af.mil. 2023115796 3600 360 604800 500
;; Query time: 52 msec
;; SERVER: 132.3.48.148#53(132.3.48.148)
;; WHEN: Wed Nov 29 19:42:47 UTC 2023
;; MSG SIZE rcvd: 111
However, from our primary perspective, the nameservers refuse the connection:
Getting the nameserver IPs from an external perspective:
$ dig caa cloudone.af.mil @PAC1.NIPR.mil.
; <<>> DiG 9.18.18-0ubuntu0.23.04.1-Ubuntu <<>> caa cloudone.af.mil @PAC1.NIPR.mil.
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 37501
;; flags: qr rd; QUERY: 1, ANSWER: 0, AUTHORITY: 6, ADDITIONAL: 7
;; WARNING: recursion requested but not available
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;cloudone.af.mil. IN CAA
;; AUTHORITY SECTION:
AF.MIL. 21600 IN NS YOKOTA-NS10.AFNOC.AF.MIL.
AF.MIL. 21600 IN NS RAMSTEIN-NS2.AFNOC.AF.MIL.
AF.MIL. 21600 IN NS OSAN-NS10.AFNOC.AF.MIL.
AF.MIL. 21600 IN NS HICKAM-NS2.AFNOC.AF.MIL.
AF.MIL. 21600 IN NS GUNTER-NS2.AFNOC.AF.MIL.
AF.MIL. 21600 IN NS LACKLAND-NS2.AFNOC.AF.MIL.
;; ADDITIONAL SECTION:
RAMSTEIN-NS2.AFNOC.AF.MIL. 21600 IN A 132.3.48.148
LACKLAND-NS2.AFNOC.AF.MIL. 21600 IN A 132.3.48.140
YOKOTA-NS10.AFNOC.AF.MIL. 21600 IN A 132.3.9.10
HICKAM-NS2.AFNOC.AF.MIL. 21600 IN A 132.3.48.156
GUNTER-NS2.AFNOC.AF.MIL. 21600 IN A 132.3.48.132
OSAN-NS10.AFNOC.AF.MIL. 21600 IN A 132.3.13.10
;; Query time: 91 msec
;; SERVER: 199.252.180.234#53(PAC1.NIPR.mil.) (UDP)
;; WHEN: Wed Nov 29 11:43:46 PST 2023
;; MSG SIZE rcvd: 312
Querying those IPs from an internal perspective:
$ for ip in 132.3.48.148 132.3.48.140 132.3.9.10 132.3.48.156 132.3.48.132 132.3.13.10 ; do
> dig +norecurse NS af.mil @${ip}
> done
; <<>> DiG 9.16.1-Ubuntu <<>> +norecurse NS af.mil @132.3.48.148
;; global options: +cmd
;; connection timed out; no servers could be reached
; <<>> DiG 9.16.1-Ubuntu <<>> +norecurse NS af.mil @132.3.48.140
;; global options: +cmd
;; connection timed out; no servers could be reached
; <<>> DiG 9.16.1-Ubuntu <<>> +norecurse NS af.mil @132.3.9.10
;; global options: +cmd
;; connection timed out; no servers could be reached
; <<>> DiG 9.16.1-Ubuntu <<>> +norecurse NS af.mil @132.3.48.156
;; global options: +cmd
;; connection timed out; no servers could be reached
; <<>> DiG 9.16.1-Ubuntu <<>> +norecurse NS af.mil @132.3.48.132
;; global options: +cmd
;; connection timed out; no servers could be reached
; <<>> DiG 9.16.1-Ubuntu <<>> +norecurse NS af.mil @132.3.13.10
;; global options: +cmd
;; connection timed out; no servers could be reached
The IPv4 range in use today that appears to be blocked is 23.178.112.0/24.
Our IPv6 ranges in use today are 2600:3000:2710:200::81/125, and 2600:3000:1511:200::81/125.
All of these will need to be unblocked by the .mil nameserver operators.
Disclaimer: these IP ranges are not guaranteed to be static, and should not be referred to in anyone’s firewall or other rules. This should serve to resolve the current issues, but should not be used for any other purpose. These IP ranges are also insufficient for issuance on their own, as our multi-perspective corroboration uses unpredictable IP addresses to validate challenges and check CAA.
6 Likes
Ack. Thanks, we were focused on this as a cause as well due to observed differences between v4 and v6 domains. Thank you for the current source info, we're looking into it.
2 Likes
We are currently running into issues pulling certs that seem related. This recently started happening but only for a particular domain. Is there any way to know which DNS endpoint you are using to validate so we can investigate? I have verified the hostname is valid from as many DNS servers as i can find, and the same process works for other domains on the same set of hosts.
IMPORTANT NOTES:
-
The following errors were reported by the server:
Domain: xxxxx.mil
Type: dns
Detail: DNS problem: query timed out looking up A for
xxxxx.mil; DNS problem: query timed out looking up
AAAA for xxxxx.mil
@rthomas There are several posts about problems with the .mil tld in recent days / weeks. I know staff is aware.
Your problem is more likely related to that rather than this thread which involved a custom DNS server. You might start your own thread or post on below so the .mil problems are better organized.
3 Likes
Thank you I will follow up there!
2 Likes
5 posts were split to a new topic: Spurious CAA SERVFAIL responses during finalize
That IP information has helped a lot. We are currently working to get those pulled from getting mitigated.
2 Likes
Anyone .MIL following this thread, issue is resolved. Thanks again for the support.
7 Likes
Works for me! Thanks for everyone's help!
4 Likes
I tested this out and was able to succesfully run a dry-run which was failing before. I will try generating a new cert shortly.
New cert generation is working, appears the issue is resolved.
Thank you
5 Likes
I concur. Working again for me as well.
Thanks very much!
5 Likes
Are you sure this was resolved? If this solution was just unblocking the current IPs thanks to the efforts of your fellow .mil colleagues above, this should just be considered a temporary fix as the IPs are not static and subject to change at any point (as mentioned above).
4 Likes
FWIW, as far as we could tell DNS queries worked to .mil everywhere EXCEPT the Let's Encrypt primary validation servers (as I said, I used a pile of RIPE Atlas probes and they all worked fine). I am not sure what caused the LE validation servers to be blocked (they don't normally tell low level grunts like me that but I sure would love to know what happened there) but since there was a lot of visibility on this I am hoping it won't happen again, regardless of any IP address changes on LE's side.
3 Likes
That is what I expected and why I suggest this may not actually be solved. We may have to wait until LetsEncrypt's IPs cycle again to see if the .mil TLD administrators actually addressed why the blocks happened in the first place - or if they just put a bandaid on the current situation by whitelisting those IPs.
Barring any communication to you from the domain administrators, or higher level people in your org, which details what happened and what steps were followed to address this and prevent it from happening again, I would cautiously assume that you will experience this problem again in the future.
This opinion is not specific to the .mil administrators - which I know nothing about and can not address - and entirely based on how other TLD domain admins have typically acted.
4 Likes
I mean ... yes? I don't disagree with anything you said. Since this caused such widespread effects AND we now have some contact information with the networking people, my hope is that if it does happen again we can get it resolved quickly.
3 Likes
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.