DNS lookup failure for .mil

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: aodms.af.mil

I ran this command:

PS C:\Windows\system32> & "$WinAcmeFolder\wacs.exe" `
>> --source iis `
>> --siteid 2 `
>> --host core-integration.affsa.aws.solidstatescientific.com,integration.affsa.aws.solidstatescientific.com,aodms.af.mil `
>> --emailaddress [affsa.aws.mbx@solidstatescientific.com](mailto:affsa.aws.mbx@solidstatescientific.com) `
>> --installation iis `
>> --installationsiteid 2 `
>> --accepttos

It produced this output:

A simple Windows ACMEv2 client (WACS)
Software version 2.1.19.1142 (release, pluggable, standalone, 64-bit)
Connecting to [https://acme-v02.api.letsencrypt.org/.](https://acme-v02.api.letsencrypt.org/)..
Scheduled task not configured yet
Please report issues at https://github.com/win-acme/win-acme
Running in mode: Unattended
Source generated using plugin IIS: [core-integration.affsa.aws.solidstatescientific.com](http://core-integration.affsa.aws.solidstatescientific.com/) and 2 alternatives

[[core-integration.affsa.aws.solidstatescientific.com](http://core-integration.affsa.aws.solidstatescientific.com/)] Cached authorization result: valid
[[integration.affsa.aws.solidstatescientific.com](http://integration.affsa.aws.solidstatescientific.com/)] Cached authorization result: valid
[[aodms.af.mil](http://aodms.af.mil/)] Authorizing...
[[aodms.af.mil](http://aodms.af.mil/)] Authorizing using http-01 validation (SelfHosting)
[[aodms.af.mil](http://aodms.af.mil/)] Authorization result: invalid
[[aodms.af.mil](http://aodms.af.mil/)] {
"type": "urn:ietf:params:acme:error:dns",
"detail": "DNS problem: query timed out looking up A for [aodms.af.mil](http://aodms.af.mil/); DNS problem: query timed out looking up AAAA for [aodms.af.mil](http://aodms.af.mil/)",
"status": 400
}
Create certificate failed: [[aodms.af.mil](http://aodms.af.mil/)] Validation failed
- No certificate generated

My web server is (include version): IIS 10

The operating system my web server runs on is (include version): Windows Server 2016

My hosting provider, if applicable, is: N/A

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): wacs version 2.1.19.1142

A few more details, which may or may not be relevant -- I don't know...

aodms.af.mil is a relatively new FQDN -- a few weeks old.

It is an A record, resolving to two IPs.

At first, one of the two IPs was correct, one was incorrect. We asked that the incorrect one be fixed, and it was.

We have previously successfully obtained a cert with all three FQDNs on it (see --host argument of above command). That was when the FQDN was serving one incorrect IP.

Ever since the incorrect IP was corrected, all attempts to get a cert that included the aodms.af.mil FQDN have failed with the DNS error. I don't know why. Everywhere I've tried to do a lookup of that FQDN -- AFNET and public internet -- has returned correct results:

phil-pc> nslookup aodms.af.mil
Server:		192.168.240.1
Address:	192.168.240.1#53

Non-authoritative answer:
Name:	aodms.af.mil
Address: 15.200.5.186
Name:	aodms.af.mil
Address: 15.200.45.183

Thanks in advance for any help you can give..

2 Likes
2

There is a recent problem where the Let's Encrypt servers aren't able to do any DNS lookups for anything under the entire .mil TLD. See these other reported problems: Certificate not getting issued due to DNS lookup timeout on A and AAAA records and DNS timeout from Let's Encrypt servers (the second issue is one I reported; that domain is NOT in .mil but the nameservers for it are).

Unfortunately, I do not have a solution; my (admittedly limited) tests show that DNS lookups for the affected domains seem to work everywhere EXCEPT from the Let's Encrypt validation servers. I'm unclear if this is a software problem on Let's Encrypt's side, a firewall block put in at a high level by DISA, or something else. But it's not just you.

3 Likes
3

Fascinating. Odd, though, that it was working for a little while before it broke.

Well, the fact that it's not just me is certainly a comfort. :-p

Thanks.

p.s., The cert for https://af.mil is a Let's Encrypt cert. That seems to work. Go figure.

1 Like
5

FWIW, my renewals were working for more than a year. And the people at www.af.mil (I'm assuming that's what you mean, there does not seem to be an A record for https://af.mil) got their certificate on November 1st, when things were still working; if you check www.af.mil now using Let's Debug you will find that it also reports a DNS timeout.

2 Likes
6

Oho. I did notice that www.af.mil's cert was last renewed the first of this month. But I was not paying enough attention to realize that this problem -- in general, not just for us -- started so recently.

Yes, I meant www.af.mil. I blame Chrome. If you point it to example.com, and there's no response there, it tries www.example.com, and doesn't bother making the added "www." visible in the address bar. Well, not until you actually try to edit the address bar, which I hadn't.

And thanks for the tip about https://letsdebug.net! That's handy.

2 Likes
7

It's weird that the SOA record isn't the same throughout:
DNS Propagation Checker - Global DNS Testing Tool (whatsmydns.net)
I see like three different values:

2023114885
2023114889
2023114896
2 Likes
8

If you have any juice with higher level networking people at your organization, maybe you could submit this issue up the chain? Sadly, I do not have so much.

9

I am perfectly willing to believe that there is something wrong with the DNS for everything under .mil. But I cannot figure out what that is! It seems to work fine everywhere EXCEPT for Let's Encrypt.

2 Likes
10

Well, dnsviz is not too happy either. Mostly with the akamai edge parts of the DNS tree. Still, it's beyond my skills to understand why unboundtest.com works fine but the LE validation servers fail. Frustrating for sure.

https://dnsviz.net/d/www.af.mil/dnssec/

4 Likes
11

Nope. My squeeze doesn't yield all that much juice.

My best hope is that when www.af.mil's cert gets to needing to be renewed, and it can't, maybe that will be big/important enough for something to happen. I hope it doen't take that long, though.

2 Likes
12

Well, dnsviz is not too happy either.

My reading of that output is those are just warnings and things should still work (and if you are using dns-01 you shouldn't be hitting akamai). And yes, it is incredibly frustrating that unboundtest.com works fine! Is my understanding correct that letsdebug.net uses the LE infrastructure to run the test validation?

13

My best hope is that when www.af.mil's cert gets to needing to be renewed, and it can't, maybe that will be big/important enough for something to happen.

If it helps at all one of the other people who reported this problem run some of the big Marines sites (e.g., https://www.10thmarines.marines.mil/). Those renewals are already failing and a bunch of those certificates expire on December 10th.

1 Like
14

Yes, usually, but some of those warnings look concerning like the wrong glue records. And, the warning about other records with a CNAME just point to sloppy construction. But, it is beyond my skills to say definitively what these all mean or their impact on LE validation.

No, you will. The whole DNS tree is searched and CNAMEs are followed. You can see this in the unboundtest.com results

Let's Debug does its own curl and DNS lookups for initial tests. After that it uses LE Staging environ for a test. Let's Debug own lookups work fine and only the LE Staging get SERVFAIL. Let's Debug does not use LE production for testing.

4 Likes
15

Well, we asked the folks who stood up our aodms.af.mil FQDN to have a look. They looked. And all they found is exactly what we found: just about every DNS server you ask gives a right answer for just about every .mil address you ask it about. Conclusion: "It must be Let's Encrypt that's broken. Or the one DNS server that Let's Encrypt is asking. What DNS server is Let's Encrypt using, anyway?"

I have to admit, it's kinda hard to argue with that conclusion. (At least for me, who am no DNS expert.) Even if DISA (or whoever) was blocking DNS lookups for some FQDNs from some clients (accidently or intentionally), how could it, effectively? The distributed nature of DNS makes it near impossible. If one DNS server doesn't give an answer, just ask another -- one that's working. And there are clearly many that are working. So why is Let's Encrypt stuck on one that isn't working? Isn't it possible just to get Let's Encrypt to use a different DNS server? Or find out from them what DNS server they are using so we can ping whomever-it-is that runs that server and ask them to fix it?

16

The DNS server they use is the authoritative ones for the domain, and it does try them all. Might be some sort of networking issue between where Let's Encrypt's servers are and where the .mil nameservers are?

3 Likes
17

Not if they are all behind the same firewall [policy].

Does .mil have something against AWS?

3 Likes
18

I thought only the secondary sites were on AWS. I'm guessing the issue is from their primary sites.

3 Likes
19

Does "the authoritative one" mean the same thing as "SOA"? (As I said, I'm no DNS expert.)

20

They don't use public DNS systems.
They go directly via all the authoritative DNSes along the way:
.
then
mil.
then
af.mil.

3 Likes
21

Um, aodms.af.mil is indeed being hosted by AWS. AFAIK, it is not true that all of the sites Let's Encrypt is choking on are AWS sites.