# BIND regression in 9.16.10 triggers SERVFAIL on CAA **URL:** \ **Category:** Help\ **Created:** [February 17, 2021, 5:19am UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482 "2021-02-17T05:19:40Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![ktk-certs](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@ktk-certs](https://community.letsencrypt.org/u/ktk-certs)\ **Post date:** [February 17, 2021, 5:19am UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482/1 "2021-02-17T05:19:40Z")
I signed up for the forum here to post a question, and in the process of gathering debugging info, figured out the problem. 🙂 But I could still use a few hints from the experts in how to narrow this down to report it "upstream" to ISC. **TL+DR summary** : ISC BIND 9.16.10 introduces a bug that, when LE installs TXT \_acme\_challenge.\, causes LE to report "SERVFAIL" even though wireshark confirms that NOERROR is returned by all name servers. The previous BIND release, 9.16.9, is not affected. **My domain is** : [moonlit-rail.com](http://moonlit-rail.com) _Nota Bene_: there is no point in checking this domain at the moment, as I have downgraded to BIND 9.16.9 on the internal server, which mitigates the bug. And yes, I now have renewed certs from LetsEncrypt. 😄 **I ran this command** : _dehydrated -c_ It's the older 0.6.5 version and runs against the ACME v2 DNS-01 API. **It produced this output** : - "_Error finalizing order :: While processing CAA for [www.moonlit-rail.com](http://www.moonlit-rail.com): DNS problem: SERVFAIL looking up CAA for [www.moonlit-rail.com](http://www.moonlit-rail.com) - the domain's nameservers may be malfunctioning_" **External tests** : [letsdebug.net](http://letsdebug.net) reported "OK", as did mxtoolbox, verisign and similar DNS checking sites. [unboundtest.com](http://unboundtest.com) also reported no errors initially, but I have been able to get it to repeatably report "_sec\_status\_bogus_" when running BIND 9.16.10 or 9.16.11, like so: - _unbound info: resolving [ns3.moonlit-rail.com](http://ns3.moonlit-rail.com). AAAA IN_ - _unbound info: response for [moonlit-rail.com](http://moonlit-rail.com). DNSKEY IN_ - _unbound info: reply from \ 88.218.94.169 #53_ - _unbound info: query response was ANSWER_ - _unbound info: validated DNSKEY [moonlit-rail.com](http://moonlit-rail.com). DNSKEY IN_ - _unbound info: validate(nodata): **sec\_status\_bogus** _ **Digging deeper** : Analyzing the raw DNS traffic via _wireshark_ was not enlightening, other than to show _NOERROR_ from all servers. However, in the BIND debug logs, I saw the following with bind 9.16.10 but _not_ with bind 9.16.9: - dnssec: warning: client @0x7fffb800cec8 66.133.109.36#34285 ([wwW.mOoNlIt-rAil.Com](http://wwW.mOoNlIt-rAil.Com)): view external: expected a exact match NSEC3, got a covering record There were several similar lines, one for each probe from LE. My assumption is that BIND is not properly maintaining the DNSSEC records when dynamic updates (from LE via dehydrated) are inserted. And that this causes a DNSSEC validation error, which in turn causes LE to report "SERVFAIL" even though the actual servers are returning queries just fine. **Questions** : All of the above seems simple enough to put into a GIT "issue" on ISC's GitLab. But I'm wondering if there may be anything non-standard in how the _[moonlit-rail.com](http://moonlit-rail.com)_ domain is organized that may also be to blame. None of the DNS checkers I ran complained. Nonetheless... - The certificate requested from LE contains the base domain name, plus the usual assortment of subdomains: www, smtp, ftp, and so forth. Each of these is implemented as a CNAME. But BIND tries hard to not allow you to add other RRs when there is a CNAME present. E.g., if in the zone for [example.com](http://example.com) there is a "_foo CNAME bar_", then BIND says there should **not** be a "_foo CAA letsencrypt_". But LE is probing for exactly that case. - Similarly, if there is a "_foo CNAME bar_" then should there also be a "_\_acme\_challenge.foo TXT \_" ? Would it make more sense to implement "foo" not as a CNAME, but instead by cloning the A/AAAA/etc of the corresponding server, "bar" ? I know that many sites use CNAMEs successfully, even LetsEncrypt's own API servers do so; but adding other RRs beside or below a CNAME is not something I see. Thanks for any thoughts... Kris Karas ---
**Author:** ![\_az](https://avatars.discourse-cdn.com/v4/letter/_/22d042/32.png) [@\_az](https://community.letsencrypt.org/u/_az)\ **Post date:** [February 17, 2021, 5:30am UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482/2 "2021-02-17T05:30:17Z")
Even with you running ISC BIND 9.16.9, your nameservers [generate responses for non-existent names which don't pass DNSSEC validation](https://letsdebug.net/nonexistent12345.moonlit-rail.com/453916). [nonexistent12345.moonlit-rail.com | DNSViz](https://dnsviz.net/d/nonexistent12345.moonlit-rail.com/dnssec/) makes gives a nice view as well. I wonder, is your zone split-horizon? Nicely written post by the way! ---
**Author:** ![ktk-certs](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@ktk-certs](https://community.letsencrypt.org/u/ktk-certs)\ **Post date:** [February 17, 2021, 5:37am UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482/3 "2021-02-17T05:37:58Z")
Thanks _\_az_! 😄 [edit] Wish I had known about DNSVis before posting the original. I'm running 9.16.11 on the public-facing DNS servers and 9.16.9 only on the internal master, figuring that the public-facing servers are not doing any DNSSEC maintenance, only copying from the internal master. But perhaps there's another bug in 9.16.10+ that is causing the NSEC3 lossage? Or perhaps, the NSEC3 issues are also present in 9.16.9 and I haven't noticed them yet (with LE at least). DNSVis would seem to suggest that. Yes, I'm running split horizon on the downstream from the master (ns3), which has quite a few views, one per VLAN plus an external one on the WAN. The tertiary servers (ns1 and ns2) only serve external, with no views. I'm using the 9.16 branch to handle the dnssec-policy feature (automatic key rollover). ---
**Author:** ![ktk-certs](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@ktk-certs](https://community.letsencrypt.org/u/ktk-certs)\ **Post date:** [February 21, 2021, 9:57pm UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482/4 "2021-02-21T21:57:26Z")
Just wanted to post an update. The issue is as-yet unresolved upstream. But working with the ISC devos, I have narrowed it down. ### Improper Usage Because my external DNS zone was not being automatically signed, as _named_ was supposed to be doing, I had tried a workaround of adding a _NSEC3PARAM_ RR directly into the zone. This caused named to sign the zone and add NSEC3 RRs. I went with this; and it worked fine up through BIND 9.16.9. But as has @_az noted, the NSEC3 RRs were improperly maintained, causing [no-such-address.moonlit-rail.com](http://no-such-address.moonlit-rail.com) to fail to prove its non-existence. In talking with the devos at _ISC_, they said that adding NSEC3PARAM into the zone is not supported, and will cause the sort of "_partially signed_" zone issues I saw with the NSEC3. The only proper way of doing this, they say, is to include a "nsec3param" statement inside the **dnssec-policy** block that controls automatic zone maintenance. Say what? Yes, the "nsec3param" statement was not added to BIND until 9.16.10! This change of API/configuration made to a minor patch revision is not going to make harried system administrators happy, when they have to upgrade bind on a 3:00am page and find it now requires reading the Bv9ARM and editing their config files. As it turns out, using ISC's approach (only controlling NSEC3 via the _dnssec-policy_ mechanism) doesn't work in one common situation, which is why I had to implement the workaround noted above, and which led to the failure of LetsEncrypt to validate the challenge. ### Zeroing in on the Bug Although the reasons are not yet understood, it seems that adding this very common control block to your zone's definition in _named.conf_ will cause BIND to fail to sign the zone, despite using ISC's mandated statement inside _dnssec-policy_: ```nohighlight update-policy { grant ddns-update-key zonesub ANY; }; ``` Remove that block, and automatic DNSSEC maintenance works just fine (and _LetsEncrypt_ will be happy). But then, you cannot dynamically update your zone using, for example, **dhcpd**. I'll update, of course, when I have a chance to work on this with ISC, upstream. ---
**Author:** ![ktk-certs](https://avatars.discourse-cdn.com/v4/letter/k/f07891/32.png) [@ktk-certs](https://community.letsencrypt.org/u/ktk-certs)\ **Post date:** [March 3, 2021, 7:27am UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482/5 "2021-03-03T07:27:58Z")
Just to keep everybody updated, _ISC_ has worked this bug and released a fix that makes _KASP_ (key and signing policy) work correctly for NSEC3. For reference, the commit is at: [ISC Merge 4739](https://gitlab.isc.org/isc-projects/bind9/-/merge_requests/4739); this is expected to debut sometime this month as part of the _BIND 9.16.13_ release. There was a bit of a disagreement over the documentation. ISC says that NSEC3 was not supported until BIND 9.16.10, whereas the _Bv9ARM_ reference manual has stated since 9.16.0 that _rndc_ can convert NSEC to NSEC3 and _named_ will maintain it. (Of course, BIND won't actually support NSEC3 properly in _dynamically updated_ zones until 9.16.13) I have not seen any other users of BIND chime in on the bug report I made over there. And I have not seen any other "me too" comments over here at LetsEncrypt. So, perhaps I'm the only one to have stumbled upon this. 😄 Well, if anybody does, at least we now know the cause and fix. ---
**Author:** ![system](https://global.discourse-cdn.com/letsencrypt/original/3X/c/a/ca6c06ea1ea201324bba7048c6841ce60236468d.png) [@system](https://community.letsencrypt.org/u/system)\ **Post date:** [April 2, 2021, 7:28am UTC](https://community.letsencrypt.org/t/bind-regression-in-9-16-10-triggers-servfail-on-caa/145482/6 "2021-04-02T07:28:01Z")
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.