CAA SERVFAIL changes

API Announcements
1

When originally implementing CAA, we ran into a number of issues with various DNS providers not properly supporting queries for this record type. Because of this we implemented a soft-fail behavior when DNS servers return the SERVFAIL error code to CAA queries instead of the expected NOERRROR code.

Over the last year and a half, we’ve worked with a number of these providers to move towards spec compliance, but there are still a few misconfigured providers. However, we are moving towards strict enforcement of CAA, where SERVFAIL responses will block issuance. This is also in line with upcoming industry requirements: all CAs will soon be required to check CAA by the CA/Browser Forum Baseline Requirements.

We’ve recently added functionality that disables our workaround for any new users but in order to provide existing users a window of time to fix any issues, we have created a list of domains that are allowed issuance despite CAA failures for the next month and a half.

If you’d like to test if your DNS provider still has this issue, try a test issuance against our staging environment. This environment has an empty exceptions list and will behave the same as our production infrastructure will once we have completely removed the exceptions code. If your provider still has this issue you will receive the error “DNS problem: SERVFAIL looking up CAA for {domain-name}” when attempting to validate a challenge for the domain in question.

People with unexpired certificates containing names on the exception list will receive an email informing them of this, but so everyone knows about the changes we are posting here too.

4 Likes
2

Note: there is a bug in PowerDNS versions 4.0.3 and below that will cause SERVFAIL problems.

If you are a DNS operator and you use PowerDNS, please upgrade to 4.0.4. If you do not operate your DNS and want to check if your DNS operator uses PowerDNS, you can in some cases check the version string. First, find out what your nameservers are:

$ dig +short ns YOUR_BASE_DOMAIN_NAME
ns1.example.net.
ns2.example.net.

Pick one of the returned nameservers, for instance ns2.example.net, and run this query (replacing ns2.example.net with one of the nameservers found from the above command):

$ dig +short version.bind chaos txt @ns2.example.net
"PowerDNS Authoritative Server 4.0.4 (built Jun 22 2017 20:14:47 by buildbot@c1b965951e5b)"

If you are getting CAA SERVFAIL errors, and this shows PowerDNS 4.0.3 or less, please contact your DNS operator to upgrade. Note that some servers may not allow querying version information. If you don't get results, or just get "Served by PowerDNS - http://www.powerdns.com", then default to contacting your DNS operator. If you aren't sure who your DNS operator is, ask your hosting provider.

Also note: Because this bug only manifests on empty responses, you may be able to work around it by adding a CAA record to your zone that authorizes issuance for Let's Encrypt. CAA is supported by PowerDNS since 4.0.0. If your nameservers are running earlier versions than that, you may also be able to work around by adding a CNAME to a domain name whose authoritative nameserver runs different software, and adding a CAA record authorizing issuance there.

5 Likes
3

Note: We’ve gotten feedback from a number of hosting providers that this email should have included a list of affected domain names. Agreed, and sorry about that! We’ve filed a couple of Boulder issues to improve our mailing process. However, rather than rush out those changes, I’m just going to post the list of affected domains. You can cross-reference this with the list of domains that you serve with a simple grep command:

fgrep --line-regexp --file servfail-list-other.txt YOUR_DOMAIN_LIST.txt
fgrep --line-regexp --file servfail-list-namebright.txt YOUR_DOMAIN_LIST.txt

Note that I’ve divided these into two lists: One for domains that use Namebright as their DNS provider, and one for the rest of the affected domains. This is because Namebright domains make up by far the largest share, at 30%, and if we can get in touch with Namebright and help them fix their nameservers, we will fix the issue with many of these domains at once. Also note that both lists contain only public data. The list of domains that Let’s Encrypt has issued certificates for is available in Certificate Transparency logs, and their CAA SERVFAIL status can be easily queried against public nameservers.

servfail-list-namebright.txt (95.2 KB)
servfail-list-other.txt (239.6 KB)

If you’d like to check any given domain name, please try unboundtest.com, a site I recently set up for this purpose.

2 Likes
4

We’ve now removed the exceptions list that @roland mentioned in his initial message. SERVFAILs for CAA are now treated as failures for all domain names. See https://letsencrypt.org/docs/caa/ for details on diagnosing and fixing CAA SERVFAIL problems.

1 Like