I ran this command: sudo certbot --text --agree-tos --non-interactive certonly --rsa-key-size 4096 -a webroot --cert-name 'lithium.kenyonralph.com' --webroot-path /var/www/html -d 'lithium.kenyonralph.com'
It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator webroot, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for lithium.kenyonralph.com
Using the webroot path /var/www/html for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. lithium.kenyonralph.com (http-01): urn:ietf:params:acme:error:dns :: No valid IP addresses found for lithium.kenyonralph.com
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: lithium.kenyonralph.com
Type: None
Detail: No valid IP addresses found for lithium.kenyonralph.com
My web server is (include version): Apache 2.4.38
The operating system my web server runs on is (include version): Debian 10 buster
I can login to a root shell on my machine (yes or no, or I don't know): yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0
lithium.kenyonralph.com has two AAAA records in DNS. One is a global unicast address, the other is a unique local unicast address (ULA):
I can only assume it isn't filtering that address out and is instead stopping - having encountered such an address. If that is the case, then the error message should be more clear about it.
Do you need to include that f7c6 address?
If so, you should test with Let's Debug.
I believe that the key to this problem might have something to do with DNSSEC.
My wild ass unsubstantiated guess that has little evidence: Let's Encrypt's DNS resolvers strip the private AAAA RR from the DNS response (the private-address option in Unbound). If the zone/response was signed, stripping RRs from it does something to the authentication status of the message, which causes either Unbound or Boulder to not use any RRs from the response.
If your zone was not configured with DNSSEC, then you wouldn't see that error message, and the public IPv6 address would be used.
I think we'd need to see a response from Let's Encrypt to be sure what's happening.
Having two address, with one being public and one being private, seems really weird, even if Let's Encrypt should technically be able to figure it out. I would expect that a random browser on the Internet would only be able to get to it half the time, if their system happened to pick the public address. Is this a system designed for internal-to-a-network use only? If so, then probably you're better off using a DNS-01 challenge instead.
It's not weird in IPv6. With legacy IP, the analogy would be having an RFC 1918 address in DNS, which doesn't work well, as you point out, because there are no address selection rules to make it work right. With IPv6, having both ULA and GUA available is a valid setup. RFC 6724 specifies IPv6 address selection behavior. Having both ULA and GUA in DNS like this allows the host to be accessible locally using the ULA, regardless of whether the GUA exists (it might disappear due to ISP problems, for example). Networks that use OpenWrt for their router work like this by default, I believe (the router advertises both ULA and GUA prefixes).
Hmm. I knew IPv6 defined source address selection behavior but was unaware of it also defining destination address selection behavior. I'd have to dig into it more, but this case still seems a little weird to me, because I'm not sure how a random node on the Internet would know whether a ULA would be accessible to its own network or not if it wasn't on the ULA subnet itself. That is, if doing a dual GUA/ULA setup like this, I would have naively expected both addresses to only be visible in a split DNS internal-only way, whereas the public DNS would only have public addresses. But that may just be because of some unintended legacy-IPv4 thinking creeping in to how I'm thinking of it.
The unboundtest conf seems to only configure as private-address the IPv4 RFC1918 space, though the Boulder code of course excludes more ranges than just that, but assuming that the real Boulder unbound is configured the same way, I don't think it's a DNSSEC-failed-due-to-DNS-server-removing-addresses thing. I would expect that if it were, that Boulder would give a SERVFAIL message instead of just saying that it couldn't find a valid IP. I may be wrong on that, though. In any event, if the way Boulder works is to just strip out any private addresses and utilize the public ones, I would have expected Boulder to be able to use this kind of setup.
In terms of a practical workaround to get you going for now, if you're not willing to take out the private address and have only the global address in DNS, then I think you're best off thinking of it the same way as you would for an actually only-accessible-internally server and using a DNS-01 challenge. I don't know if your DNS provider has an API for updating DNS records that certbot has a plugin for, but if they don't then you can use something like acme-dns where you set up a DNS server that you can update programmatically and delegate the DNS challenge to it. I don't know if that's the way you really want to go, though.
Or maybe I should try to bring in the big guns: @jsha: Can you take a look if you get a chance and see why Boulder returns No valid IP addresses found for lithium.kenyonralph.com while that address has no A records and two AAAA records, where one of the AAAA records is a ULA but the other seems to be a perfectly normal public IPv6 address? Thanks!
Yeah, as a workaround for now I'm just going to remove the ULA from DNS, since it's not strictly necessary, really only there as a backup. There is a certbot plugin for Gandi's DNS, so I might use that as well. But I figured I'd report this because it does seem like a Let's Encrypt bug. Thanks.
I personally really like using DNS for the challenges in general (assuming it can be automated fairly straightforwardly), since it avoids all the problems of needing to ensure that your actual system is accessible from everywhere on the Internet and you only need to ensure that your DNS server is accessible from everywhere on the Internet. So give it a go, and see how it works!
I am endlessly amazed by the ability of the smart people here to figure out subtle problems like this one without the benefit of being able to directly query our nameservers.
Indeed as @_az surmised, our Unbound instances are stripping out private addresses.
As @petercooperjr supposed, there was a difference between the config on unboundtest.com and what's in production. A while ago we updated production to remove a larger set of private addresses, matching Boulder's list. I forgot to update unboundtest.com. Sorry about that! It's now updated.
As for why this manifests as "No valid IP addresses" rather than SERVFAIL? Boulder only returns a SERVFAIL (or other DNS error) if both A and AAAA lookups had an error. In this case, the A lookup was successful, with NOERROR and an empty response, so we fell through to "No valid IP addressses."
So, at a minimum we should probably have Boulder return the DNS error in preference to "No valid IP addresses." We might also want to consider making Boulder solely responsible for filtering private IP addresses, since Unbound has do-not-query-address for its own internal queries.
private-address: <IP address or subnet>
Give IPv4 of IPv6 addresses or classless subnets. These are ad-
dresses on your private network, and are not allowed to be re-
turned for public internet names. Any occurrence of such ad-
dresses are removed from DNS answers. Additionally, the DNSSEC
validator may mark the answers bogus. ...
Heh. I certainly thought it was possible, but I didn't really expect that to be the case. Thanks for figuring this one out!
I think part of the problem (not just with this, but also some past cases of the Boulder error messages being confusing) is that it's trying to return one error message for two actually-independent lookups. I'm almost thinking that it should outright give two error messages every time if it can't find an IP that works, separately listing the A and AAAA lookup results. Maybe that'd end up causing more confusion for people than it'd solve, though.
Yeah, saying "couldn't find any IPs" is really confusing for the cases where there is an IP there but it's not a publicly usable one. Saying, "we found an IP, but it's in RFC 6598 space so it's not publicly accessible" would definitely help a case that comes up more often than one might expect on this forum. But then again, I do understand a CA wanting the belt-and-suspenders approach to ensure it's not accidentally issuing based on a private IP.
