Using the dns-persist-01 challenge, I was wondering why the issuer-domain-names array in the challenge object doesn't necessarily have to match the caaIdentities array (from the directory). draft-ietf-acme-dns-persist-00 says:
Note: The issuer-domain-names provided in the challenge object MAY
be drawn from the machine-readable caaIdentities array in the ACME
server's directory object, as specified in [RFC8555], Section
9.7.6. This creates a clearer programmatic link between the
server's advertised identities and the challenge object.
If both arrays would be the same, one could create the needed DNS entry for a specific domain only by looking at the directory, before even creating an order, authz and challenges.
Is there a reason why this is only a "MAY" and not a "MUST"?
So the only "correct" way is to create an order and get the needed values from the challenges?
IIRC that potion of caavalues was optional in 8555 server side, so you can't require client to use sometihng that don't exist at server side:
and some may have a dozen CAA domain they accept but not all of them are on acme server's vaule https://ccadb.my.salesforce-sites.com/ccadb/AllCAAIdentifiersReport
Worth noting because I hadn't checked until just now. Of the free ACME capable CAs I'm aware of, only one does not currently publish the caaIdentities field in their directory object, which is Actalis.
As of that draft, yes. I would also prefer it if it were different - the way many clients work is that they setup challenge configuration before actually starting to submit orders (e.g. you configure a webroot for HTTP-01 before order creation), but with dns-persist-01 the client can't really tell the user what to put into the dns-persist TXT record before having the challenge ready, except if the client makes some guess based of CAA or so of course.
I've been sort of following some of the in-progress discussions about the draft as those folks continue to work on it and it seems like they're trying to account for the pre-auth validation model so you shouldn't need to make a new order request before knowing what to put in the record. I'd be highly surprised if any of the CAs currently publishing the caaIdentities field wouldn't allow any of those values to be used as the issuer-domain.
If I understand correctly, the only down side to doing pre-auth is that you can only use your literal account URI in the record which negates one of the privacy aspects they're trying to include where the challenge object will have an order-specific URI you can use instead of the actual account URI. The order-specific URI makes it so that strangers querying your validation records won't necessarily be able to link disparate certs to the same account identity.
The fundamental answer here is straightforward: dns-persist is very new, while CAA is not.
There are CAs that have gone through mergers, acquisitions, rebrandings, etc. They might have 5 or 6 different URLs in their caaIdentities array. They have to keep them there, because some customers put CAA records in place back when those old URLs were the new/correct/best URLs, and backwards compatibility is important.
Should the CAs be required to support all of those historical URLs for a brand new thing, for which there is no backwards compatibility to maintain?
For DNS challenges in particular there has always been a potential for first time use to require stopping and resuming (e.g. manual dns, acme-dns), in this case it will be a little less of a problem and the client could attempt to cache last know issuer-domain-names per CA in order to help prompt the user preemptively.
My main concern would be the possibility of an issuer-domain-name being dropped and invalidating existing records, which would make it instantly non-persistent, but I'm assuming that would be rare.
I think the bigger problem will be that it naturally encourages multi-instance account key distribution in order to minimize different account URIs, but that's possibly better than multi-instance DNS credential distribution.
Perhaps tangential, but, this is a bit confusing for a "persistent" method. The ACME Client may not have to provision a TXT record at all if one preexists.
Challenge Response and Verification
To respond to the challenge, the ACME client provisions a DNS TXT record for the Authorization Domain Name being validated.
The draft is pretty clear later about TXT record persistence and contradicts the above. Seems that should say the "user or the ACME Client" or some such.
4.2. Just-in-Time Validation
When processing a new authorization request, a CA MAY perform an immediate DNS lookup for _validation-persist TXT records at the Authorization Domain Name corresponding to the requested domain identifier.
...
If at least one record meets all validation requirements, the CA MAY transition the authorization to the "valid" status without returning a "pending" challenge to the client.
...
This mechanism enables efficient reuse of persistent validation records
Personally, I think the biggest benefit from dns-persist is to avoid needing to dynamically provision a TXT record. Just manually create one in advance. Although, I can imagine larger setups or service providers may prefer otherwise.
Yes, that is the whole point. When the draft says "provisions a record", that doesn't necessarily mean "puts a record in place". It also encompasses "checks to see if a record is already in place".
The client should not be assuming that the record is already in place; that way lies misconfiguration and failed issuance. So it does need to do some work. But that work may just be checking that a good-looking record is present, and prompting the user to do one-time manual effort if a record is not already present.
Well, we differ on what "provision" implies But, a quibble.
I don't think the ACME Client pre-validating the dns-persist record is a good practice. Presumably the ACME Server's error message will be clear enough why it failed. Seems like an unnecessary duplication of code logic. And, an unnecessary duplication of DNS queries.
Not only do those "extra" DNS queries add resource burden but it can also lead to false failures if the client's DNS queries fail when the CA's would not. We see this regularly with dns-01 challenges. It adds complexity to the client system's setup to ensure their DNS queries work the same as the CA.
For a failed dns-persist challenge, some ACME Client may wish to perform postmortem DNS queries to guide the user to the proper corrective action. But, validating beforehand seems wasteful and error prone.
I recognize that I'm the CA operator, not the client operator, and therefore its very easy for me to say "make the client do it". But, as that CA operator, I strongly disagree with that stance, for several reasons.
First, a client performing a single DNS TXT lookup to make sure the record is in place is a significantly smaller waste of resources than going through an entire ACME order creation flow only to discover that validation fails. It uses a tiny amount of the client's CPU and a miniscule amount of bandwith. In contrast, going through the whole ACME flow consumes considerably more of the client's CPU and bandwidth, and it consumes our (donation-supported) CPU and bandwidth, too. Given that something like 80% of all validations fail, we would love it for more clients to perform the bare minimum self-checks before reaching out to the ACME server.
Second, let's think through the workflow for setting up a dns-persist-01 record for the first time. The client operator configures their client to use this challenge. But they don't know what the CA's issuerDomainNames are, and they likely don't know their own ACME URL (it's stored in a config file... somewhere). So they can't just create the TXT record themselves. Instead, they tell the client to begin the issuance process, and it prompts them to create the record, printing out exactly what they should put in the text record and waiting for the user to confirm that they've done so. But how does it know that it needs to prompt and wait? If it did that every time, all renewals would be manual, and that's a problem. If it never did the prompt-and-wait, the user would never be told what record they should put in place. So the easy solution is for the client to check for the record itself, and only prompt if a matching record isn't found.
So I think the clear best practice is for clients to do this cheap and easy DNS TXT lookup on their own, as a first filter to prevent doomed-to-failure requests from wasting LE's resources and to make sure their operator is informed of failure as quickly as possible.
That's a sorry state of affairs and which you have far more insight to than I do. I hope the much simpler and persistent challenge will reduce those failure rates. But, I better appreciate why you'd want the clients to do more.
Well, we know the CA Identities either from the CA's own docs/instructions on using this challenge or from the /directory. And, the accountURL is easy enough for an ACME Client to show after registration. None of these involve placing an order.
Designing the TXT record for this challenge takes more effort than placing a simple value. The user needs to decide about the wildcard and expiration policies and possible future elements or CA variances.
An ACME Client could offer a "helper" function for this challenge rather than trying to interweave a "pre-check" into each request. A common website tool like the CAA builder by SSL Mate is even an option.
Given the trend towards built-in ACME Clients like Apache mod_md, nginx-acme, caddy, the CDNs and all the rest it seems that education and out-of-band preparation is going to be needed anyway. Only ACME Clients that offer an interactive UI will be able to provide your suggested coaching.
The problem with client initiated pre-validation is that a decent number of client systems don't have a real view of the public facing DNS zone. Enterprise environments often resolve an internal-only view of a zone that doesn't contain the public facing validation records. And the client doesn't really have a good way of knowing this is the case. It'll just get NXDOMAIN responses when it tries to check even if the record really is there in the public copy.
Conversely, you might also have users who end up accidentally adding the necessary records to the wrong copy of the zone. Then the pre-validation succeeds, but the real validation doesn't and the user is left scratching their head as to why.
You can't rely on directly querying public resolvers like Google/Cloudflare either as many of those same enterprise environments either block queries to public resolvers or transparently proxy those queries to an internal resolver to prevent managed endpoints from bypassing security controls.
I remember early on in Posh-ACME development wishing that the CAs exposed some sort of /dnscheck endpoint that would allow clients to effectively do pre-validation specifically from the CA's DNS point of view. But ultimately, it's not really much different than just proceeding with the challenge ack process.
On this topic, I am considering adding pre-flight validation checks to my in-development ACME client using Let's Debug. This allows for an outbound perspective, similar to the CA, to check for potential problems.
This is particularly helpful for "first-time" issuances. Once we have a working configuration my opinion is that one can skip at least some "difficult" checks until authorization fails e.g. due to a configuration change - then we're back to "first-time" issuance until it works again.
Let's Debug already has an (underdocumented) API for this, and I am planning to extend it to let the requester configure more aspects around the checkers, i.e. disable the test staging validation to avoid duplicating CA load.
(I am currently busy with other stuff so this isn't going to happen soon, but is planned for later this year)
I had hoped the ACME client did not have to start the issuance process at all (by creating a new order) just to be able to get the CA's issuerDomainNames. (To get all needed values for the dns-persist-01 challenge)
Imagine for example using the "shortlived"-Profile where the "Pending Authorization Lifetime" is only 1 hour, so the DNS record has to be in place before the authz expires while prompt-and-wait. Otherwise the order is wasted and a new one has to be created later on when getting the certificate.
Even when a ACME client only wants to check wheter the correct dns record is present, it has to waste an order for that, assuming a stateless client with no caches.
I think having the directory metadata include the currently supported issuing domains would be a good idea for preemptive provisioning but I guess there's all sorts of edge cases where that's a gotcha, maybe it could be optional metadata if it's supported by the CA.
Even if a client caches previous record values per-CA and suggests those pre-emptively to provision new records (manually or automatically), the worst case is that the first order for that new domain fails. It's not very "clean" but it'll do for many use cases. Working with with acme-dns like services (which many thousands of orgs do) is a bit like this currently, because the user commonly misconfigures the initial CNAME etc. I don't think we need to try to solve user error here.
However, provisioning the TXT record automatically is possible for a decent number for clients but it's still pretty hard due to the chaotic, unstable and bespoke nature of the many thousands of DNS APIs. We also want to move away from giving ACME clients DNS credentials to play with.
The easiest thing to do (for any acme client) is let the first order fail as a prompt to the operator. If they rely on automation for provisioning then they also rely on failure reporting. It's the first order out of thousands for that cert and a little uncertainty around your initial config is a reasonable price to pay for the overall ongoing convenience of the validation method.
We've learned that being pragmatic and letting things be a little fluid while the process shakes out is better than over engineering for all the edge cases we can imagine - there are still all the other established domain validation options to chose from.
The first dns-persist order is not guaranteed to fail. The user may have setup the DNS record properly before requesting the cert
For an interactive ACME Client, it could prompt if it detects a request for a cert that it hasn't handled before using dns-persist. That could be instead of trying to mimic CA DNS validation.
But, a quibble.