DNS based validation fails on renew

[mrtux]

I’m using Crypt::LE the perl client for requesting certs and that worked like a charm in the past and stopped working a few days ago.

The issue I have right now if for requesting a new cert for: DNS:soldaten-ausruestung.de, DNS:soldatenausruester.de, DNS:soldatenausruestung.com, DNS:soldatenausruestung.de, DNS:www.soldaten-ausruestung.de, DNS:www.soldatenausruester.de, DNS:www.soldatenausruestung.com, DNS:www.soldatenausruestung.de, DNS:www.xn–soldaten-ausrstung-y6b.de, DNS:www.xn–soldatenausrster-uzb.de, DNS:www.xn–soldatenausrstung-e3b.de, DNS:www.xn–soldatenausrstung-e3b.net, DNS:xn–soldaten-ausrstung-y6b.de, DNS:xn–soldatenausrster-uzb.de, DNS:xn–soldatenausrstung-e3b.de, DNS:xn–soldatenausrstung-e3b.net

Most of the domains got correctly validated, however, one domain failed. When I run Crypt::LE now I’m asked to validate “soldatenausrüstung.net” and I create the required DNS-entry _acme-challenge.xn--soldatenausrstung-e3b.net IN TXT "3BSPI4At6AkLGk0gtgNXFEtZzXnDXimaIIP0-hQILfM". However, when validating I get the error DNS problem: NXDOMAIN looking up TXT for _acme-challenge.www.soldatenausruestung.de. This error persists when I re-run the Crypt::LE script.

Debug-Info:

Requesting challenge for domain www.soldatenausruestung.de.
Received challenges for www.soldatenausruestung.de.
Requesting challenge for domain soldaten-ausruestung.de.
Received challenges for soldaten-ausruestung.de.
Requesting challenge for domain soldatenausruester.de.
Received challenges for soldatenausruester.de.
Requesting challenge for domain soldatenausruestung.com.
Received challenges for soldatenausruestung.com.
Requesting challenge for domain soldatenausruestung.de.
Received challenges for soldatenausruestung.de.
Requesting challenge for domain www.soldaten-ausruestung.de.
Received challenges for www.soldaten-ausruestung.de.
Requesting challenge for domain www.soldatenausruester.de.
Received challenges for www.soldatenausruester.de.
Requesting challenge for domain www.soldatenausruestung.com.
Received challenges for www.soldatenausruestung.com.
Requesting challenge for domain www.xn--soldaten-ausrstung-y6b.de.
Received challenges for www.xn--soldaten-ausrstung-y6b.de.
Requesting challenge for domain www.xn--soldatenausrster-uzb.de.
Received challenges for www.xn--soldatenausrster-uzb.de.
Requesting challenge for domain www.xn--soldatenausrstung-e3b.de.
Received challenges for www.xn--soldatenausrstung-e3b.de.
Requesting challenge for domain www.xn--soldatenausrstung-e3b.net.
Received challenges for www.xn--soldatenausrstung-e3b.net.
Requesting challenge for domain xn--soldaten-ausrstung-y6b.de.
Received challenges for xn--soldaten-ausrstung-y6b.de.
Requesting challenge for domain xn--soldatenausrster-uzb.de.
Received challenges for xn--soldatenausrster-uzb.de.
Requesting challenge for domain xn--soldatenausrstung-e3b.de.
Received challenges for xn--soldatenausrstung-e3b.de.
Requesting challenge for domain xn--soldatenausrstung-e3b.net.
Received challenges for xn--soldatenausrstung-e3b.net.
Requested challenges for 16 domain(s).
Domain www.soldatenausruestung.de has been already verified, skipping.
Domain soldaten-ausruestung.de has been already verified, skipping.
Domain soldatenausruester.de has been already verified, skipping.
Domain soldatenausruestung.com has been already verified, skipping.
Domain soldatenausruestung.de has been already verified, skipping.
Domain www.soldaten-ausruestung.de has been already verified, skipping.
Domain www.soldatenausruester.de has been already verified, skipping.
Domain www.soldatenausruestung.com has been already verified, skipping.
Domain www.xn--soldaten-ausrstung-y6b.de has been already verified, skipping.
Domain www.xn--soldatenausrster-uzb.de has been already verified, skipping.
Domain www.xn--soldatenausrstung-e3b.de has been already verified, skipping.
Domain www.xn--soldatenausrstung-e3b.net has been already verified, skipping.
Domain xn--soldaten-ausrstung-y6b.de has been already verified, skipping.
Domain xn--soldatenausrster-uzb.de has been already verified, skipping.
Domain xn--soldatenausrstung-e3b.de has been already verified, skipping.
Domain xn--soldatenausrstung-e3b.net has failed verification (status code 200).

Can anyone tell what’s wrong (or maybe check the server side)?

[umbi]

Hast Du schon auf dem server dehydrated -c -x (mit -X) versucht ?

[mrtux]

I debugged a bit more and found that the on the new-order call on the response the order of the identifiers does not match the order of the authorizations entries.

Has this recently changed in boulder? Is the order guaranteed?

[leader]

OK, that seems quite odd indeed. Do I get this right that you are using exactly the same version as before, which you mentioned worked fine until a few days ago?

[mrtux]

Do I get this right that you are using exactly the same version as before, which you mentioned worked fine until a few days ago?

I switched tools a few months ago, so it might be possible that this specific issue never worked before with Crypt::LE and just came up with the listed domains - lots of other domains worked since then.

[leader]

Acme draft doesn’t seem to be specific regarding the order of entries - @cpu, @jsha is it known whether for the newOrder request the order of entries in the authorizations array is guaranteed to match the order of entries in the identifiers array? I’ll have a look anyway of course, thanks for reporting @mrtux .

[mrtux]

I also filed an issue for Crypt::LE: https://github.com/do-know/Crypt-LE/issues/21

I suppose specifying that the order must match makes the life for clients easier.

[mrtux]

Short update: This issue also seems to arise for the tool I used before (acme_tiny.py, old version with acme v1). That script goes into an endlees loop.

[cpu]

Hi @leader,

@leader

is it known whether for the newOrder request the order of entries in the authorizations array is guaranteed to match the order of entries in the identifiers array?

There are no guarantees about order of order identifiers/authorizations. The spec. doesn't say anything about order of authorizations or identifiers. Client developers should sort these resources themselves if they are relying on a stable order. When time permits I'm intending to make this explicit in Pebble by randomizing the authorization/identifier order: Randomize order of identifiers in Order objects returned to clients. · Issue #104 · letsencrypt/pebble · GitHub

Hope that helps clarify!

[leader]

Hi @cpu,

Thanks for the clarification. Personally to me using arrays normally implies that there is a certain order (otherwise it would make more sense to use hash and associate authorizations with identifiers, rather than returning two arrays), but we have what we have I guess So since the order is not guaranteed, effectively we won’t know which domain we are requesting auth for until we have requested it and got the response. Any chance the specification gets changed rather than arrays get randomized?

@mrtux, will account for this specific behaviour in v0.32

[cpu]

@leader

So since the order is not guaranteed, effectively we won’t know which domain we are requesting auth for until we have requested it and got the response.

Maybe I'm missing some context (apologies, juggling a few threads today!). I don't understand this comment fully.

When you send a newOrder request with a set of identifiers you get back an order object with authorization URLs that the server wants satisfied. The authorization objects available at those URLs have an identifier field that includes the domain name being authorized. There's no guarantee at the protocol level that a newOrder request with 5 identifiers will get back an order object with 5 authorization IDs. Similarly there is no guarantee that if the newOrder request's first identifier was foo.com that the first authorization URL in the order object will be for an authorization with identifier foo.com.

I'm not sure why you need to match an order's identifier to an authorization URL from the order but if you do need that isn't the identifier in the authorization object sufficient?

@leader

Any chance the specification gets changed rather than arrays get randomized?

You'd have to take that request to the IETF WG Mailing List but let's make sure we understand each other first, maybe it isn't necessary for anything to change (Since the draft protocol is in last-call its also quite an uphill battle to try and land non-editorial changes)

Thanks!

[mrtux]

@cpu

There’s no guarantee at the protocol level that a newOrder request with 5 identifiers will get back an order object with 5 authorization IDs.

Does the newOrder response then include more than 5 identifiers or just the 5 ones which it has authorization IDs for?

Regarding the specification: There are two options which would help developers by explicitly negating possible missassumptions. 1) Explicitly state that the order if not guaranteed OR 2) that the order of both arrays must match.

[cpu]

@mrtux

Does the newOrder response then include more than 5 identifiers or just the 5 ones which it has authorization IDs for?

The order returned in response to a newOrder request can contain any number of authorizations. It's entirely dependent on server policy and not specified by ACME.

As one trivial example, Let's Encrypt happens to require a distinct authorization for each identifier but you can imagine a fictional CA that would receive a newOrder request with these identifiers:

[ 
   {"type": "dns", "value":"www.example.com"}, 
   {"type": "dns", "value": "example.com" } 
]

and return an order with only 1 authorization, with that authorization having identifier: {"type":"dns","value":"example.com"} and a set of challenges.

[mrtux]

@cpu

As one trivial example, Let’s Encrypt happens to require a distinct authorization for each identifier but you can imagine a fictional CA that would receive a newOrder request with these identifiers:

Please note, that I don't say that the identifiers must or should match those in the request. I'm talking about those (identifiers) included in the response which are controlled by the server (as well as the authorizations).

[leader]

@cpu

Maybe I’m missing some context (apologies, juggling a few threads today!). I don’t understand this comment fully.

When you send a newOrder request with a set of identifiers you get back an order object with authorization URLs that the server wants satisfied. The authorization objects available at those URLs have an identifier field that includes the domain name being authorized. There’s no guarantee at the protocol level that a newOrder request with 5 identifiers will get back an order object with 5 authorization IDs.

I meant that while processing that authorizations array, for each element you don't know which domain that is related to, until you request that URL and check the identifier in the response. Which is quite different concept from v1 (when you would be requesting authz for a specific domain). But it is certainly good to know, I just wish that specification was as detailed as your explanation!

[cpu]

@mrtux

I’m talking about those (identifiers) included in the response which are controlled by the server (as well as the authorizations).

@mrtux, Can you provide a more concrete example of a request and a response? I'm not tracking what you're asking. Is the question whether the authorizations returned must contain an identifier from the initial newOrder request's list of identifiers?

@leader

I meant that while processing that authorizations array, for each element you don’t know which domain that is related to, until you request that URL and check the identifier in the response.

@leader Aha! I understand. Yes, that is true but I'm not sure its a substantial problem. You have to GET the authorization to know which challenges are available and to learn the challenge token. Once you have those pieces of information you also have the corresponding identifier.

@leader

But it is certainly good to know, I just wish that specification was as detailed as your explanation!

Thanks! I'll think about whether there's a good way to clarify this in the spec text. I think @mrtux's suggestion to explicitly state that there is no guarantees about identifier/authorization sort order in order objects is a sensible one

[leader]

Got one more question - since (as you pointed out) the amount of elements in both arrays doesn’t have to match either, is it a viable scenario to see a response where the authorizations array is empty or missing completely?

[cpu]

Good questions

@leader

is it a viable scenario to see a response where the authorizations array is empty or missing completely?

The authorizations array is a required field in Section 7.1.3 so I expect it should always be there. I'm not sure what the semantics of an empty authorizations array are/should be. In theory I think the server could return an order object that is status: "ready" and has no authorizations if its willing to issue for any names without proof but I don't know that its a practical scenario to consider.

[jsha]

I think clients should gracefully handle the “empty authorizations array” case by considering the order ready to proceed to issuance. I’m guessing that some commercial CAs that do validation out-of-band might choose to do things that way (though they could also create synthetic authorization objects that represent the validation they’ve already done out-of-band).

[cpu]

@jsha

I think clients should gracefully handle the “empty authorizations array” case by considering the order ready to proceed to issuance.

Funny enough I think this is another point in favour of the "ready" order status. I think it clarifies what to do here vs having an empty authorizations array and "'status: pending".