New implementation of acme client but struggling with even account creation

Client dev
1

Hi, I write after exhausting everything else I can think of.
I am looking to integrate implement Acme and have been testing my implementation against the staging server while cross referencing 'dehydrated' - the bash client.
But I cannot get create account to work and it's really hard to know what I'm doing wrong because the error information isn't extensive.
I think I am sending the right information, here's an example - don't worry there's no leakage here the keys are all throw away anyway:

{"protected": "eyJhbGciOiAiRVMzODQiLCAiandrIjogeyJrZXlfb3BzIjogWyJ2ZXJpZnkiXSwgImV4dCI6IHRydWUsICJrdHkiOiAiRUMiLCAieCI6ICJCaWZtdldDX0FkYWJ5dXdmenRYbTY0ems5VVB6aTg3ZVJUYXVKYTNUWURVUG9mN0hNelVmeE9zSkI4RDVnVzEwIiwgInkiOiAibmpJNDF2VjIwbzZ3bXpkVk5DTnNlRUM3em5nTkZjZ25TdVVkSHBCeUdNNHpRLUZuX2luY01NSGVTaHhLcDM5TSIsICJjcnYiOiAiUC0zODQifSwgInVybCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1hY2N0IiwgIm5vbmNlIjogIktfV2NNWFJxVWJzLTlOV3lSMEw3MXd1dmtkSWdhN0RzZXZXa1pwNGQwQXJnNnJWMkZfYyJ9", "payload": "eyJ0ZXJtc09mU2VydmljZUFncmVlZCI6IHRydWUsICJjb250YWN0IjogWyJtYWlsdG86bmljQGZlcnJpZXIubWUudWsiXX0", "signature": "eyJhbGciOiAiRVMzODQiLCAiandrIjogeyJrZXlfb3BzIjogWyJ2ZXJpZnkiXSwgImV4dCI6IHRydWUsICJrdHkiOiAiRUMiLCAieCI6ICJCaWZtdldDX0FkYWJ5dXdmenRYbTY0ems5VVB6aTg3ZVJUYXVKYTNUWURVUG9mN0hNelVmeE9zSkI4RDVnVzEwIiwgInkiOiAibmpJNDF2VjIwbzZ3bXpkVk5DTnNlRUM3em5nTkZjZ25TdVVkSHBCeUdNNHpRLUZuX2luY01NSGVTaHhLcDM5TSIsICJjcnYiOiAiUC0zODQifSwgInVybCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL25ldy1hY2N0IiwgIm5vbmNlIjogIktfV2NNWFJxVWJzLTlOV3lSMEw3MXd1dmtkSWdhN0RzZXZXa1pwNGQwQXJnNnJWMkZfYyJ9.eyJ0ZXJtc09mU2VydmljZUFncmVlZCI6IHRydWUsICJjb250YWN0IjogWyJtYWlsdG86bmljQGZlcnJpZXIubWUudWsiXX0.ApCx0mjqRdArdrYfdjOUa2_DaAQ7PTi2ko03DUPEkZ61TSW2-CZ3U4OUzKujeyCKaMdaGjHB0F-kOqdPZsIo4fjjPfyLb25t2JHmTDl_S3NqLaOR3E7TKPkFNhAVxugB"}

of course, under neath these base64 strings is this:

protected: {"alg": "ES384", "jwk": {"key_ops": ["verify"], "ext": true, "kty": "EC", "x": "BifmvWC_AdabyuwfztXm64zk9UPzi87eRTauJa3TYDUPof7HMzUfxOsJB8D5gW10", "y": "njI41vV20o6wmzdVNCNseEC7zngNFcgnSuUdHpByGM4zQ-Fn_incMMHeShxKp39M", "crv": "P-384"}, "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct", "nonce": "K_WcMXRqUbs-9NWyR0L71wuvkdIga7DsevWkZp4d0Arg6rV2F_c"}

and:

payload: {"termsOfServiceAgreed": true, "contact": ["mailto:nic@RESTRACTED"]}

But I get:

{"type": "urn:ietf:params:acme:error:malformed", "detail": "Unable to validate JWS :: Parse error reading JWS", "status": 400}

It's tricky to know what 'parse error' means in this context? the JSON I'm sending? the base64 decoded JSON? something wrong with the key setup?

Does anyone know?

Is there a version of the acme cert server that I can run locally and debug maybe?

Thanks.

2

Welcome to the Let's Encrypt Community! :slightly_smiling_face:

You need to sort the json keys in a certain order in your payload and protected for the signature to be correct.

As a side note, there's no need to include the contact field since Let's Encrypt discards it anyhow.

2 Likes
3

thank you! that's interesting because the RFC8555 example has no such ordering...
in both the payload and the protected? because I just changed the payload and it made no difference to my error!

4

Updated my post above.

You need to sign "$protected.$payload" where $protected and $payload are base64-encoded json, so the order of both matters.

1 Like
5

Right, I thought that's what I'd done.

I even wrote some more code to import the key and verify my sig and my sig does verify with that code:

const keyJson = fs.readFileSync("./test-new-account.json", "utf8");
const keyObject = JSON.parse(keyJson);
const {protected:protectedVal} = keyObject;
const protectedStr = Buffer.from(protectedVal, "base64url").toString();
const protectedJson = JSON.parse(protectedStr);
const importedKey = await subtle.importKey(
    "jwk",
    protectedJson.jwk, {
        name: "ECDSA",
        namedCurve: "P-384",
    },
    true,
    ["verify"]
);
const {payload:payloadVal, signature:signatureVal} = keyObject;
const payloadStr = Buffer.from(payloadVal, "base64url").toString();
const payloadJson = JSON.parse(payloadStr);
const [sigProtected, sigPayload, signatureEncoded] = signatureVal.split(".");
console.log({protected:protectedStr, payload: payloadStr});
const signature = Buffer.from(signatureEncoded, "base64url");
const signedData =
      Buffer.from(protectedStr).toString("base64url")
      + "."
      + Buffer.from(payloadStr).toString("base64url");

const verified = await subtle.verify({
    name: "ECDSA",
    hash: {name: "SHA-384" }
}, importedKey, signature, Buffer.from(signedData));

console.log("verified:", verified);

this just uses what I'm submitting to Acme. And it verifies the key.

So I am still confused about what acme is telling me.

You seem to be confirming that acme is processing my signature and telling me that things aren't wright.

I have tried:

  • reordering all the JSON property keys in my input (including in the jwk)
  • reordering the protected JSON property keys and the payload keys

I have not tried putting payload first (pa comes before pr, is that what you mean?)

I don't understand why the ordering of property keys matters to acme, it must be reordering what I send it, I guess?

6

When you sign (meaning encrypt) the string of "base64(protected json)"."base64(payload json)" with your account key, the order of the json data matters because the ciphertext of the signature will differ when the order of the json data differs.

1 Like
7

See lines 94 - 278 of my certsage.php for a working example using an RSA account key. There will only be a slight modification for EC account key. Line 122 might warrant special focus.

1 Like
8

Maybe I am just missing something fundamental... it's possible!
But can you see that I have written code to verify my signature?
My verify does this:

  • take the JSON keys for protected and payload
  • base64url decode them into strings
  • import the key (from JSON decoded object versions of the strings)
  • put the strings back together with ".": protected.payload
  • verify that data with the sig from the JSON

If Acme is doing something different then ok. But nothing says that and I don't understand what you're saying about reordering... the text of the file I'm importing above is:

{"protected": "eyJhbGciOiAiRVMzODQiLCAiandrIjogeyJjcnYiOiAiUC0zODQiLCAiZXh0IjogdHJ1ZSwgImtleV9vcHMiOiBbInZlcmlmeSJdLCAia3R5IjogIkVDIiwgIngiOiAiV0M5Ni1wMkNJMHVpOUE1a0huSGdxSHZ2eUQ5dWt2VXJBWVdyNHhBSEtlM3hoYUZZaURYZ0dVYk04T0ZKLTNDUyIsICJ5IjogImlGTkZLMmJseDcwNDFad1B3ekYzUmc5V2NwUHhhR3J4eERKaU91Nk9QRjF2dHVQcFVrU3NVNU85R1hpdEc2TFgifSwgIm5vbmNlIjogIktfV2NNWFJxU1FudTJCRWVWOERlSzQ2U2RDc2V1R2hlcER1VzFrTmNneEtTbC00Wnd2USIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctYWNjdCJ9", "payload": "eyJjb250YWN0IjogWyJtYWlsdG86bmljQGZlcnJpZXIubWUudWsiXSwgInRlcm1zT2ZTZXJ2aWNlQWdyZWVkIjogdHJ1ZX0", "signature": "eyJhbGciOiAiRVMzODQiLCAiandrIjogeyJjcnYiOiAiUC0zODQiLCAiZXh0IjogdHJ1ZSwgImtleV9vcHMiOiBbInZlcmlmeSJdLCAia3R5IjogIkVDIiwgIngiOiAiV0M5Ni1wMkNJMHVpOUE1a0huSGdxSHZ2eUQ5dWt2VXJBWVdyNHhBSEtlM3hoYUZZaURYZ0dVYk04T0ZKLTNDUyIsICJ5IjogImlGTkZLMmJseDcwNDFad1B3ekYzUmc5V2NwUHhhR3J4eERKaU91Nk9QRjF2dHVQcFVrU3NVNU85R1hpdEc2TFgifSwgIm5vbmNlIjogIktfV2NNWFJxU1FudTJCRWVWOERlSzQ2U2RDc2V1R2hlcER1VzFrTmNneEtTbC00Wnd2USIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctYWNjdCJ9.eyJjb250YWN0IjogWyJtYWlsdG86bmljQGZlcnJpZXIubWUudWsiXSwgInRlcm1zT2ZTZXJ2aWNlQWdyZWVkIjogdHJ1ZX0.YwHCjOuccBDKXFDORL9NwE0PvAL-MoDeT5Vxg-cMUGy36o3G1DymnK-6wwVfVsY03rztYXhUr_zprth57QWJ2qNr8Ysi6_s4ws8cujUT3UVZKEzVfAHd37OS7c-oZdyk"}

and now that has reordered JSON inside the base64 stuff:

protected: {"alg": "ES384", "jwk": {"crv": "P-384", "ext": true, "key_ops": ["verify"], "kty": "EC", "x": "WC96-p2CI0ui9A5kHnHgqHvvyD9ukvUrAYWr4xAHKe3xhaFYiDXgGUbM8OFJ-3CS", "y": "iFNFK2blx7041ZwPwzF3Rg9WcpPxaGrxxDJiOu6OPF1vtuPpUkSsU5O9GXitG6LX"}, "nonce": "K_WcMXRqSQnu2BEeV8DeK46SdCseuGhepDuW1kNcgxKSl-4ZwvQ", "url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct"}

and:

payload: {"contact": ["mailto:nic@ferrier.me.uk"], "termsOfServiceAgreed": true}

But it still gives me exactly the same message.

In your next message you say to look at your example and point out the base64-url encoding... I am not sure why you're doing that. I will look at your implementation but my point is I've been over implementations and I can't see what I'm doing different and I can't see why Acme is responding negatively when my own verifications tests check out.

But thanks so much for taking the time to try to help a fellow implementor.

9

For each of protected and payload, which should be json object strings, you need to base64-encode the json object string then URL-encode the result. Then you concatenate those URL-encoded protected and payload strings with a period in between then sign the resulting string. That signature then needs to be base64-encoded and URL-encoded. Then you construct a json object containing the URL-encoded versions of protected, payload, and signature (in that order) and send the resulting json object string as the HTTP body via POST.

1 Like
10

Yes! I know! :slight_smile:
I was explaining how I verify what I've made not how I produce it.
I am doing exactly what you say. I am also able to verify what I produce with the code above.
But when I POST to let's encrypt staging server it still gives me the same result.
I guess I should test it with someone else's imlementation to see if staging works at all.
Maybe it doesn't.

11

Staging works fine with the CertSage code I sent you. :slightly_smiling_face:

1 Like
12

The simplest verification is that the account actually gets created.

1 Like
13

I get that. That's why I am asking for help. Because I think I am doing everything correctly and yet I am getting an error from staging and it's not an informative error.

Nothing you've said so far (I am not criticizing your help!) has been something wrong in my implementation.

Guess I'll check staging and come back here with more info.

14

Is your protected in this order?

$protected = [
  "url"   => $url,
  "alg"   => "RS256",
  "nonce" => $nonce
];

I may have misled you earlier with my "alphabetical" comment.

1 Like
15

I just tried it that way, it makes no difference.

I am not sure why you think any of that ordering matters, it certainly doesn't seem to OR it does but my error is before it even gets to that.

But again, as an implementor I don't understand why the server would reorder what I am sending it. Surely it just does basically what my verification example does above. So why would it need to reorder keys?

Anyway, as I say, I tried it that way. It doesn't make any difference:

acme> protected: {"url": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct", "alg": "ES384", "nonce": "XVP_df_uIpPKxFxo2sHFxkYmDrKyzREo_4kHeQ-Hd4QD-BArUp8", "jwk": {"crv": "P-384", "ext": true, "key_ops": ["verify"], "kty": "EC", "x": "WC96-p2CI0ui9A5kHnHgqHvvyD9ukvUrAYWr4xAHKe3xhaFYiDXgGUbM8OFJ-3CS", "y": "iFNFK2blx7041ZwPwzF3Rg9WcpPxaGrxxDJiOu6OPF1vtuPpUkSsU5O9GXitG6LX"}}

(I presume you meant that the jwk goes in there last, like your implementation does)

16

Ok. Dehydrated does not seem to work on staging either.

:smiley:

17

Maybe this will help?

1 Like
18

No, it won't help.
As I said at the beginning I've been through everything I could find, including that thread.
My understanding of that issues is that they were sending the wrongly formatted webkeys...

It does have good advice though: write a test for what you're generating to prove that it works.

I did that. My test works.

Let's encrypt staging doesn't.

This is why it would be super useful to get a version of the code they use so I could run locally and check the basics.

I always say if someone isn't prepared to give you the code they run in staging there must be something fishy.

I'm sure that's not true with letsencrypt... everything is FOSS surely?

19

When we see this type of problem here, it's usually caused by one of these issues:

  • incorrectly-ordered JSON
  • incorrect application of json-encoding
  • incorrect application of URL-encoding
  • incorrectly-structured strings
2 Likes
20

The whole CA code is here:

If you want a light version to test against:

3 Likes