Challenge authorization failing; appears truncated

KadeemH · November 17, 2019, 3:08am

I’m using the ACME.js library with a custom validation plugin, which I believe is working correctly now. However, it appears that the challenges I’m making are failing to authorize in a way that I’m not clear on how to resolve. I am using http-01 for authorization.

I have two sets of executions to share so that they can be cross referenced as I wanted to show the consistency of the issue.

Note: the error listed from ACME.js is misleading as it suggests the challenge must be in the pending state; it’s in the invalid state after attempting authorization (can be seen by following the link to the challenge)

It produced this output:
First:

{
  "errorType": "Error",
  "errorMessage": "[acme-v2] promo.cheapcows.com status:400 Unable to update challenge :: authorization must be pending",
  "code": "E_ACME_UNKNOWN",
  "auth": {
    "identifier": {
      "type": "dns",
      "value": "promo.cheapcows.com"
    },
    "status": "pending",
    "expires": "2019-11-24T02:06:30Z",
    "challenges": [
      {
        "type": "http-01",
        "status": "pending",
        "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21509306/8nkPTg",
        "token": "FHjUbUHD0Me3ln5GUZUCwhQ5pWyTjz4ZWoTsl49RaTs"
      },
      {
        "type": "dns-01",
        "status": "pending",
        "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21509306/wAAS6Q",
        "token": "FHjUbUHD0Me3ln5GUZUCwhQ5pWyTjz4ZWoTsl49RaTs"
      },
      {
        "type": "tls-alpn-01",
        "status": "pending",
        "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21509306/4yyvxw",
        "token": "FHjUbUHD0Me3ln5GUZUCwhQ5pWyTjz4ZWoTsl49RaTs"
      }
    ],
    "type": "http-01",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21509306/8nkPTg",
    "token": "FHjUbUHD0Me3ln5GUZUCwhQ5pWyTjz4ZWoTsl49RaTs",
    "hostname": "promo.cheapcows.com",
    "altname": "promo.cheapcows.com",
    "thumbprint": "5ie-PuXNSp6RgE_t1ObBi4O8oX4U-HMsgN2xB7MOjN0dgVp5dXnVdJDND8CcfC4giZSDYj5_cD1G8sfjO3W6yg",
    "keyAuthorization": "FHjUbUHD0Me3ln5GUZUCwhQ5pWyTjz4ZWoTsl49RaTs.5ie-PuXNSp6RgE_t1ObBi4O8oX4U-HMsgN2xB7MOjN0dgVp5dXnVdJDND8CcfC4giZSDYj5_cD1G8sfjO3W6yg",
    "challengeUrl": "http://promo.cheapcows.com/.well-known/acme-challenge/FHjUbUHD0Me3ln5GUZUCwhQ5pWyTjz4ZWoTsl49RaTs"
  },
  "altname": "promo.cheapcows.com",
  "type": "http-01",
  "stack": [
    "Error: [acme-v2] promo.cheapcows.com status:400 Unable to update challenge :: authorization must be pending",
    "    at transformError (/opt/nodejs/node_modules/@root/acme/acme.js:819:10)",
    "    at process._tickCallback (internal/process/next_tick.js:68:7)"
  ]
}

Second:

{
  "errorType": "Error",
  "errorMessage": "[acme-v2] promo.cheapcows.com status:400 Unable to update challenge :: authorization must be pending",
  "code": "E_ACME_UNKNOWN",
  "auth": {
    "identifier": {
      "type": "dns",
      "value": "promo.cheapcows.com"
    },
    "status": "pending",
    "expires": "2019-11-24T02:40:07Z",
    "challenges": [
      {
        "type": "http-01",
        "status": "pending",
        "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21513126/89PaBg",
        "token": "8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs"
      },
      {
        "type": "dns-01",
        "status": "pending",
        "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21513126/1eCWBw",
        "token": "8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs"
      },
      {
        "type": "tls-alpn-01",
        "status": "pending",
        "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21513126/MMC5Pg",
        "token": "8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs"
      }
    ],
    "type": "http-01",
    "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21513126/89PaBg",
    "token": "8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs",
    "hostname": "promo.cheapcows.com",
    "altname": "promo.cheapcows.com",
    "thumbprint": "5ie-PuXNSp6RgE_t1ObBi4O8oX4U-HMsgN2xB7MOjN0dgVp5dXnVdJDND8CcfC4giZSDYj5_cD1G8sfjO3W6yg",
    "keyAuthorization": "8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs.5ie-PuXNSp6RgE_t1ObBi4O8oX4U-HMsgN2xB7MOjN0dgVp5dXnVdJDND8CcfC4giZSDYj5_cD1G8sfjO3W6yg",
    "challengeUrl": "http://promo.cheapcows.com/.well-known/acme-challenge/8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs"
  },
  "altname": "promo.cheapcows.com",
  "type": "http-01",
  "stack": [
    "Error: [acme-v2] promo.cheapcows.com status:400 Unable to update challenge :: authorization must be pending",
    "    at transformError (/opt/nodejs/node_modules/@root/acme/acme.js:819:10)",
    "    at process._tickCallback (internal/process/next_tick.js:68:7)"
  ]
}

My web server is (include version): It’s a bespoke server based on netty (Java)

The operating system my web server runs on is (include version): Ubuntu 18.04

My hosting provider, if applicable, is: AWS

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): ACME.js v3.0.8 (@root/acme@3.0.8)

As an example, when following this link: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/21513126/89PaBg
I see the authorization failed: Invalid response from http://promo.cheapcows.com/.well-known/acme-challenge/8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs [34.213.96.136]: "8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs.5ie-PuXNSp6RgE_t1ObBi4O8oX4U-HMsgN2xB7MOjN0dgVp5dXnVdJDND8CcfC4giZSDYj5_cD1G8sfjO3W6"

However, when I query the URL myself:

curl -v 'http://promo.cheapcows.com/.well-known/acme-challenge/8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs' -w '\n'
*   Trying 34.213.96.136...
* TCP_NODELAY set
* Connected to promo.cheapcows.com (34.213.96.136) port 80 (#0)
> GET /.well-known/acme-challenge/8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs HTTP/1.1
> Host: promo.cheapcows.com
> User-Agent: curl/7.54.0
> Accept: */*
> 
< HTTP/1.1 200 OK
< Date: Sun, 17 Nov 2019 02:47:10 GMT
< Content-Type: application/octet-stream
< Content-Length: 130
< Connection: close
< P3P: CP="This is not a privacy policy."
< X-Proxy-Backend: page-server
< 
* Closing connection 0
8928kYRWus0cr48hw5FOu-fn75GHHLFfHTv9odG8Yrs.5ie-PuXNSp6RgE_t1ObBi4O8oX4U-HMsgN2xB7MOjN0dgVp5dXnVdJDND8CcfC4giZSDYj5_cD1G8sfjO3W6yg

This is also with resolving the same address (the other address returns the same content).

It appears that the correct response is being received, but something is being truncated and I’m not clear why that would be the case. (the last two letters yg appear to be missing in the authorization failure message)

cpu · November 18, 2019, 3:58pm

Hi @KadeemH, thanks for writing a detailed post. I just wanted to leave you a note to say that I’ve seen your post and will get back to you when I’ve had a chance to look more closely.

cpu · November 18, 2019, 5:18pm

Hi again @KadeemH,

It looks like the key authorizations have been deprovisioned from your end. My GET requests return 404s now. I think I was able to spot the problem based on the output you shared. Thanks again for including all the details, it makes investigation much easier!

It looks to me like your ACME client is miscalculating the account thumbprint, and as a result the overall key authorization.

The thumbprint here is 86 bytes, where a normal account thumbprint should be shorter. It should be just 256 bits (32 bytes) of output from SHA256. When it is encoded in base64url there should be something like ~33% overhead. I'd expect all valid thumbprints to be between 32 and ~43 bytes and 86 indicates something fishy is up with your calculation.

The reason that the error message you receive from Let's Encrypt is truncated and not a typical mismatched key authorization error is because the encoded token, the . separator and your over-sized thumbprint exceeds the maxResponseSize we have configured (128 bytes).

Can you share more information about how you're calculating the thumbprint of your account JWS with ACME.js? You might also want to report an issue with the maintainer if we aren't able to spot the problem here.

Hope that helps,

KadeemH · November 18, 2019, 5:24pm

Interesting. The library is definitely the thing which is calculating the thumbprint, so I’ll see if I can find why it might be creating one so large.

Library code is here: https://git.rootprojects.org/root/acme.js.git
I’ll see how the calculation is working and get back to you as soon as I have a better idea.

cpu · November 18, 2019, 5:26pm

Sounds good I'm not a Javascript programmer and probably won't be a lot of use digging through the library code myself.

KadeemH · November 18, 2019, 5:51pm

I have been able to see that the library I’m using is not calculating the thumbprint correctly for the version of the library it’s using versus the one it is supposed to use.

I’ve found this to be an issue with the library using an older version of their keypairs library which calculates the thumbprint versus the newer version of their library which calculates correctly. I was able to work around the previous issue, so I will continue to see if there’s a way to work around this issue as well.

KadeemH · November 18, 2019, 5:56pm

I believe I’ve narrowed the issue down to the Keypairs library using SHA-512 for the thumbprint (same as the issue I mentioned earlier with signing JWS requests). I believe that if I can get it to correctly use SHA-256, this issue would be resolved.

The n length on my private key is long, hence why I’m seeing this issue where it would likely not present.

cpu · November 18, 2019, 6:01pm

Aha! That would definitely explain it. RFC 8555 / ACME specifically require SHA-256 for the account thumbprint: RFC 8555 - Automatic Certificate Management Environment (ACME)

That shouldn't matter since the size of the output hash digest is constant no matter the length of the input. Once the code is switched to using SHA-256 you'll always get a 256 bit output.

Thanks for the update

KadeemH · November 18, 2019, 6:40pm

In the old version of the library, it chooses SHA-512/SHA-384 for longer n lengths, which the new version always selects SHA-256 for.

I’ll be raising this as an issue for the maintainer as they simply don’t use the new version of their library which I don’t understand.

cpu · November 18, 2019, 6:42pm

Thanks for clarifying. Now I understand what you meant before about the key length changing your outcome (and what a confusing decision by the library...)

Good luck!

KadeemH · November 18, 2019, 6:44pm

Forcing signing with SHA-256 worked.

Thanks for helping me figure this one out! I didn’t even think the thumbprint would have been the issue.

cpu · November 18, 2019, 6:46pm

Happy to help Glad to hear you’re all set

system · December 18, 2019, 6:46pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Client ACME not working "Pending" Client dev	23	1406	September 26, 2021
Http challenge failure Help	6	266	June 14, 2024
Certificate generation is failing Help	7	108	August 29, 2024
Unable to update challenge :: authorization must be pending Help	9	4061	December 30, 2020
Issue with acme challenging Help	32	5271	September 15, 2018

Challenge authorization failing; appears truncated

Related topics