No Such Authorization

Help
#1

TL;DR
I just want to know what this output from certbot means:

The request message was malformed :: No such authorization

Furthermore, if an individual domain being requested is to blame, how might I learn that? Can LE’s error handling for this case be improved to include domain/hostname?

Full Explanation

The following command has been working without incident for ~8 months and began failing on July 4th (4 days ago).

    letsencrypt certonly --webroot --staging --csr /path/to/my.csr -w /path/to/html -d www.somedoman1.com -d www.somedomain2.com

The domains above are just example. However, it only seems to fail when certain domains are used; the problem appears to be with the domains and we still succeed with most domains. However, when there has been a failure with a domain in the past, that has always been evident from the log message we get from certbot. For example:

Failed authorization procedure. www.bsd100.org (http-01): urn:acme:error:unauthorized

^^ That is an example of 1 of many errors we get which contains the causal domain and allows our automated scripts to parse out the problem domain retry w/o it. Those scripts recover from any error like this reliably.

However, the failure output we’re now getting is:

The request message was malformed :: No such authorization

Which puts us in a difficult position for diagnosing. Our recovery scripts are unable to recover. I believe my next step is to determine a causal domain by manual process of elimination.

Here is the actual command with full list of domains…

    letsencrypt certonly --webroot --staging --csr /path/to/my.csr -w /path/to/html -d pleasanthillschools.com -d admin.alhs.nyc -d admin.ardmore.lakeviewpublicschools.org -d admin.avalongardenselementary.com -d admin.chowchillahigh.k12.ca.us -d admin.cpdlf.org -d admin.dcsedu.com -d admin.dhs.dcsedu.com -d admin.dms.dcsedu.com -d admin.elcaonline.org -d admin.es.pleasanthillschools.com -d admin.greenwood.lakeviewpublicschools.org -d admin.greenwoodscharter.org -d admin.harmon.lakeviewpublicschools.org -d admin.is.pleasanthillschools.com -d admin.jefferson.lakeviewpublicschools.org -d admin.lakeviewhs.lakeviewpublicschools.org -d admin.lakeviewpublicschools.org -d admin.midwoodhighschool.org -d admin.ms.pleasanthillschools.com -d admin.onwardleaders.org -d admin.princeton.lakeviewpublicschools.org -d admin.ps.pleasanthillschools.com -d admin.refugiohs.org -d admin.st-augustine.org -d admin.stdominicschool.org -d admin.sunriseacademy.net -d admin.syvpirates.org -d admin.syvuhsd.org -d admin.usj.dcsedu.com -d admin.wes.dcsedu.com -d admin.wheat.lakeviewpublicschools.org -d alhs.nyc -d ardmore.lakeviewpublicschools.org -d avalongardenselementary.com -d chowchillahigh.k12.ca.us -d coretca.org -d dcsedu.com -d dhs.dcsedu.com -d dms.dcsedu.com -d edisonchargers.com -d elcaonline.org -d es.pleasanthillschools.com -d greenwood.lakeviewpublicschools.org -d harmon.lakeviewpublicschools.org -d is.pleasanthillschools.com -d jefferson.lakeviewpublicschools.org -d lakeviewhs.lakeviewpublicschools.org -d lakeviewpublicschools.org -d losarbolesmiddle.org -d midwoodhighschool.org -d onwardleaders.org -d panolacharterschool.net -d princeton.lakeviewpublicschools.org -d ps.pleasanthillschools.com -d refugiohs.org -d school.stpaulannarbor.org -d st-augustine.org -d staugschool.org -d stdominicschool.org -d sunriseacademy.net -d syvpirates.org -d syvuhsd.org -d torahdayschoolofphoenix.com -d usj.dcsedu.com -d wes.dcsedu.com -d wheat.lakeviewpublicschools.org -d www.alhs.nyc -d www.avalongardenselementary.com -d www.chowchillahigh.k12.ca.us -d www.coretca.org -d www.dcsedu.com -d www.edisonchargers.com -d www.elcaonline.org -d www.greenwoodscharter.org -d www.lakeviewpublicschools.org -d www.losarbolesmiddle.org -d www.midwoodhighschool.org -d www.onwardleaders.org -d www.panolacharterschool.net -d www.pleasanthillschools.com -d www.refugiohs.org -d www.st-augustine.org -d www.stdominicschool.org -d www.sunriseacademy.net -d www.syvpirates.org -d www.syvuhsd.org

Further mandatory diagnostic info…

My web server is (include version):
Apache 2

The operating system my web server runs on is (include version):
Ubuntu

I can login to a root shell on my machine (yes or no, or I don’t know):
Yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
Very old certbot: letsencrypt 0.4.1 CLI, but with the webroot plugin

#2

Hi @lancedolan

are you sure your “my.csr” matches your long domain list?

And you should really update your 0.4 version.

#3

Looking at my certbot logs…

certbot sends a POST to /acme/new-authz and gets back a 201 containing references to this auth with ID v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8

2019-07-08 19:23:06,194:DEBUG:root:Received <Response [201]>. Headers: {'Content-Length': '1394', 'Expires': 'Mon, 08 Jul 2019 19:23:06 GMT', 'Strict-Transport-Security': 'max-age=604800', 'Server': 'nginx', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Connection': 'keep-alive', 'Link': '<https://acme-staging.api.letsencrypt.org/acme/new-cert>;rel="next"', 'Location': 'https://acme-staging.api.letsencrypt.org/acme/authz/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8', 'Pragma': 'no-cache', 'Boulder-Requester': '1576757', 'Date': 'Mon, 08 Jul 2019 19:23:06 GMT', 'X-Frame-Options': 'DENY', 'Content-Type': 'application/json', 'Replay-Nonce': '-pkPdJ41COFKNTVAw2Ps8GhcTW8aJ4rfINrxUQAD30g'}. Content: '{\n  "identifier": {\n    "type": "dns",\n    "value": "admin.rallsisd.org"\n  },\n  "status": "valid",\n  "expires": "2019-07-26T02:06:49Z",\n  "challenges": [\n    {\n      "type": "tls-alpn-01",\n      "status": "pending",\n      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8/325064510",\n      "token": "tB_7kp9B5a4Q8Y1LbTzTCSHV37lngtB506nppEpLYCw"\n    },\n    {\n      "type": "http-01",\n      "status": "valid",\n      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8/325064511",\n      "token": "SHJd6FyEI0bMkpksWDUrUacIkfOAZXCCkC44DDJGDLg",\n      "validationRecord": [\n        {\n          "url": "http://admin.rallsisd.org/.well-known/acme-challenge/SHJd6FyEI0bMkpksWDUrUacIkfOAZXCCkC44DDJGDLg",\n          "hostname": "admin.rallsisd.org",\n          "port": "80",\n          "addressesResolved": [\n            "151.101.68.80",\n            "2a04:4e42:10::80"\n          ],\n          "addressUsed": "2a04:4e42:10::80"\n        }\n      ]\n    },\n    {\n      "type": "dns-01",\n      "status": "pending",\n      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8/325064512",\n      "token": "c_xcmYC6Q7GXm8a0IHVQVzYwjZ8nLlNcyoayoDWisLU"\n    }\n  ],\n  "combinations": [\n    [\n      0\n    ],\n    [\n      1\n    ],\n    [\n      2\n    ]\n  ]\n}'

30 seconds later, it appears LE gives a 404 for that auth resource?

2019-07-08 19:23:34,531:DEBUG:root:Sending GET request to https://acme-staging.api.letsencrypt.org/acme/authz/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8. args: (), kwargs: {}
2019-07-08 19:23:34,532:INFO:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
2019-07-08 19:23:34,731:DEBUG:requests.packages.urllib3.connectionpool:"GET /acme/authz/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8 HTTP/1.1" 404 94
2019-07-08 19:23:34,733:DEBUG:root:Received <Response [404]>. Headers: {'Content-Length': '94', 'Expires': 'Mon, 08 Jul 2019 19:23:34 GMT', 'Server': 'nginx', 'Connection': 'keep-alive', 'Pragma': 'no-cache', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Date': 'Mon, 08 Jul 2019 19:23:34 GMT', 'Content-Type': 'application/problem+json', 'Replay-Nonce': '7QLmIWiwRtM8xBztYY0ncluXXIaf2ZVfWARx2j76SJM'}. Content: '{\n  "type": "urn:acme:error:malformed",\n  "detail": "No such authorization",\n  "status": 404\n}'
2019-07-08 19:23:34,733:DEBUG:acme.client:Received response <Response [404]> (headers: {'Content-Length': '94', 'Expires': 'Mon, 08 Jul 2019 19:23:34 GMT', 'Server': 'nginx', 'Connection': 'keep-alive', 'Pragma': 'no-cache', 'Cache-Control': 'max-age=0, no-cache, no-store', 'Date': 'Mon, 08 Jul 2019 19:23:34 GMT', 'Content-Type': 'application/problem+json', 'Replay-Nonce': '7QLmIWiwRtM8xBztYY0ncluXXIaf2ZVfWARx2j76SJM'}): '{\n  "type": "urn:acme:error:malformed",\n  "detail": "No such authorization",\n  "status": 404\n}'
2019-07-08 19:23:34,733:INFO:letsencrypt.auth_handler:Cleaning up challenges
2019-07-08 19:23:34,734:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/z6-qfurvBKmDWjE-ZdAQw_-6CXEfO-p_oSuPcxWTWD4
2019-07-08 19:23:34,734:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/0ZbCqgQ_LE0SW49YvfRWzhEoaabpqsMHObHOj6Ywl-s
2019-07-08 19:23:34,734:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/Xz51mQ2bw_kUcmuYHEDyIU0ZXgHy8wn5kyslyqX-3Uk
2019-07-08 19:23:34,735:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/4I7f06ncD670TLtcABmsUypLFGek_7fUMojEPZbAAAg
2019-07-08 19:23:34,735:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/9f-I7sFkMgelDDMRPgFvjpVwteJ2TsCehk8B5ZUltDY
2019-07-08 19:23:34,735:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/j7kYBJA-ZMTxrLm00E4TLo260o__r6ZFm22l6fvt3Z0
2019-07-08 19:23:34,735:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/ruOCb5ptnS_yp_WHa0RvoP3RH_Lr7eCF3QtH467tviM
2019-07-08 19:23:34,735:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/hCDSXN_MK_yj3MHEu_dWHkEAvjWojxIAl8Ua37IuskA
2019-07-08 19:23:34,736:DEBUG:letsencrypt.plugins.webroot:Removing /var/www/html/.well-known/acme-challenge/Ro6URwIo2hw_CkoQ7JSjXRlIvJnTIlAAqBxmf8qleG0
--
  File "/usr/lib/python2.7/dist-packages/letsencrypt/auth_handler.py", line 186, in _poll_challenges
    domain, chall_update[domain])
  File "/usr/lib/python2.7/dist-packages/letsencrypt/auth_handler.py", line 215, in _handle_check
    self.authzr[domain], _ = self.acme.poll(self.authzr[domain])
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 290, in poll
    response = self.net.get(authzr.uri)
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 627, in get
    self._send_request('GET', url, **kwargs), content_type=content_type)
  File "/usr/lib/python2.7/dist-packages/acme/client.py", line 568, in _check_response
    raise messages.Error.from_json(jobj)
Error: urn:acme:error:malformed :: The request message was malformed :: No such authorization

Therefore, I don’t think it is a malformed request at all? Malformed request should be 400? This is just a matter of the auth no longer existing for some reason?

Anybody understand this better?

are you sure your “my.csr” matches your long domain list?

Very confident, yes. If you’re strongly suspicious that this is a root cause, I can take steps to monitor the CSR that is generated and confirm. Keep in mind the same algorithm has been producing the CSR for 2+ years without ever once failing, and is still correctly generating the CSR at this very moment as long as we enter in a different set of domains than those ~70 domains mentioned in this thread.

I believe I’m going to begin bisecting the list of 70 domains to try to prove a causal domain…

#4

The result is curious.

Do you have the order url?

I’ve never used v.1 (startet with v.2), perhaps the order url (the first url from Letsencrypt) has another name.

PS: The idea: There are other challenges, one is already invalid, so the order is invalid

#5

There are no orders in ACME v1

1 Like
#6

There appears to be a legit problem in staging.

Check this out: our script we use for clearing authz from staging environment is hitting the same error, and it doesn’t use certbot at all. It looks at all existing “authz” and attempts to “accept” each in order to clear them.

Here’s the GO script:

Output:

Checking 226 authzs to see if they are pending ...
Failed to fetch authz https://acme-staging.api.letsencrypt.org/acme/authz/v2mvy-6xXeJANcWyrHYtqwhb5QsqxZ6ZU_iGU3y8KD8: 404 urn:acme:error:malformed: No such authorization
Failed to fetch authz https://acme-staging.api.letsencrypt.org/acme/authz/v2: 404 urn:acme:error:malformed: No such authorization

This is not normal output for that script, we’ve never seen it do that before

2 Likes
#7

I agree. I’m working on asking our SRE to revert the change that is at fault.

Please, please, please update your Certbot ASAP. This feedback is delivered consistently on each of your forum threads and it continues to be ignored. In this case it is unrelated to your problem at hand but this is rapidly becoming untrue with time.

1 Like
#8

What are chances this is due to that auth key beginning with v2 ? This means that the very beginning of the request is /acme/authz/v2 which is the prefix to hit the v2 acme, right?

#9

That is indeed the cause of the problem here.

No, we use an entirely different directory URL for the V2 API. It isn’t a path component. The “v2” is causing problems because of an in-progress rework of our authorization storage that we’re calling “v2 authorizations” internally.

I will share a Boulder issue with details shortly. The feature flag at fault is being reverted in staging.

2 Likes
#10

This has been done now.

#11

Yes, you’re right. I expect that feedback 100% of the time I post on this forum, and I get it that often. Given how helpful and responsive you all are (especially you, cpu), I’d like to upgrade simply to satisfy you guys and to do you a favor. In fact we started a small project to upgrade it once before and hit some issues which I don’t recall and we didn’t have the time-budget to solve. Also, we’ve recently purchased a product that will replace our SSL solution entirely by end of year. Given that upgrade is non-trivial and the system being upgraded will be decommissioned entirely, it’s difficult to make the argument within my organization to spend time on it.

I’d like to mention, without ruffling feathers, that our outdated client hasn’t caused any issues that I’m aware of to date. Each time I post, the culprit has been something else, and our outdated client is criticized along the way.

As usual, a very sincere and hearty thank you for you help :smiley: :smiley:

2 Likes
#12
1 Like
#13

@lancedolan It’s heartening to hear you folks started working on an upgrade and are evaluating replacement systems. Thanks for sharing those details. I definitely understand what it’s like to work in a resource constrained environment where it can be challenging to prioritize projects.

Thanks for reporting the problem. Apologizes for the bug!

1 Like
#14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.