Renewal : another client-lacks-sufficient-authorization

Help
1

Hello,

First, thanks a lot to Let’s Encrypt !

As stated in the title, this is another client-lacks-sufficient-authorization problem.
I have read the posts with similar topic, and fixed accordingly when it seemed fit.

My domain is: www.alchimie-web.com (plus a couple other domain.alchimie-web.com)

I first issued the certificate with Docker certbot, in manual mode. It worked like a charm.

The webserver is an nginx Docker container dedicated to the cert renewal (I’m leaving it on so you guys can test for yourselves).
It serves nothing except the default nginx page and the .well-known/acme-challenge dir with a test file to make sure it is accessible : http://www.alchimie-web.com/.well-known/acme-challenge/test.html
Il also checked with a plain text file, and it is served correctly as well.

Here’s the nginx log when I hit the test URL :

123.123.123.123 - - [28/Aug/2018:10:54:10 +0000] "GET /.well-known/acme-challenge/test.html HTTP/1.1" 200 46 "-" "Mozilla/5.0 (Windows NT 6.3; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/68.0.3440.106 Safari/537.36" "-"

Now the cert has expired, and I’m trying to renew it, with this command :

docker run --rm --name certbot \
    -v '/home/etc/letsencrypt:/etc/letsencrypt' \
    -v '/var/lib/letsencrypt:/var/lib/letsencrypt' \
    -v "/tmp/acme-challenge:/tmp/acme-challenge:rw" \
    certbot/certbot renew --webroot \
    --webroot-path "tmp/acme-challenge" \
    --dry-run

The directory /tmp/acme-challenge is correctly mapped to nginx container’s /usr/share/nginx/html/.well-known/acme-challenge and to make things easy to test it’s all chmod 777

It produced this output:

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/atanor.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Plugins selected: Authenticator webroot, Installer None
Renewing an existing certificate
/usr/local/lib/python2.7/site-packages/josepy/jwa.py:107: CryptographyDeprecationWarning: signer and verifier have been deprecated. Please use sign and verify instead.
  signer = key.signer(self.padding, self.hash)
Performing the following challenges:
http-01 challenge for alchimie-web.com
http-01 challenge for cloud.alchimie-web.com
http-01 challenge for gitea.alchimie-web.com
http-01 challenge for jenkins.alchimie-web.com
http-01 challenge for pga.alchimie-web.com
http-01 challenge for pma.alchimie-web.com
http-01 challenge for sonar.alchimie-web.com
http-01 challenge for www.alchimie-web.com
Using the webroot path /tmp/.well-known/acme-challenge for all unmatched domains.
Waiting for verification...
Cleaning up challenges
Attempting to renew cert (atanor) from /etc/letsencrypt/renewal/atanor.conf produced an unexpected error: Failed authorization procedure. www.alchimie-web.com (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://www.alchimie-web.com/.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
(skip same error for all domains)
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/atanor/fullchain.pem (failure)

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/atanor/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: www.alchimie-web.com
   Type:   unauthorized
   Detail: Invalid response from
   http://www.alchimie-web.com/.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro:
   "<html>
   <head><title>404 Not Found</title></head>
   <body bgcolor="white">
   <center><h1>404 Not Found</h1></center>
   <hr><center>"
(skip same output for all domains)
   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

I have no AAAA record for the domain(s) and the A record seems correct.

I checked /etc/letsencrypt/renewal/atanor.conf (for webroot mode instead of manual I used to install the cert) :

# renew_before_expiry = 30 days
version = 0.23.0
archive_dir = /etc/letsencrypt/archive/atanor
cert = /etc/letsencrypt/live/atanor/cert.pem
privkey = /etc/letsencrypt/live/atanor/privkey.pem
chain = /etc/letsencrypt/live/atanor/chain.pem
fullchain = /etc/letsencrypt/live/atanor/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
account = abcdef123456
manual_public_ip_logging_ok = True

The certbot container is able to reach through to nginx, as it appears in nginx log :

66.133.109.36 - - [28/Aug/2018:11:24:05 +0000] "GET /.well-known/acme-challenge/jocAHKSQzdvxHQ1fDG47i8H-rSbm2v8HdPudjQT5K_o HTTP/1.1" 404 170 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
2018/08/28 11:24:05 [error] 7#7: *67 open() "/usr/share/nginx/html/.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro" failed (2: No such file or directory), client: 52.29.173.72, server: *.alchimie-web.com, request: "GET /.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro HTTP/1.1", host: "www.alchimie-web.com"
34.213.106.112 - - [28/Aug/2018:11:24:05 +0000] "GET /.well-known/acme-challenge/jocAHKSQzdvxHQ1fDG47i8H-rSbm2v8HdPudjQT5K_o HTTP/1.1" 404 170 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" "-"
2018/08/28 11:24:05 [error] 7#7: *69 open() "/usr/share/nginx/html/.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro" failed (2: No such file or directory), client: 13.58.30.69, server: *.alchimie-web.com, request: "GET /.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro HTTP/1.1", host: "www.alchimie-web.com"

It seems to me the challenge files are simply not created. They’re not in /tmp/acme-challenge, but certbot seems to do some cleanup (Cleaning up challenges), so I can’t tell…

Is there a debug mode that would prevent certbot / letcert from cleaning the challenge files, so I can check that they are actually created (though I suppose it would throw an error if it wasn’t able to create them in the first place) ?

Also, I noticed the IPs in the nginx log a bit puzzling :
66.133.109.36 - - [28/Aug/2018:11:24:05 +0000] “GET /.well-known/acme-challenge/jocAHKSQzdvxHQ1fDG47i8H-rSbm2v8HdPudjQT5K_o HTTP/1.1” 404 170 “-” “Mozilla/5.0 (compatible; Let’s Encrypt validation server; +https://www.letsencrypt.org)” “-”
2018/08/28 11:24:05 [error] 7#7: *67 open() “/usr/share/nginx/html/.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro” failed (2: No such file or directory), client: 52.29.173.72, server: *.alchimie-web.com, request: “GET /.well-known/acme-challenge/HOyN2fOK-00WWJkf_xrv1zS28M1ufYUInsJ4wLNrbro HTTP/1.1”, host: “www.alchimie-web.com”

Anybody got a clue ?

Thanks in advance for your help.

2

I tried to pass this option :

docker run --rm -it \
     --name certbot \
     -v '/home/etc/letsencrypt:/etc/letsencrypt' \
     -v '/var/lib/letsencrypt:/var/lib/letsencrypt' \
     -v "/tmp/acme-challenge:/tmp/acme-challenge:rw" \
     certbot/certbot renew --webroot \
     --webroot-path "/tmp/acme-challenge" \
     --debug-challenges -v \
    --dry-run

And I do get an intermediary message :

-------------------------------------------------------------------------------
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
-------------------------------------------------------------------------------
Cleaning up challenges

But it doesn't wait for me to press continue, it goes on anyway... so I still can't check if my challenges are there.

With the -v option, I get more output. Sample for www.alchimie-web.com :

certbot asking for a challenge token :

Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549450:
{
  "protected": "eyJub25jZSI6ICJ5V2NHcTBVOU9VVEpfRkRIalFCcjVhU0pPTGZJUTl4UksxU0s5WUl3cUdzIiwgInVybCI6ICJodHRwczovL2FjbWUtc3RhZ2luZy12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2NoYWxsZW5nZS9nUlliMTdyMlZ6VE5hTHhkNFpfNXMteVdvTXlLbTJWUU5fWXo2cFl4bXc4LzE2NDU0OTQ1MCIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hY2N0LzY4MjIxMTQiLCAiYWxnIjogIlJTMjU2In0", 
  "payload": "ewogICJrZXlBdXRob3JpemF0aW9uIjogImJwVm9qbUNTeXVyMENPaHQ4a3BURmxkekVoNWxxWVdyb0FVU0Nka1lpNjQua016OXdJM1JvVjNZcF84dzl6elZvU1FrNldobkVDNzB3TzZCa2tpMTczdyIsIAogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9", 
  "signature": "EHPMjl7B0J8_dUgYEmarJ4O4C9Zpi_VQwPvukKW527-vGSMG3yKEuKdHAiW2ZN28w4nm4LPW4YrApUIFfYIxRIyLPihYHBc3VfcROejlsN2r06JruW87HNY60Asb7sU_J7JhdB8aCcL18HuWul-m9k3xenQIbmuQ9-oR8zjomMr-V42pontMemdrX_KqhsSqlM2kIieGA0LwRFGGVAIB2OgYEmCtEcJjsTiMZ6sESX0RcOw-VNotYe3OkxzrEhtONOW8ULbYP1CtEyCMk43WDKY_RFkSy59RarmQwuelWPaUxgSSOYKNDjsBSiHxuoWnu_HW6j6PtKR9IJ-63jg4Wg"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549450 HTTP/1.1" 200 230
Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 230
Boulder-Requester: 6822114
Link: <https://acme-staging-v02.api.letsencrypt.org/acme/authz/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549450
Replay-Nonce: Lqw81232Va7R1-0KJ6QnUVFKgLCW59mQpAuicNkocoo
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 28 Aug 2018 13:06:20 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 28 Aug 2018 13:06:20 GMT
Connection: keep-alive
{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549450",
  "token": "bpVojmCSyur0COht8kpTFldzEh5lqYWroAUSCdkYi64"
}
Storing nonce: Lqw81232Va7R1-0KJ6QnUVFKgLCW59mQpAuicNkocoo

certbot trying to authenticate www.alchimie-web.com

Sending GET request to https://acme-staging-v02.api.letsencrypt.org/acme/authz/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8.
https://acme-staging-v02.api.letsencrypt.org:443 "GET /acme/authz/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8 HTTP/1.1" 200 1805
Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 1805
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Tue, 28 Aug 2018 13:06:25 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Tue, 28 Aug 2018 13:06:25 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "www.alchimie-web.com"
  },
  "status": "invalid",
  "expires": "2018-09-04T13:06:16Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549449",
      "token": "oAtfhiguq91hpHOJGGkQPA_3-Gn9xhEGCbadL0wZxUs"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from http://www.alchimie-web.com/.well-known/acme-challenge/bpVojmCSyur0COht8kpTFldzEh5lqYWroAUSCdkYi64: \"\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody bgcolor=\"white\"\u003e\r\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003e\"",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549450",
      "token": "bpVojmCSyur0COht8kpTFldzEh5lqYWroAUSCdkYi64",
      "validationRecord": [
        {
          "url": "http://www.alchimie-web.com/.well-known/acme-challenge/bpVojmCSyur0COht8kpTFldzEh5lqYWroAUSCdkYi64",
          "hostname": "www.alchimie-web.com",
          "port": "80",
          "addressesResolved": [
            "37.59.46.195"
          ],
          "addressUsed": "37.59.46.195"
        }
      ]
    },
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549451",
      "token": "nXmyw8m_qn-0uLEt2axrsDxtsgu4gzQqlpB8yRO4sjE"
    }
  ]
}
tls-alpn-01 was not recognized, full message: {u'status': u'invalid', u'url': u'https://acme-staging-v02.api.letsencrypt.org/acme/challenge/gRYb17r2VzTNaLxd4Z_5s-yWoMyKm2VQN_Yz6pYxmw8/164549451', u'token': u'nXmyw8m_qn-0uLEt2axrsDxtsgu4gzQqlpB8yRO4sjE', u'type': u'tls-alpn-01'}
Reporting to user: The following errors were reported by the server:
(...)
Domain: www.alchimie-web.com
Type:   unauthorized
Detail: Invalid response from http://www.alchimie-web.com/.well-known/acme-challenge/bpVojmCSyur0COht8kpTFldzEh5lqYWroAUSCdkYi64: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>"

And also the cleanup debug :

Calling registered functions
Cleaning up challenges
Removing /tmp/acme-challenge/.well-known/acme-challenge/HXBJCZZ9HMVuv9q5wfObgOAwyAFbULMOLbk4TKod530
Removing /tmp/acme-challenge/.well-known/acme-challenge/rT6_31SAANv5LxASc2d1e-EXmZOrC5yT-HqcPv_mqqk
Removing /tmp/acme-challenge/.well-known/acme-challenge/XUWECP4K2SSRrp9Krkg_xjk-KxIddD8ZgINavO727ww
Removing /tmp/acme-challenge/.well-known/acme-challenge/fnGbSoQ2zAfCwdi8U1tQ-GHwPlKlWawHgccsTzaIDK0
Removing /tmp/acme-challenge/.well-known/acme-challenge/A2y7roPqJXzCV9M98-vO0NYPnFBqIPb5QAN-z5NTQMQ
Removing /tmp/acme-challenge/.well-known/acme-challenge/nOJ8oR0yjteTk_gEzmavbOCpN0toCQBBWA1oAuH_5mU
Removing /tmp/acme-challenge/.well-known/acme-challenge/6KnBhfPRLxFhkkuTmpbIvmdhnreaBppRsM3HsbyeS2M
Removing /tmp/acme-challenge/.well-known/acme-challenge/gw6d9wz3U7jHgGj6rqrbWeyPGbdyL8L_iJL1cu7M664
All challenges cleaned up

So that was my problem : the webroot-path must point to the document_root of the server, not to the actual location of the challenge files, which will be in webroot_path/.well-known/acme-challenge

(and it's actually explained in the doc... my bad !)

. In addition, you’ll need to specify --webroot-path or -w with the top-level directory (“web root”) containing the files served by your webserver.
User Guide — Certbot 2.7.0.dev0 documentation

3

So the full solution to this docker / nginx / certbot problem :

on the host, use a tmp directory as document_root, writable by everyone

mkdir /tmp/nginx && chmod 777 /tmp/nginx

launch a temporary nginx container

docker run --rm \
  --name nginx-cert \
  -v /tmp/nginx:/usr/share/nginx/html \
  -v /tmp/nginx.conf:/etc/nginx/conf.d/default.conf:rw \
  -p 80:80 
  nginx:latest

one will need to adjust the server_name directive in /tmp/nginx.conf(path on the host)

server {
    listen       80;
    server_name  *.yourdomain.com;
   # (...)
}

launch the certbot container

docker run --rm --name certbot \
    -v '/home/etc/letsencrypt:/etc/letsencrypt' \
    -v '/var/lib/letsencrypt:/var/lib/letsencrypt' \
    -v '/tmp/nginx:/tmp/nginx:rw" \
    certbot/certbot renew --webroot \
    --webroot-path /tmp/nginx \
    --dry-run

tada :

Waiting for verification...
Cleaning up challenges

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/atanor/fullchain.pem
-------------------------------------------------------------------------------

-------------------------------------------------------------------------------
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

Congratulations, all renewals succeeded. The following certs have been renewed:
  /etc/letsencrypt/live/atanor/fullchain.pem (success)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
-------------------------------------------------------------------------------

stop the nginx container and cleanup

docker stop nginx-cert && rm -rf /tmp/nginx*

Hope that helps, and sorry if it’s a bit verbose

1 Like
4

I tihnk the underlying problem is that Certbot would create .well-known/acme-challenge within the webroot. So if the specified webroot is mapped to /usr/share/nginx/html/.well-known/acme-challenge, Certbot would create /usr/share/nginx/html/.well-known/acme-challenge/.well-known/acme-challenge, which isn't what you intended.

5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.