Certbot renewal fails with "The client lacks sufficient authorization" error

Please fill out the fields below so we can help you better.

My domain is: domain name has been replaced with [mydomain] in all instances

I ran this command:

certbot --dry-run renew

It produced this output:

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.mydomain.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.mydomain.net
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /usr/share/nginx/roundcubemail/.well-known/acme-challenge
Attempting to renew cert from /etc/letsencrypt/renewal/mail.mydomain.net.conf produced an unexpected error: Failed authorization procedure. mail.mydomain.net (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.mydomain.net/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk: "<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>". Skipping.

My web server is (include version):

nginx version: nginx/1.11.9

The operating system my web server runs on is (include version):

Ubuntu 16.04.2 x64

My hosting provider, if applicable, is:

DigitalOcean

I can login to a root shell on my machine (yes or no, or I don’t know):

yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):

no

Contents of /etc/letsencrypt/renewal/mail.mydomain.net.conf:

# renew_before_expiry = 30 days
version = 0.12.0
archive_dir = /etc/letsencrypt/archive/mail.mydomain.net
cert = /etc/letsencrypt/live/mail.mydomain.net/cert.pem
privkey = /etc/letsencrypt/live/mail.mydomain.net/privkey.pem
chain = /etc/letsencrypt/live/mail.mydomain.net/chain.pem
fullchain = /etc/letsencrypt/live/mail.mydomain.net/fullchain.pem

# Options used in the renewal process
[renewalparams]
authenticator = webroot
installer = None
webroot_path = /usr/share/nginx/roundcubemail
account = [account #]
[[webroot_map]]
mail.mydomain.net = /usr/share/nginx/roundcubemail

Contents of /etc/nginx/sites-available/roundcubemail:

server {
        listen 80;
        server_name mail.mydomain.net;

        include /etc/nginx/snippets/letsencrypt-acme-challenge.conf;

        location / {
                return 301 https://$host$request_uri;
        }
}

# HTTPS server
server {
        listen 443 ssl http2 default_server;
        server_name mail.mydomain.net;
        root /usr/share/nginx/roundcubemail;
        index index.html index.php;
        autoindex off;

        ssl on;
        include /etc/nginx/snippets/ssl-mail.mydomain.net.conf;
        include /etc/nginx/snippets/ssl-params.conf;

        location ~ ^/(README|INSTALL|LICENSE|CHANGELOG|UPGRADING)$ {
                deny all;
        }

        location ~ ^/(config|bin|SQL|logs|temp)/ {
                deny all;
        }

        location ~ ^/.*\.php {
                try_files $uri =404;
                include fastcgi_params;
                fastcgi_pass php7.0-fpm-sock;
                fastcgi_param HTTPS on;
                fastcgi_param SCRIPT_FILENAME $document_root$fastcgi_script_name;
                fastcgi_intercept_errors on;
        }
}

Contents of /etc/nginx/snippets/letsencrypt-acme-challenge.conf

#############################################################################
# Configuration file for Let's Encrypt ACME Challenge location
# This file is already included in listen_xxx.conf files.
# Do NOT include it separately!
#############################################################################
#
# This config enables to access /.well-known/acme-challenge/xxxxxxxxxxx
# on all our sites (HTTP), including all subdomains.
# This is required by ACME Challenge (webroot authentication).
# You can check that this location is working by placing ping.txt here:
# /var/www/letsencrypt/.well-known/acme-challenge/ping.txt
# And pointing your browser to:
# http://xxx.domain.tld/.well-known/acme-challenge/ping.txt
#
# Sources:
# https://community.letsencrypt.org/t/howto-easy-cert-generation-and-renewal-with-nginx/3491
#
#############################################################################

# Rule for legitimate ACME Challenge requests (like /.well-known/acme-challenge/xxxxxxxxx)
# We use ^~ here, so that we don't check other regexes (for speed-up). We actually MUST cancel
# other regex checks, because in our other config files have regex rule that denies access to files with dotted names.
location ^~ /.well-known/acme-challenge/ {

    # Set correct content type. According to this:
    # https://community.letsencrypt.org/t/using-the-webroot-domain-verification-method/1445/29
    # Current specification requires "text/plain" or no content header at all.
    # It seems that "text/plain" is a safe option.
    default_type "text/plain";

    # This directory must be the same as in /etc/letsencrypt/cli.ini
    # as "webroot-path" parameter. Also don't forget to set "authenticator" parameter
    # there to "webroot".
    # Do NOT use alias, use root! Target directory is located here:
    # /var/www/common/letsencrypt/.well-known/acme-challenge/
    root         /usr/share/nginx/roundcubemail;
}

# Hide /acme-challenge subdirectory and return 404 on all requests.
# It is somewhat more secure than letting Nginx return 403.
# Ending slash is important!
location = /.well-known/acme-challenge/ {
    return 404;
}

Directory listing of /usr/share/nginx/roundcubemail/.well-known/acme-challenge:

root@iorek:/usr/share/nginx/roundcubemail/.well-known/acme-challenge# ls -al
total 16
drwxr-xr-x 2 root root 4096 Jun 30 13:37 .
drwxr-xr-x 3 root root 4096 Jun 30 09:03 ..
-rw-r--r-- 1 root root    5 Jun 30 09:04 index.html
-rw-r--r-- 1 root root    5 Jun 30 09:04 test.txt

test.txt is reachable from a remote machine using http:

the validation file does get created in /usr/share/nginx/roundcubemail/.well-known/acme-challenge during the certbot update (click on image below to view GIF of “watch -n 1 ls -al”):

Content of /var/log/letsencrypt/letsencrypt.log:

2017-06-30 18:05:55,975:DEBUG:certbot.storage:Should renew, less than 30 days before certificate expiry 2017-06-29 23:08:00 UTC.
2017-06-30 18:05:55,975:INFO:certbot.renewal:Cert is due for renewal, auto-renewing...
2017-06-30 18:05:55,975:DEBUG:certbot.plugins.selection:Requested authenticator webroot and installer None
2017-06-30 18:05:55,976:DEBUG:certbot.plugins.selection:Single candidate plugin: * webroot
Description: Place files in webroot directory
Interfaces: IAuthenticator, IPlugin
Entry point: webroot = certbot.plugins.webroot:Authenticator
Initialized: <certbot.plugins.webroot.Authenticator object at 0x7f65121a6e50>
Prep: True
2017-06-30 18:05:55,976:DEBUG:certbot.plugins.selection:Selected authenticator <certbot.plugins.webroot.Authenticator object at 0x7f65121a6e50> and installer None
2017-06-30 18:05:55,979:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(status=None, contact=(), agreement=u'https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf', key=JWKRSA(key=<ComparableRSAKey(<cryptography.hazmat.backends.openssl.rsa._RSAPublicKey object at 0x7f65121b0d90>)>)), uri=u'https://acme-staging.api.letsencrypt.org/acme/reg/1674310', new_authzr_uri=u'https://acme-staging.api.letsencrypt.org/acme/new-authz', terms_of_service=u'https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf'), 54a8c2d3be522d1952c8802b06569a3a, Meta(creation_host=u'iorek.mydomain.net', creation_dt=datetime.datetime(2017, 3, 25, 3, 9, 29, tzinfo=<UTC>)))>
2017-06-30 18:05:55,980:DEBUG:acme.client:Sending GET request to https://acme-staging.api.letsencrypt.org/directory.
2017-06-30 18:05:55,981:DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging.api.letsencrypt.org
2017-06-30 18:05:56,073:DEBUG:requests.packages.urllib3.connectionpool:https://acme-staging.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 473
2017-06-30 18:05:56,074:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Content-Length: 473
Boulder-Request-Id: ua6xwZeAfuPKdVOjHQZNwWefswrNgEx_bElmR50jcao
Replay-Nonce: QFwMsu-isjOqazpmC-DMGpgmRsJOLM-uZUMsfjmsDG4
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Fri, 30 Jun 2017 18:05:56 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 30 Jun 2017 18:05:56 GMT
Connection: keep-alive

{
  "key-change": "https://acme-staging.api.letsencrypt.org/acme/key-change",
  "kxXYWJIR2JY": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "new-authz": "https://acme-staging.api.letsencrypt.org/acme/new-authz",
  "new-cert": "https://acme-staging.api.letsencrypt.org/acme/new-cert",
  "new-reg": "https://acme-staging.api.letsencrypt.org/acme/new-reg",
  "revoke-cert": "https://acme-staging.api.letsencrypt.org/acme/revoke-cert"
}
2017-06-30 18:05:56,074:INFO:certbot.main:Renewing an existing certificate
2017-06-30 18:05:56,076:DEBUG:acme.client:Requesting fresh nonce
2017-06-30 18:05:56,076:DEBUG:acme.client:Sending HEAD request to https://acme-staging.api.letsencrypt.org/acme/new-authz.
2017-06-30 18:05:56,141:DEBUG:requests.packages.urllib3.connectionpool:https://acme-staging.api.letsencrypt.org:443 "HEAD /acme/new-authz HTTP/1.1" 405 0
2017-06-30 18:05:56,142:DEBUG:acme.client:Received response:
HTTP 405
Server: nginx
Content-Type: application/problem+json
Content-Length: 91
Allow: POST
Boulder-Request-Id: qQZMqF5fHnGK3nQjvFduO2uyvKzQQ0zy4yto6fJCNc8
Replay-Nonce: jxnhHImV5lz7NwnbIu3_HKl5liFB4EUjCObPzpa5zK4
Expires: Fri, 30 Jun 2017 18:05:56 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 30 Jun 2017 18:05:56 GMT
Connection: keep-alive


2017-06-30 18:05:56,142:DEBUG:acme.client:Storing nonce: jxnhHImV5lz7NwnbIu3_HKl5liFB4EUjCObPzpa5zK4
2017-06-30 18:05:56,142:DEBUG:acme.client:JWS payload:
{
  "identifier": {
    "type": "dns",
    "value": "mail.mydomain.net"
  },
  "resource": "new-authz"
}
2017-06-30 18:05:56,146:DEBUG:acme.client:Sending POST request to https://acme-staging.api.letsencrypt.org/acme/new-authz:
{
  "header": {
    "alg": "RS256",
    "jwk": {
      "e": "AQAB",
      "kty": "RSA",
      "n": "xKcAGtZHWvCOhM4zgwClIZztTCd7g1-SQutMaeX3y5hY-jNGwrQIoRDq4QCZZBfOAXoq-VH3d4Q-W2ZtsQD09ea2_nViX4UfzwB7njaBQ-qYx7YUjRqwQR0IVMtyzMyThQ8ip_tW-WOt8WBs1MIY3xYZSt1vncP0Gp7irPETlHYKqehJlJpaf9DS9Mkf3pUNC5yT8Ksinqu3pSnZi2jA0ON7OO-TNVncw17cE-8SC78S22oP8LUGq9EyySC2rCSNiIfj6LSfjM0hCJZGJy1eo0P7cLya3NlU0Yn40sju84uC0S9H0iL7LHIOBpbb-A9vp0Q3bzIpuOp6diFmjB2vNQ"
    }
  },
  "protected": "eyJub25jZSI6ICJqeG5oSEltVjVsejdOd25iSXUzX0hLbDVsaUZCNEVVakNPYlB6cGE1eks0In0",
  "payload": "ewogICJpZGVudGlmaWVyIjogewogICAgInR5cGUiOiAiZG5zIiwgCiAgICAidmFsdWUiOiAibWFpbC5tb3Vra2kubmV0IgogIH0sIAogICJyZXNvdXJjZSI6ICJuZXctYXV0aHoiCn0",
  "signature": "TUbQhcJTtEMpYyujlClzXjqNqErls1Hsodpf_jJJgxku8IXr6ae0wWVzTwGT-TzdfwBHsK0LgWne55otVDVvnhu53IEoTsZCDI9RGD-W7uVrXWk14Wcz3IhJsYbnKXU5cTyT_78S7VOWd_RQ0salCVhV0Ncih3Ymzfe7qLSZwcpSb3oTYeY6dW27sDmci-3tlELQMV3k8mgeELdURJkA-0kBdWclwAbzybpRjfAqEDiUTOpijB_0L1gBO5D1j8wvgfsViVtL4vdD6pQt4L_0womaCo7RIovcoK0Fp0iv8NzXx8CewS9fjK8Vfr94Oa4p7-as3vagKqpNugHfarCLXQ"
}
2017-06-30 18:05:56,237:DEBUG:requests.packages.urllib3.connectionpool:https://acme-staging.api.letsencrypt.org:443 "POST /acme/new-authz HTTP/1.1" 201 1009
2017-06-30 18:05:56,238:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Content-Type: application/json
Content-Length: 1009
Boulder-Request-Id: IGC-sgge796MBgTq2GzpZETD1TjdCChPGC3l0fkFBwo
Boulder-Requester: 1674310
Link: <https://acme-staging.api.letsencrypt.org/acme/new-cert>;rel="next"
Location: https://acme-staging.api.letsencrypt.org/acme/authz/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI
Replay-Nonce: IjiVksKB5a4RAC9Kt_2uulbypCEgNOo30bg4LxHvWOo
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Expires: Fri, 30 Jun 2017 18:05:56 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 30 Jun 2017 18:05:56 GMT
Connection: keep-alive

{
  "identifier": {
    "type": "dns",
    "value": "mail.mydomain.net"
  },
  "status": "pending",
  "expires": "2017-07-07T18:05:56.190546331Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209413",
      "token": "06lCR7xJgbt8lRJQxP6ywNl3dTCv2g7OXamtWVU2NKM"
    },
    {
      "type": "tls-sni-01",
      "status": "pending",
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209414",
      "token": "BrUrrMTk6PvhaptEHDakgnCJ4Lbm5BT1wcKCq3M7V08"
    },
    {
      "type": "http-01",
      "status": "pending",
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209415",
      "token": "0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk"
    }
  ],
  "combinations": [
    [
      2
    ],
    [
      0
    ],
    [
      1
    ]
  ]
}
2017-06-30 18:05:56,239:DEBUG:acme.client:Storing nonce: IjiVksKB5a4RAC9Kt_2uulbypCEgNOo30bg4LxHvWOo
2017-06-30 18:05:56,239:INFO:certbot.auth_handler:Performing the following challenges:
2017-06-30 18:05:56,240:INFO:certbot.auth_handler:http-01 challenge for mail.mydomain.net
2017-06-30 18:05:56,240:DEBUG:certbot.plugins.webroot:Creating root challenges validation dir at /usr/share/nginx/roundcubemail/.well-known/acme-challenge
2017-06-30 18:05:56,243:DEBUG:certbot.plugins.webroot:Attempting to save validation to /usr/share/nginx/roundcubemail/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk
2017-06-30 18:05:56,244:INFO:certbot.auth_handler:Waiting for verification...
2017-06-30 18:05:56,244:DEBUG:acme.client:JWS payload:
{
  "keyAuthorization": "0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk.R_138jwlMfn-8DshdSuaxf72U7PSTNxLVjts6x4VX24",
  "type": "http-01",
  "resource": "challenge"
}
2017-06-30 18:05:56,247:DEBUG:acme.client:Sending POST request to https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209415:
{
  "header": {
    "alg": "RS256",
    "jwk": {
      "e": "AQAB",
      "kty": "RSA",
      "n": "xKcAGtZHWvCOhM4zgwClIZztTCd7g1-SQutMaeX3y5hY-jNGwrQIoRDq4QCZZBfOAXoq-VH3d4Q-W2ZtsQD09ea2_nViX4UfzwB7njaBQ-qYx7YUjRqwQR0IVMtyzMyThQ8ip_tW-WOt8WBs1MIY3xYZSt1vncP0Gp7irPETlHYKqehJlJpaf9DS9Mkf3pUNC5yT8Ksinqu3pSnZi2jA0ON7OO-TNVncw17cE-8SC78S22oP8LUGq9EyySC2rCSNiIfj6LSfjM0hCJZGJy1eo0P7cLya3NlU0Yn40sju84uC0S9H0iL7LHIOBpbb-A9vp0Q3bzIpuOp6diFmjB2vNQ"
    }
  },
  "protected": "eyJub25jZSI6ICJJamlWa3NLQjVhNFJBQzlLdF8ydXVsYnlwQ0VnTk9vMzBiZzRMeEh2V09vIn0",
  "payload": "ewogICJrZXlBdXRob3JpemF0aW9uIjogIjBzSGRZeDlEVHZHdjQ5MVhqSjE1c2RyejZybjNMbFV2Vm9sN0ZVZkpmSWsuUl8xMzhqd2xNZm4tOERzaGRTdWF4ZjcyVTdQU1ROeExWanRzNng0VlgyNCIsIAogICJ0eXBlIjogImh0dHAtMDEiLCAKICAicmVzb3VyY2UiOiAiY2hhbGxlbmdlIgp9",
  "signature": "uO5AGMUVhlCdPsjocth4q7g8BX_wsHBVYYIb465Mtm72QZyO-U5FRV-lg9WtOUnuGLkC-jwY5-hdhrR5i8dGYtGABwOR9-RA-fQex0ZdqLxf397STeaMMETVNwGt5hkNJ7Mfi6KCjJd35LLlbX5WkqWaonnGio2VIhyZgfXcWkDVZZ2m3tkIEQ0R9fF1amBrUrKkFSJhSkgXuzUuxCdvsjtdACXRewsDHB0K1jVsRUQzCOf6z-M0lQLQqk_g6tnzIgd8E292Lv_dIW3AnRlfIaDO27oVO-AqRrKnkM9ZLEiy4wxzQmdctvlCbCuyX5nLefEQoDl_nIgTvPPSyUScdg"
}
2017-06-30 18:05:56,328:DEBUG:requests.packages.urllib3.connectionpool:https://acme-staging.api.letsencrypt.org:443 "POST /acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209415 HTTP/1.1" 202 338
2017-06-30 18:05:56,329:DEBUG:acme.client:Received response:
HTTP 202
Server: nginx
Content-Type: application/json
Content-Length: 338
Boulder-Request-Id: e3ekmnm5QZ0St8I23C6U-FDJIiR3ENwyoS7UIwKxGNo
Boulder-Requester: 1674310
Link: <https://acme-staging.api.letsencrypt.org/acme/authz/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI>;rel="up"
Location: https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209415
Replay-Nonce: fyaD7TDBByFDAlcG-gVKowohxmWeSNjRqExIVxs02bY
Expires: Fri, 30 Jun 2017 18:05:56 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 30 Jun 2017 18:05:56 GMT
Connection: keep-alive

{
  "type": "http-01",
  "status": "pending",
  "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209415",
  "token": "0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk",
  "keyAuthorization": "0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk.R_138jwlMfn-8DshdSuaxf72U7PSTNxLVjts6x4VX24"
}
2017-06-30 18:05:56,330:DEBUG:acme.client:Storing nonce: fyaD7TDBByFDAlcG-gVKowohxmWeSNjRqExIVxs02bY
2017-06-30 18:05:59,333:DEBUG:acme.client:Sending GET request to https://acme-staging.api.letsencrypt.org/acme/authz/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI.
2017-06-30 18:05:59,708:DEBUG:requests.packages.urllib3.connectionpool:https://acme-staging.api.letsencrypt.org:443 "GET /acme/authz/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI HTTP/1.1" 200 2054
2017-06-30 18:05:59,709:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Content-Type: application/json
Boulder-Request-Id: _qdrKX5_FyF6LVpvZJOLGJHbWwSLkFZTnoWfbXFi9Rg
Link: <https://acme-staging.api.letsencrypt.org/acme/new-cert>;rel="next"
Replay-Nonce: _ktBN_uX6l-KbwUwb9OTzDxyM3yOMAXnNoJy3fk1rqs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800
Content-Length: 2054
Expires: Fri, 30 Jun 2017 18:05:59 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 30 Jun 2017 18:05:59 GMT
Connection: keep-alive
{
  "identifier": {
    "type": "dns",
    "value": "mail.mydomain.net"
  },
  "status": "invalid",
  "expires": "2017-07-07T18:05:56Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "pending",
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209413",
      "token": "06lCR7xJgbt8lRJQxP6ywNl3dTCv2g7OXamtWVU2NKM"
    },
    {
      "type": "tls-sni-01",
      "status": "pending",
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209414",
      "token": "BrUrrMTk6PvhaptEHDakgnCJ4Lbm5BT1wcKCq3M7V08"
    },
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:acme:error:unauthorized",
        "detail": "Invalid response from http://mail.mydomain.net/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk: \"\u003chtml\u003e\r\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\r\n\u003cbody bgcolor=\"white\"\u003e\r\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\r\n\u003chr\u003e\u003ccenter\u003e\"",
        "status": 403
      },
      "uri": "https://acme-staging.api.letsencrypt.org/acme/challenge/MiqyJcDoSisHIkU_h1rkxUmUz-QWqSi1MncOwKEKyGI/46209415",
      "token": "0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk",
      "keyAuthorization": "0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk.R_138jwlMfn-8DshdSuaxf72U7PSTNxLVjts6x4VX24",
      "validationRecord": [
        {
          "url": "http://mail.mydomain.net/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk",
          "hostname": "mail.mydomain.net",
          "port": "80",
          "addressesResolved": [
            "159.203.180.253",
            "2604:a880:400:d0::2616:2001"
          ],
          "addressUsed": "2604:a880:400:d0::2616:2001",
          "addressesTried": []
        }
      ]
    }
  ],
  "combinations": [
    [
      2
    ],
    [
      0
    ],
    [
      1
    ]
  ]
}
2017-06-30 18:05:59,711:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: mail.mydomain.net
Type:   unauthorized
Detail: Invalid response from http://mail.mydomain.net/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk: "<html>^M
<head><title>404 Not Found</title></head>^M
<body bgcolor="white">^M
<center><h1>404 Not Found</h1></center>^M
<hr><center>"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A record(s) for that domain contain(s) the right IP address.
2017-06-30 18:05:59,711:INFO:certbot.auth_handler:Cleaning up challenges
2017-06-30 18:05:59,711:DEBUG:certbot.plugins.webroot:Removing /usr/share/nginx/roundcubemail/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk
2017-06-30 18:05:59,711:INFO:certbot.plugins.webroot:Unable to clean up challenge directory /usr/share/nginx/roundcubemail/.well-known/acme-challenge
2017-06-30 18:05:59,712:DEBUG:certbot.plugins.webroot:Error was: [Errno 39] Directory not empty: '/usr/share/nginx/roundcubemail/.well-known/acme-challenge'
2017-06-30 18:05:59,713:WARNING:certbot.renewal:Attempting to renew cert from /etc/letsencrypt/renewal/mail.mydomain.net.conf produced an unexpected error: Failed authorization procedure. mail.mydomain.net (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.mydomain.net/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk: "<html>^M
<head><title>404 Not Found</title></head>^M
<body bgcolor="white">^M
<center><h1>404 Not Found</h1></center>^M
<hr><center>". Skipping.
2017-06-30 18:05:59,714:DEBUG:certbot.renewal:Traceback was:
Traceback (most recent call last):
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 418, in handle_renewal_request
    main.renew_cert(lineage_config, plugins, renewal_candidate)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 640, in renew_cert
    _get_and_save_cert(le_client, config, lineage=lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/main.py", line 77, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python2.7/dist-packages/certbot/renewal.py", line 296, in renew_cert
    new_certr, new_chain, new_key, _ = le_client.obtain_certificate(domains)
  File "/usr/lib/python2.7/dist-packages/certbot/client.py", line 313, in obtain_certificate
    self.config.allow_subset_of_names)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 81, in get_authorizations
    self._respond(resp, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 138, in _respond
    self._poll_challenges(chall_update, best_effort)
  File "/usr/lib/python2.7/dist-packages/certbot/auth_handler.py", line 202, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
FailedChallenges: Failed authorization procedure. mail.mydomain.net (http-01): urn:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from http://mail.mydomain.net/.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk: "<html>^M
<head><title>404 Not Found</title></head>^M
<body bgcolor="white">^M
<center><h1>404 Not Found</h1></center>^M
<hr><center>"

Entry from /var/log/nginx/access.log:

2600:3000:2710:300::1d - - [30/Jun/2017:14:05:56 -0400] "GET /.well-known/acme-challenge/0sHdYx9DTvGv491XjJ15sdrz6rn3LlUvVol7FUfJfIk HTTP/1.1" 404 136 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)"

Okay, my last edit which included the entry from /var/log/nginx/access.log provided the necessary clue. I noticed that the validation request was coming from an IPv6 address. When I tried the following command from a remote server:

curl -6 http://mail.mydomain.net/.well-known/acme-challenge/test.txt

I got the following response:

<html>
<head><title>404 Not Found</title></head>
<body bgcolor="white">
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

and the following entry appeared in /var/log/nginx/access.log:

XXXX:XXX:XXXX:XXXX:XXXX:XXXX:XXX:XXXX - - [30/Jun/2017:15:51:16 -0400] "GET /.well-known/acme-challenge/test.txt HTTP/1.1" 404 162 "-" "curl/7.35.0"

I followed these instructions from a DigitalOcean community question:

IPv6 connectivity issues with nginx

When it comes to NGINX, enabling IPv6 should only require that you're server block contain the lines shown below (I've intentionally cut the rest of the sever block out since the important thing is getting IPv6 working for you -- the other lines don't really matter).

server
{
    listen                          80 default_server;
    listen                          [::]:80 ipv6only=on;
}

or

server
{
    listen                          443 default_server;
    listen                          [::]:443 ipv6only=on;
}

and then ran

certbot --dry-run renew

again. Success!

-------------------------------------------------------------------------------
Processing /etc/letsencrypt/renewal/mail.mydomain.net.conf
-------------------------------------------------------------------------------
Cert is due for renewal, auto-renewing...
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for mail.mydomain.net
Waiting for verification...
Cleaning up challenges
Unable to clean up challenge directory /usr/share/nginx/roundcubemail/.well-known/acme-challenge

-------------------------------------------------------------------------------
new certificate deployed without reload, fullchain is
/etc/letsencrypt/live/mail.mydomain.net/fullchain.pem
-------------------------------------------------------------------------------

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.