Cannot renew/create a new cert when I had no issue previously

Ackis · September 29, 2019, 4:13am

Howdy folks. I just noticed this issue today when I tried creating a new cert - it also errored out when trying to renew a cert.

The domain I was trying to renew the cert on was ackis.duckdns.org and the domain I was trying to create a cert on was grafana.ackis.duckdns.org (I'll give anyone a cookie if they guess what I'm trying to set up ).

First off I'll post my nginx configs, and then the output of certbot. I have 38 different certificates with LE and I want to get on top of this before they try to auto-renew and things decide to blow up.

Any tips, pointers, or advice would be helpful.

/etc/nginx/sites-enabled/internet

#server {
#       listen                                  80 default;
#       listen                                  [::]:80;
#       listen                                  443;
#       listen                                  [::]:443 ssl;
#       return                                  301 https://ackis.duckdns.org;
#
#       server_name                             "";
#       return                                  444;
#
#       access_log                              syslog:server=localhost,tag=nginx_access_undefined,severity=info;
#       error_log                               syslog:server=localhost,tag=nginx_error_undefined,severity=error;
#}


# HTTP server - redirect to HTTPS

server {
        listen                                  80;
        listen                                  [::]:80;
        server_name                             www.ackis.duckdns.org ackis.duckdns.org;
        return                                  301 https://ackis.duckdns.org$request_uri;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;
}

# www domain - redirect to domain without www

server {
        listen                                  443 ssl;
        listen                                  [::]:443 ssl;
        server_name                             www.ackis.duckdns.org;
        return                                  301 https://ackis.duckdns.org$request_uri;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        ssl_certificate                         /etc/letsencrypt/live/www.ackis.duckdns.org/fullchain.pem;
        ssl_certificate_key                     /etc/letsencrypt/live/www.ackis.duckdns.org/privkey.pem;
}

server {
        listen                                  443 ssl default_server;
        listen                                  [::]:443 ssl;
        server_name                             ackis.duckdns.org;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;
        ssl_certificate                         /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key                     /etc/letsencrypt/live/ackis.duckdns.org/privkey.pem; # managed by Certbot

        ssl_stapling                            on;
        ssl_stapling_verify                     on;
        server_tokens                           off;

        etag                                    off;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";

#       location ^~ /.well-known/acme-challenge/ {
#               default_type                    "text/plain";
#               root                            /var/www/internet;
#       }
#
#       location = /.well-known/acme-challenge/ {
#               return                          404;
#       }

        location ~* /\.\./ {
                deny                            all;
                return                          404;
        }

        location ~* "^(?:.+\.(?:htaccess|make|txt|test|markdown|md|engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Repository|/Root|/Tag|/Template|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save))$" {
                return                          404;
        }

        location = /favicon.ico {
                try_files                       /favicon.ico =204;
        }

        location / {
                root                            /var/www/internet;
                index                           index.html;
        }

        location /nginx_status {
                access_log                      syslog:server=localhost,tag=nginx_access_admin,severity=info;
                error_log                       syslog:server=localhost,tag=nginx_error_admin;
                allow                           192.168.0.0/24;
                deny                            all;

                auth_basic                      "Restricted access";
                auth_basic_user_file            /etc/nginx/auth/admin.htpasswd;

                stub_status                     on;
        }
}

/etc/nginx/sites-enabled/grafana

upstream grafana {
server 192.168.0.199:3000;
keepalive 512;
}

server {
listen 80;
server_name grafana.ackis.duckdns.org;
return 301 https://$server_name$request_uri;

    access_log                              syslog:server=localhost,tag=nginx_access_grafana,severity=info combined;
    error_log                               syslog:server=localhost,tag=nginx_error_grafana;

}

server {
listen 443 ssl;
server_name grafana.ackis.duckdns.org;

    access_log                              syslog:server=localhost,tag=nginx_access_grafana,severity=info combined;
    error_log                               syslog:server=localhost,tag=nginx_error_grafana;

ssl_certificate /etc/ssl/certs/ssl-cert-snakeoil.pem;
ssl_certificate_key /etc/ssl/private/ssl-cert-snakeoil.key;

    ssl_stapling                    off;
    ssl_stapling_verify             off;
    server_tokens                   off;

    add_header                              Strict-Transport-Security "max-age=31536000; includeSubdomains; preload";
    add_header                              X-Frame-Options SAMEORIGIN;
    add_header                              X-Content-Type-Options nosniff;
    add_header                              X-XSS-Protection "1; mode=block";

    include                                 /etc/nginx/conf.d/include.whitelist;

    location = /.well-known/acme-challenge/ {
            return                          404;
    }

    location ~* /\.\./ {
            deny                            all;
            return                          404;
    }

    location ~* "^(?:.+\.(?:htaccess|make|txt|test|markdown|md|engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Repository|/Root|/Tag|/Template|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save))$" {
            return                          404;
    }

    location = /favicon.ico {
            try_files                       /favicon.ico =204;
    }

    location / {
            proxy_pass                      http://192.168.0.199:3000/;
            proxy_redirect          http://192.168.0.199:3000/ /;
            proxy_set_header        X-Forwarded-Proto https;
            include                         /etc/nginx/conf.d/proxy.conf;
    }

}

Output from certbot when I ran it:

What would you like to do?

1: Keep the existing certificate for now
2: Renew & replace the cert (limit ~5 per 7 days)

Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 2
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ackis.duckdns.org
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. ackis.duckdns.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/dlVk1U9ZqC-fLWHckMSB0cgbluxR5bHx6E0M2QXFqZ8 [174.3.126.96]: "\r\n404 Not Found\r\n\r\n
404 Not Found
\r\n
nginx\r\n"

IMPORTANT NOTES:

The following errors were reported by the server:

Domain: ackis.duckdns.org
Type: unauthorized
Detail: Invalid response from
https://ackis.duckdns.org/.well-known/acme-challenge/dlVk1U9ZqC-fLWHckMSB0cgbluxR5bHx6E0M2QXFqZ8
[174.3.126.96]: "\r\n404 Not
Found\r\n\r\n
404 Not
Found
\r\n
nginx\r\n"

To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.

_az · September 29, 2019, 4:26am

What authenticator are you using?

You can find that information in /etc/letsencrypt/renewal/ackis.duckdns.org.conf.

Ackis · September 29, 2019, 4:29am

Nginx.

Installer is Nginx as well, and the server is acme-v02.

_az · September 29, 2019, 4:33am

Take this line, and wrap it in a location block.

location / {
    return 301 https://ackis.duckdns.org$request_uri;
}

it will have the exact same effect in terms of HTTP to HTTPS redirect, except it won't bludgeon the challenge response rule that Certbot's nginx authenticator temporarily adds.

Restart nginx and try again.

Ackis · September 29, 2019, 4:36am

Done. Still received the same error "The client lacks sufficient authorization"

_az · September 29, 2019, 4:41am

Weird. Could you post the /var/log/letsencrypt/letsencrypt.log log file from your most recent attempt?

It shows what modifications were done to your nginx during authentication.

Edit: also, did you definitely wrap the return line that was in the port 80 virtualhost? Not the one from the other port 443 virtualhost?

Ackis · September 29, 2019, 4:45am

Yup

2019-09-28 22:34:42,720:DEBUG:certbot.main:certbot version: 0.31.0
2019-09-28 22:34:42,721:DEBUG:certbot.main:Arguments: []
2019-09-28 22:34:42,721:DEBUG:certbot.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2019-09-28 22:34:42,727:DEBUG:certbot.log:Root logging level set at 20
2019-09-28 22:34:42,727:INFO:certbot.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2019-09-28 22:34:42,728:DEBUG:certbot.plugins.selection:Requested authenticator None and installer None
2019-09-28 22:34:49,965:DEBUG:certbot.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx.configurator:NginxConfigurator
Initialized: <certbot_nginx.configurator.NginxConfigurator object at 0x7fa24bc5ec50>
Prep: True
2019-09-28 22:34:49,968:DEBUG:certbot.plugins.selection:Selected authenticator <certbot_nginx.configurator.NginxConfigurator object at 0x7fa24bc5ec50> and installer <certbot_nginx.configurator.NginxConfigurator object at 0x7fa24bc5ec50>
2019-09-28 22:34:49,968:INFO:certbot.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2019-09-28 22:34:49,972:DEBUG:certbot.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-v01.api.letsencrypt.org/acme/reg/1690363', new_authzr_uri=None, terms_of_service=None), 8ffbfcc2a88125b899a281120a5f6b44, Meta(creation_dt=datetime.datetime(2016, 5, 8, 0, 4, 40, tzinfo=<UTC>), creation_host='localhost'))>
2019-09-28 22:34:49,973:DEBUG:acme.client:Sending GET request to https://acme-v02.api.letsencrypt.org/directory.
2019-09-28 22:34:49,975:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-v02.api.letsencrypt.org
2019-09-28 22:34:50,538:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 658
2019-09-28 22:34:50,539:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 29 Sep 2019 04:34:50 GMT
Content-Type: application/json
Content-Length: 658
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "-wCRDQDMjr0": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org"
  },
  "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert"
}
2019-09-28 22:34:53,647:INFO:certbot.renewal:Cert not yet due for renewal
2019-09-28 22:34:56,782:INFO:certbot.main:Renewing an existing certificate
2019-09-28 22:34:56,909:DEBUG:certbot.crypto_util:Generating key (2048 bits): /etc/letsencrypt/keys/0351_key-certbot.pem
2019-09-28 22:34:56,916:DEBUG:certbot.crypto_util:Creating CSR: /etc/letsencrypt/csr/0351_csr-certbot.pem
2019-09-28 22:34:56,917:DEBUG:acme.client:Requesting fresh nonce
2019-09-28 22:34:56,917:DEBUG:acme.client:Sending HEAD request to https://acme-v02.api.letsencrypt.org/acme/new-nonce.
2019-09-28 22:34:56,997:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2019-09-28 22:34:56,997:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 29 Sep 2019 04:34:56 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001OA4r3gOvAATu2FdWQIfWkBMvNKzfXl57pyBimSJFUCU
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2019-09-28 22:34:56,998:DEBUG:acme.client:Storing nonce: 0001OA4r3gOvAATu2FdWQIfWkBMvNKzfXl57pyBimSJFUCU
2019-09-28 22:34:56,998:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "ackis.duckdns.org"\n    }\n  ]\n}'
2019-09-28 22:34:57,002:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL3JlZy8xNjkwMzYzIiwgIm5vbmNlIjogIjAwMDFPQTRyM2dPdkFBVHUyRmRXUUlmV2tCTXZOS3pmWGw1N3B5QmltU0pGVUNVIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
  "signature": "i3xY8Za3EJpPSRCVge1Ftoh0QPO7W8yYZWn3wyCkJLMhaoFoXjcFwT_sHvBWcP3uqwnJl7p8coAXGKoa7HeicB4a_tP_vu5VMkqRyIvVkJbL3XHbdsip10SYkdw5iZvohcpw6HoxcBMQBFr4Eu_Q-u8s9fVmqHZfa-2eYZvCU00ZyHWtJoT9eu0u3_XRYi-TtuIUyJGs5v2RDz5XJ7XCnotHq1F9T6S_6bMMFm_pGSsF5eElKdDe0zrJc9TAXQgx5MwEF9WsuLjElcYHySRbPN8nVO_C-VbK1pjsXZnj94zyXjGdtVx7A1YUr6rnbkEm179bf9iqnkF6bjq-ihsTPw",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImFja2lzLmR1Y2tkbnMub3JnIgogICAgfQogIF0KfQ"
}
2019-09-28 22:34:57,170:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 345
2019-09-28 22:34:57,171:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Sun, 29 Sep 2019 04:34:57 GMT
Content-Type: application/json
Content-Length: 345
Connection: keep-alive
Boulder-Requester: 1690363
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-v02.api.letsencrypt.org/acme/order/1690363/1181990325
Replay-Nonce: 0001f_1G6ByXf7TGvPXrqOcvJHm_cOm3zB08AqJN1CyOvtM
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2019-10-06T04:34:57.126079845Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "ackis.duckdns.org"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/549729261"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/1690363/1181990325"
}
2019-09-28 22:34:57,171:DEBUG:acme.client:Storing nonce: 0001f_1G6ByXf7TGvPXrqOcvJHm_cOm3zB08AqJN1CyOvtM
2019-09-28 22:34:57,172:DEBUG:acme.client:JWS payload:
b''
2019-09-28 22:34:57,174:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/549729261:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL3JlZy8xNjkwMzYzIiwgIm5vbmNlIjogIjAwMDFmXzFHNkJ5WGY3VEd2UFhycU9jdkpIbV9jT20zekIwOEFxSk4xQ3lPdnRNIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My81NDk3MjkyNjEifQ",
  "signature": "iOZCWnfF1hu4p1edwrHy7bATuqzndx7C99b5SdPynik_8mYw-Vg_92MJI-RcWF4ZtndfKwVC6trC5hww95_bxpd2SR92eyovER1w_aBt9t0sst8g29q1iPtF0OtlY7rfcLiMPAM2hFzTIiAJTmZmesshN8kFnbwRaRw3b0S9B5uJuBtuGz4iM3ayioaZ6xLqATOSsYR9U0YTe_x5CSEJbqUjHSSmK6aa1i7uA1QoZztYX7GKXY1-hGmemyS7l6csPRc2DS-fG97ixYgnDNYWhwQnqvqPfhBv4eYOv7eLp5pfLjMWYk2kEYLsxmvdSqFwpQ-C1HbEuSQZdVYucG_5pA",
  "payload": ""
}
2019-09-28 22:34:57,258:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/549729261 HTTP/1.1" 200 792
2019-09-28 22:34:57,259:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 29 Sep 2019 04:34:57 GMT
Content-Type: application/json
Content-Length: 792
Connection: keep-alive
Boulder-Requester: 1690363
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 00024gmwHBBMnFWB6NLJwoHgiDZ3AsRTfrUcImi65u__OJs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "ackis.duckdns.org"
  },
  "status": "pending",
  "expires": "2019-10-06T04:34:57Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/4EZyKA",
      "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/T4uEqA",
      "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/CXnUTw",
      "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4"
    }
  ]
}
2019-09-28 22:34:57,259:DEBUG:acme.client:Storing nonce: 00024gmwHBBMnFWB6NLJwoHgiDZ3AsRTfrUcImi65u__OJs
2019-09-28 22:34:57,260:INFO:certbot.auth_handler:Performing the following challenges:
2019-09-28 22:34:57,260:INFO:certbot.auth_handler:http-01 challenge for ackis.duckdns.org
2019-09-28 22:34:57,387:DEBUG:certbot_nginx.http_01:Generated server block:
[]
2019-09-28 22:34:57,387:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/plex
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/znc
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/channydraws-dev
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/asf
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/sonarr
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/grafana
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/denon
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/home
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/monit
2019-09-28 22:34:57,388:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/transmission
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/calibre
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/hdhrextend
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/nextcloud
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/webmin
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/conf.d/include.whitelist
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/lidarr
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/plexwebtools
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/glances
2019-09-28 22:34:57,389:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/internet
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/mf4890
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/ubooquity
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/datacrow
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/canon
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/radarr
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/wiki
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/sabnzbd
2019-09-28 22:34:57,390:DEBUG:certbot.reverter:Creating backup of /etc/nginx/nginx.conf
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/chantelle
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/channydraws
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/usbserver
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/mylar
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/mime.types
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/zoneminder2
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/plexpy
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/oleary
2019-09-28 22:34:57,391:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/ombi
2019-09-28 22:34:57,392:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/obi200
2019-09-28 22:34:57,392:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:
user www-data;
worker_processes 4;
pid /run/nginx.pid;

events {
        worker_connections 1024;
        # multi_accept on;
}

http {
include /etc/letsencrypt/le_http_01_cert_challenge.conf;
server_names_hash_bucket_size 128;

        ##
        # Basic Settings
        ##

        sendfile on;
        tcp_nopush on;
        tcp_nodelay on;
        keepalive_timeout 65;
        types_hash_max_size 2048;
        server_tokens off;

        # server_names_hash_bucket_size 64;
        # server_name_in_redirect off;

        include /etc/nginx/mime.types;
        default_type application/octet-stream;

        ##
        # Gzip Settings
        ##

        gzip on;
        gzip_disable "msie6";

        # gzip_vary on;
        # gzip_proxied any;
        # gzip_comp_level 6;
        # gzip_buffers 16 8k;
        # gzip_http_version 1.1;
        # gzip_types text/plain text/css application/json application/x-javascript text/xml application/xml application/xml+rss text/javascript;

        ##
        # nginx-naxsi config
        ##
        # Uncomment it if you installed nginx-naxsi
        ##

        #include /etc/nginx/naxsi_core.rules;

        ##
        # nginx-passenger config
        ##
        # Uncomment it if you installed nginx-passenger
        ##

        #passenger_root /usr;
        #passenger_ruby /usr/bin/ruby;

        ##
        # Virtual Host Configs
        ##

#       include /etc/nginx/conf.d/*.conf;
        include /etc/nginx/sites-enabled/*;
}


#mail {
#       # See sample authentication script at:
#       # http://wiki.nginx.org/ImapAuthenticateWithApachePhpScript
#
#       # auth_http localhost/auth.php;
#       # pop3_capabilities "TOP" "USER";
#       # imap_capabilities "IMAP4rev1" "UIDPLUS";
#
#       server {
#               listen     localhost:110;
#               protocol   pop3;
#               proxy      on;
#       }
#
#       server {
#               listen     localhost:143;
#               protocol   imap;
#               proxy      on;
#       }
#}

2019-09-28 22:34:57,394:DEBUG:certbot_nginx.parser:Writing nginx conf tree to /etc/nginx/sites-enabled/internet:
# HTTP server - redirect to HTTPS

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


        listen                                  80;
        listen                                  [::]:80;
        server_name                             www.ackis.duckdns.org ackis.duckdns.org;
        location / {
                return                                  301 https://ackis.duckdns.org$request_uri;
        }

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;
location = /.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4{default_type text/plain;return 200 ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4.WQr5aZhiL_lm2g0Gd4sKtM4CUAKHTqL_jIdP25wamqc;} # managed by Certbot

}

# www domain - redirect to domain without www

server {
        listen                                  443 ssl;
        listen                                  [::]:443 ssl;
        server_name                             www.ackis.duckdns.org;
        return                                  301 https://ackis.duckdns.org$request_uri;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;

        ssl_certificate                         /etc/letsencrypt/live/www.ackis.duckdns.org/fullchain.pem;
        ssl_certificate_key                     /etc/letsencrypt/live/www.ackis.duckdns.org/privkey.pem;
}

server {
        listen                                  443 ssl default_server;
        listen                                  [::]:443 ssl;
        server_name                             ackis.duckdns.org;

        access_log                              syslog:server=localhost,tag=nginx_access_internet,severity=info;
        error_log                               syslog:server=localhost,tag=nginx_error_internet;
        ssl_certificate                         /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem; # managed by Certbot
        ssl_certificate_key                     /etc/letsencrypt/live/ackis.duckdns.org/privkey.pem; # managed by Certbot

        ssl_stapling                            on;
        ssl_stapling_verify                     on;
        server_tokens                           off;

        etag                                    off;

        add_header                              Strict-Transport-Security "max-age=31536000;";
        add_header                              X-Frame-Options SAMEORIGIN;
        add_header                              X-Content-Type-Options nosniff;
        add_header                              X-XSS-Protection "1; mode=block";

#       location ^~ /.well-known/acme-challenge/ {
#               default_type                    "text/plain";
#               root                            /var/www/internet;
#       }
#
#       location = /.well-known/acme-challenge/ {
#               return                          404;
#       }

        location ~* /\.\./ {
                deny                            all;
                return                          404;
        }

        location ~* "^(?:.+\.(?:htaccess|make|txt|test|markdown|md|engine|inc|info|install|module|profile|po|sh|.*sql|theme|tpl(?:\.php)?|xtmpl)|code-style\.pl|/Entries.*|/Repository|/Root|/Tag|/Template|^#.*#$|\.php(~|\.sw[op]|\.bak|\.orig\.save))$" {
                return                          404;
        }

        location = /favicon.ico {
                try_files                       /favicon.ico =204;
        }

        location / {
                root                            /var/www/internet;
                index                           index.html;
        }

        location /nginx_status {
                access_log                      syslog:server=localhost,tag=nginx_access_admin,severity=info;
                error_log                       syslog:server=localhost,tag=nginx_error_admin;
                allow                           192.168.0.0/24;
                deny                            all;

                auth_basic                      "Restricted access";
                auth_basic_user_file            /etc/nginx/auth/admin.htpasswd;

                stub_status                     on;
        }
}

2019-09-28 22:35:04,454:INFO:certbot.auth_handler:Waiting for verification...
2019-09-28 22:35:04,455:DEBUG:acme.client:JWS payload:
b'{\n  "resource": "challenge",\n  "keyAuthorization": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4.WQr5aZhiL_lm2g0Gd4sKtM4CUAKHTqL_jIdP25wamqc",\n  "type": "http-01"\n}'
2019-09-28 22:35:04,458:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/4EZyKA:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL3JlZy8xNjkwMzYzIiwgIm5vbmNlIjogIjAwMDI0Z213SEJCTW5GV0I2TkxKd29IZ2lEWjNBc1JUZnJVY0ltaTY1dV9fT0pzIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbC12My81NDk3MjkyNjEvNEVaeUtBIn0",
  "signature": "L8CDWJxsfXlUeuJtOy18CCDAxQKfjYbFPT2b8M4n8vIZaBfMGLqFtIx011CphgKQXKDnRchUZ30XjgfOuup3Zq6aZClB4Q7oCipUvlt40oBj9fnUy01BcbyB6in_AOe9b0d8D9fD7kgBxjRczVOcCnNwhU4RxW92i9UF5CI-1kBvPG1FUcoGKDPlinyH70lVikJBXOfWL-EeQdFV920vzITI9CCrP3c_vVHSrpkZlF00ejgci4EfFmGszvHzwKbBcX1elndooLuuNUANTsNetKF10cdrKYN-ezfj7XSDWfd_58kmmtGuBua7CtLUQTCIcV15dBylMcUjjKF_LW5bww",
  "payload": "ewogICJyZXNvdXJjZSI6ICJjaGFsbGVuZ2UiLAogICJrZXlBdXRob3JpemF0aW9uIjogIkJ5clJNQ0tKN2sxeU5iR21iN3JGY0w0MThIN3VxeVZQcjFWb1ZSU3lkVzQuV1FyNWFaaGlMX2xtMmcwR2Q0c0t0TTRDVUFLSFRxTF9qSWRQMjV3YW1xYyIsCiAgInR5cGUiOiAiaHR0cC0wMSIKfQ"
}
2019-09-28 22:35:04,714:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/549729261/4EZyKA HTTP/1.1" 200 184
2019-09-28 22:35:04,715:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 29 Sep 2019 04:35:04 GMT
Content-Type: application/json
Content-Length: 184
Connection: keep-alive
Boulder-Requester: 1690363
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-v02.api.letsencrypt.org/acme/authz-v3/549729261>;rel="up"
Location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/4EZyKA
Replay-Nonce: 0001Kv0tFbHGPHxJM1-RJDuW5j82fnVAyQhiGt5tAUcmEqg
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/4EZyKA",
  "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4"
}
2019-09-28 22:35:04,715:DEBUG:acme.client:Storing nonce: 0001Kv0tFbHGPHxJM1-RJDuW5j82fnVAyQhiGt5tAUcmEqg
2019-09-28 22:35:07,718:DEBUG:acme.client:JWS payload:
b''
2019-09-28 22:35:07,721:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/549729261:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDEuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL3JlZy8xNjkwMzYzIiwgIm5vbmNlIjogIjAwMDFLdjB0RmJIR1BIeEpNMS1SSkR1VzVqODJmblZBeVFoaUd0NXRBVWNtRXFnIiwgInVybCI6ICJodHRwczovL2FjbWUtdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My81NDk3MjkyNjEifQ",
  "signature": "Y3rhkBBjI5Ja7IiAqw6DSTKGsuzf6p4B58MREd5qnJ3jjHuNnA3Y6UGZr8s_3BdjeKTM_3X595LP7kJ5xI4HfXsQG3W0d0APUlImR2zho7H01QmNpDXcKC_Cv6tZZNvQHoMuJgm-BAqgq4kRUwqt2AsZh3fIWnTfSSqwlXmwbpEQca4AtQy1FAKuDXqgwlJs1Clx9YiZ8KYVXRi8D_6jA2_phSBu78oHBAUK8sPyDL78KaS-Z-MCrqH0Yvc-y5iQF5EkQ_YOtXNmMdWhytjAxrYj0_U4TZIROmfPSCqw1tidChqkiEfvariCRCwJFZyQptu0vyLfS5GAwG8lLhjkjw",
  "payload": ""
}
2019-09-28 22:35:07,816:DEBUG:urllib3.connectionpool:https://acme-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/549729261 HTTP/1.1" 200 2010
2019-09-28 22:35:07,817:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 29 Sep 2019 04:35:07 GMT
Content-Type: application/json
Content-Length: 2010
Connection: keep-alive
Boulder-Requester: 1690363
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0001xYwB5iUfLLzbA9MlCr59vbWXEWu7WGhDXjDZCXC5Eqw
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "ackis.duckdns.org"
  },
  "status": "invalid",
  "expires": "2019-10-06T04:34:57Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4 [174.3.126.96]: \"\u003chtml\u003e\\r\\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\\r\\n\u003cbody\u003e\\r\\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\\r\\n\u003chr\u003e\u003ccenter\u003enginx\u003c/center\u003e\\r\\n\"",
        "status": 403
      },
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/4EZyKA",
      "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4",
      "validationRecord": [
        {
          "url": "http://ackis.duckdns.org/.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4",
          "hostname": "ackis.duckdns.org",
          "port": "80",
          "addressesResolved": [
            "174.3.126.96"
          ],
          "addressUsed": "174.3.126.96"
        },
        {
          "url": "https://ackis.duckdns.org/.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4",
          "hostname": "ackis.duckdns.org",
          "port": "443",
          "addressesResolved": [
            "174.3.126.96"
          ],
          "addressUsed": "174.3.126.96"
        }
      ]
    },
    {
      "type": "dns-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/T4uEqA",
      "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4"
    },
    {
      "type": "tls-alpn-01",
      "status": "invalid",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/549729261/CXnUTw",
      "token": "ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4"
    }
  ]
}
2019-09-28 22:35:07,817:DEBUG:acme.client:Storing nonce: 0001xYwB5iUfLLzbA9MlCr59vbWXEWu7WGhDXjDZCXC5Eqw
2019-09-28 22:35:07,818:DEBUG:certbot.reporter:Reporting to user: The following errors were reported by the server:

Domain: ackis.duckdns.org
Type:   unauthorized
Detail: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4 [174.3.126.96]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2019-09-28 22:35:07,821:DEBUG:certbot.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 239, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. ackis.duckdns.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4 [174.3.126.96]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

2019-09-28 22:35:07,821:DEBUG:certbot.error_handler:Calling registered functions
2019-09-28 22:35:07,821:INFO:certbot.auth_handler:Cleaning up challenges
2019-09-28 22:35:15,758:DEBUG:certbot.log:Exiting abnormally:
Traceback (most recent call last):
  File "/usr/bin/certbot", line 11, in <module>
    load_entry_point('certbot==0.31.0', 'console_scripts', 'certbot')()
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1365, in main
    return config.func(config, plugins)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 1119, in run
    certname, lineage)
  File "/usr/lib/python3/dist-packages/certbot/main.py", line 116, in _get_and_save_cert
    renewal.renew_cert(config, domains, le_client, lineage)
  File "/usr/lib/python3/dist-packages/certbot/renewal.py", line 310, in renew_cert
    new_cert, new_chain, new_key, _ = le_client.obtain_certificate(domains, new_key)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 353, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/lib/python3/dist-packages/certbot/client.py", line 389, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 82, in handle_authorizations
    self._respond(aauthzrs, resp, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 168, in _respond
    self._poll_challenges(aauthzrs, chall_update, best_effort)
  File "/usr/lib/python3/dist-packages/certbot/auth_handler.py", line 239, in _poll_challenges
    raise errors.FailedChallenges(all_failed_achalls)
certbot.errors.FailedChallenges: Failed authorization procedure. ackis.duckdns.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/ByrRMCKJ7k1yNbGmb7rFcL418H7uqyVPr1VoVRSydW4 [174.3.126.96]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

_az · September 29, 2019, 4:53am

Thanks.

From that, we can see the Certbot is making the correct modification to the correct place, and that you made the right adjustment earlier, but it doesn’t actually seem to be effective.

To that port 80 virtualhost, can you also add this test:

location = /.well-known/acme-challenge/test {
  return 200 "hello world";
}

Restart nginx, confirm that it’s happy with the config:

nginx -t

and then try:

curl -i ackis.duckdns.org/.well-known/acme-challenge/test

Ackis · September 29, 2019, 4:55am

I got "hello world" as I should.

_az · September 29, 2019, 5:01am

Jeez. The only remaining theory I can think of is that Certbot’s nginx authenticator isn’t actually reloading nginx before it tells Let’s Encrypt to check the challenge.

You can force Certbot to temporarily pause before that moment with --debug-challenges:

certbot renew --cert-name ackis.duckdns.org --debug-challenges --dry-run

which will allow you to do two things:

Check that the contents of the port 80 virtualhost are modified correctly during authentication
Allow you to manually restart nginx at that moment, to see if it helps, and then press Enter to make Certbot keep going

If that doesn’t work, I’d probably look at switching over to webroot authenticator, unless the mystery is solved some other way.

Ackis · September 29, 2019, 5:03am

Alright - appreciate the help. I’m self taught so everything I’ve done is put together in pieces - like that return 301 being put in a location. I have 30 config files to update now with that new knowledge because of you.

I’ll try that debugging in the morning when I’m fresh to see what’s happening and post back with results.

Ackis · September 30, 2019, 12:05am

Okay so the flag --debug-challenges doesn’t seem to actually pause the process.

When it says “Press continue to…” it just keeps on going.

sudo certbot renew --cert-name ackis.duckdns.org --debug-challenges --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/ackis.duckdns.org.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cert not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Renewing an existing certificate
Performing the following challenges:
http-01 challenge for ackis.duckdns.org
Waiting for verification...

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Cleaning up challenges
Attempting to renew cert (ackis.duckdns.org) from /etc/letsencrypt/renewal/ackis.duckdns.org.conf produced an unexpected error: Failed authorization procedure. ackis.duckdns.org (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://ackis.duckdns.org/.well-known/acme-challenge/7bO_DNxtDDyO_hPdRJcpGWEJHaLTRwtTsMWpWtQREDE [174.3.126.96]: "<html>\r\n<head><title>404 Not Found</title></head>\r\n<body>\r\n<center><h1>404 Not Found</h1></center>\r\n<hr><center>nginx</center>\r\n". Skipping.
All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem (failure)

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates below have not been saved.)

All renewal attempts failed. The following certs could not be renewed:
  /etc/letsencrypt/live/ackis.duckdns.org/fullchain.pem (failure)
** DRY RUN: simulating 'certbot renew' close to cert expiry
**          (The test certificates above have not been saved.)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: ackis.duckdns.org
   Type:   unauthorized
   Detail: Invalid response from
   https://ackis.duckdns.org/.well-known/acme-challenge/7bO_DNxtDDyO_hPdRJcpGWEJHaLTRwtTsMWpWtQREDE
   [174.3.126.96]: "<html>\r\n<head><title>404 Not
   Found</title></head>\r\n<body>\r\n<center><h1>404 Not
   Found</h1></center>\r\n<hr><center>nginx</center>\r\n"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

_az · September 30, 2019, 12:17am

Hmm. It should.

There seems to be a bug though, where if you press Enter after you run Certbot but before that prompt appears, it will buffer the keypress and the prompt won’t wait.

Could you try again, avoiding pressing any keys after you start Certbot?

Adding --force-interactive could help too.

Ackis · September 30, 2019, 12:27am

The only key I pressed was enter to run the command - I even copy/pasted the command with enter in it so I could get around that and it still wouldn't work.

--force-interactive cannot be used with renew

One thing I did do to see if it was reloading nginx was to edit my config but not reload it before running certbot and to see if the change went through. It looks like the change did go through.

I readded your snipped you suggested:

location = /.well-known/acme-challenge/test {
  return 200 "hello world";
}

And after running cerbot I was able to get the 200 status with Hello World back.

_az · September 30, 2019, 12:33am

. I really have no idea then, sorry. I'm missing something but can't see it.

There's always the option to add this to your port 80 vhosts and use webroot:

location /.well-known/acme-challenge/ {
    root /var/www/letsencrypt;
}

mkdir -p /var/www/letsencrypt
service nginx reload

certbot renew --cert-name ackis.duckdns.org \
-a webroot -w /var/www/letsencrypt --dry-run

You could also try hard-code this and not change to webroot:

location ~ ^/\.well-known/acme-challenge/([-_a-zA-Z0-9]+)$ {
  default_type text/plain;
  return 200 "$1.WQr5aZhiL_lm2g0Gd4sKtM4CUAKHTqL_jIdP25wamqc";
}

Ackis · September 30, 2019, 12:39am

No problem, again I appreciate your time and help. One day it just stopped working, and there was nothing I could see either. Maybe someone else will pick up on it.

Thanks for that suggestion, I hadn't thought it through yet - I was thinking on how to use the DNS route. I only need to put that in the port 80 area?

Will my token be the same for each website, everytime I try to renew?

_az · September 30, 2019, 12:40am

Yes, but the token will differ between --dry-run and live renewal. You can find it in the Certbot log files, but I believe the one above is for your live renewals.

Yes, just port 80 is really required, unless it gets redirected to HTTPS (but it shouldn't). No harm of putting it in both.

Webroot is certainly the simpler/more correct fix of the two.

beret · September 30, 2019, 6:42pm

Your issue seems similar to mine. The debug-challenges also fails to pause it, but if I loop systemctl reload nginx I'm able to make it provide a renewal response. It looks like my renews fail because certbot fails to reload nginx but new certs work fine.

What's your platform and setup? I'm on Ubuntu 16.04 LTS with the NGINX and certbot repositories.

Ackis · October 1, 2019, 3:51am

I'm running Ubuntu 18.04.3 LTS with nginx 1.17.4 (note that nginx 1.14.0 is the one that it's Ubuntu's repo's).

Ackis · October 2, 2019, 4:10am

I've got things working with webroot - I noticed that the config files don't update, so /etc/letsencrypt/renewal/ackis.duckdns.org.conf is still saying that I'm using nginx as an installer.

This is going to break things when it comes time to auto-renew, so should I go through and manually update the configs?

Also, would it be worthwhile making a ticket on github about nginx not reloading properly?