Certbot failing to create .well-known files

Help
1

My domain is:
lollyde.online

I ran this command:
sudo certbot certonly --dry-run -d lollyde.online

It produced this output:

[sudo] password for lollyde: 
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Could not find ssl_module; not disabling session tickets.

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin (apache)
2: Nginx Web Server plugin (nginx)
3: Spin up a temporary webserver (standalone)
4: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-4] then [enter] (press 'c' to cancel): 2
Plugins selected: Authenticator nginx, Installer None
Simulating a certificate request for lollyde.online
Performing the following challenges:
http-01 challenge for lollyde.online
Waiting for verification...
Challenge failed for domain lollyde.online
http-01 challenge for lollyde.online
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: lollyde.online
   Type:   unauthorized
   Detail: Invalid response from
   https://lollyde.online/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ
   [2606:4700:3033::6815:3b4b]: "<html>\n<head><title>404 Not
   Found</title></head>\n<body>\n<center><h1>404 Not
   Found</h1></center>\n<hr><center>nginx/1.19.6</center"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

My web server is (include version):
nginx version: nginx/1.19.6

The operating system my web server runs on is (include version):
Arch Linux 5.9.14-arch1-1 with all updates installed

My hosting provider, if applicable, is:
hetzner.com

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.11.0

I believe it is a failure of certbot to create the files because of the following:

The nginx access log at a debug level creates the following during the challenge:

...
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 try files handler
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 http script var: "/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ"
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 trying to use file: "/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ" "/data/personal-site/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ"
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 http script var: "/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ"
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 trying to use dir: "/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ" "/data/personal-site/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ"
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 trying to use file: "=404" "/data/personal-site=404"
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 http finalize request: 404, "/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ?" a:1, c:1
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 http special response: 404, "/.well-known/acme-challenge/VxnZCX_RjdIMF52XcZLQ6IB1u2QTKquSliHU4sjOisQ?"
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 http set discard body
2021/01/17 02:37:25 [debug] 3843664#3843664: *138842 HTTP/1.1 404 Not Found
...

as well as running inotifywait -m /data/personal-site with the following output before/during the challenge (note me manually creating and then deleting the .well-known folder before starting the challenge, then nothing until the end of the runtime):

Setting up watches.
Watches established.
./ CREATE,ISDIR .well-known
./ DELETE,ISDIR .well-known
^C

For completion sake, here is the relevant nginx config:

server {
        root /data/personal-site;
        index index.html;
        server_name lollyde.online;
        error_log logs/default.error debug;
        access_log logs/default.access;

        location / {
                try_files $uri $uri/ =404; 
                add_header X-Easteregg "You're a curious one, arent you?";
        }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/services.lollyde.online/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/services.lollyde.online/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = lollyde.online) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        server_name lollyde.online;
    listen 80;
    return 404; # managed by Certbot


}

Any idea on what I could do to further debug why certbot isnt creating the files?

1 Like
2

Welcome to the Let's Encrypt Community, Lily :slightly_smiling_face:

The nginx authenticator does not actually create the /.well-known/acme-challenge/ folder to serve http-01 challenge files. It creates a temporary exception in your nginx configuration to serve the files.

Try running this to see:

sudo certbot certonly --nginx -d "lollyde.online" --debug-challenges --dry-run

I noticed that you're using Cloudflare. You really should look into using a Cloudflare Origin CA certificate instead of a Let's Encrypt certificate. They're easier to manage with Cloudflare and last much longer.

Certificate History
1 Like
3

I see - that actually makes sense.

Does the resulting debug log file contain sensitive information or can I just share it as-is through e.g. pastebin? I didnt see any skimming over it but I'd rather ask to be sure.

Yes, I am indeed using Cloudflare, however it has not caused issues in the past and I have used Lets's Encrypt certificates to encrypt the traffic between my host server and Cloudflare for I'm fairly certain over a year now, without any problems, also not all of my subdomains are behind the Cloudflare proxy, for technical reasons. I will look into it more for sure, though!

1 Like
4

If you're meaning the certbot log file, there's nothing sensitive in it. You can even post it here directly with

```text

on a line above the output and

```

on a line below the output.

1 Like
5

Alright, sounds good!

2021-01-17 03:05:55,077:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2021-01-17 03:05:55,227:DEBUG:certbot._internal.main:certbot version: 1.11.0
2021-01-17 03:05:55,227:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/889/bin/certbot
2021-01-17 03:05:55,227:DEBUG:certbot._internal.main:Arguments: ['--nginx', '-d', 'lollyde.online', '--debug-challenges', '--dry-run', '--preconfigured-renewal']
2021-01-17 03:05:55,227:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-01-17 03:05:55,240:DEBUG:certbot._internal.log:Root logging level set at 20
2021-01-17 03:05:55,240:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-01-17 03:05:55,241:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2021-01-17 03:05:55,761:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7f27a1d8acd0>
Prep: True
2021-01-17 03:05:55,762:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7f27a1d8acd0>
Prep: True
2021-01-17 03:05:55,762:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7f27a1d8acd0> and installer <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7f27a1d8acd0>
2021-01-17 03:05:55,762:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2021-01-17 03:05:55,764:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/16945623', new_authzr_uri=None, terms_of_service=None), 1a1b549c66ad1ad7a4d61cb9012e1dc1, Meta(creation_dt=datetime.datetime(2020, 12, 4, 18, 20, 59, tzinfo=<UTC>), creation_host='lollyde-rootserver', register_to_eff=None))>
2021-01-17 03:05:55,765:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
2021-01-17 03:05:55,766:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
2021-01-17 03:05:56,522:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 724
2021-01-17 03:05:56,523:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 02:05:56 GMT
Content-Type: application/json
Content-Length: 724
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "m0aRICtbP8o": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
2021-01-17 03:05:56,525:DEBUG:certbot.display.util:Notifying user: Simulating a certificate request for lollyde.online
2021-01-17 03:05:56,562:DEBUG:acme.client:Requesting fresh nonce
2021-01-17 03:05:56,562:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
2021-01-17 03:05:56,729:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2021-01-17 03:05:56,730:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 02:05:56 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0004Gq7A8_OP6hM2h6c-FuldhfGHFkWhfxy5_GU8Om2WH7M
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2021-01-17 03:05:56,730:DEBUG:acme.client:Storing nonce: 0004Gq7A8_OP6hM2h6c-FuldhfGHFkWhfxy5_GU8Om2WH7M
2021-01-17 03:05:56,731:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "lollyde.online"\n    }\n  ]\n}'
2021-01-17 03:05:56,733:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDA0R3E3QThfT1A2aE0yaDZjLUZ1bGRoZkdIRmtXaGZ4eTVfR1U4T20yV0g3TSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
  "signature": "o5evQG7GBr8kvBt9gFperGNQ-yCY87ixTbNu26MEi6uOwytU4Sy7_EnPWBKSuLUJ67Xg_0DFcrUQB7bB2SrODYkfD09ddHRWABOHYEYTWsxDRzvyT4D6BgEn7hdzdyJXOPmhig0Pe2RLGngEMi9xf1iplGKZLvKElu8BnfstkhgHzGIjpGIUXNcLDgXrFIAmfCFfp0xrSl9DRl0FoE-JjE3bohuONWp2Ay5dvtsd2jLWMfXFp8vAXMftYYDVQRJPAz5_T8uvmTXL65rJdlegecTMGwGULk4xmBAq8iUk_pvNil3HFodCQK2l-y26-PQ3P10NeCvFF1xd5o7nNb6ujw",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImxvbGx5ZGUub25saW5lIgogICAgfQogIF0KfQ"
}
2021-01-17 03:05:56,927:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 348
2021-01-17 03:05:56,928:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Sun, 17 Jan 2021 02:05:56 GMT
Content-Type: application/json
Content-Length: 348
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/16945623/222739045
Replay-Nonce: 0003ORTl1zgCfd9LtNuoJk0gYTbjyt2tJ-Asssp7Er5TlBI
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2021-01-24T02:05:56Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "lollyde.online"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193001885"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/16945623/222739045"
}
2021-01-17 03:05:56,928:DEBUG:acme.client:Storing nonce: 0003ORTl1zgCfd9LtNuoJk0gYTbjyt2tJ-Asssp7Er5TlBI
2021-01-17 03:05:56,928:DEBUG:acme.client:JWS payload:
b''
2021-01-17 03:05:56,930:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193001885:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDAzT1JUbDF6Z0NmZDlMdE51b0prMGdZVGJqeXQydEotQXNzc3A3RXI1VGxCSSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8xOTMwMDE4ODUifQ",
  "signature": "s14zQFK5R9ZxzTKias-kF0Ezzn1WuooIAHiKW-ff8pveGxnbwt2lOU7KD3nnpOVmFzhbV6saGMHSgszv49TKWO-OddHDNNtuOJB7ii9tw6_0lRvxw7Yqx6GHIlTxv5tT1YicOHcuNep7D0O2piyPYVhC0I2W3n_Vq6pXmnnFrVVRyPzH8YdOUeXNq4TCz4grr2VmRWAcVgtdJ_6MfG8lGgPk4mga3QhACoo-r2hT1tdY0xGruFXpE9nX94aUOLLKxjuron_hmbeDMw7PRj5mv2GY6ZHtkjwWI-IbjZcpEQ6WbkOxzvI40l7pPBGYzQDkNL-c9j9Kh6txOoIPlxMsHA",
  "payload": ""
}
2021-01-17 03:05:57,101:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/193001885 HTTP/1.1" 200 813
2021-01-17 03:05:57,102:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 02:05:57 GMT
Content-Type: application/json
Content-Length: 813
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0004S4APoah_x7vWNHgUW7-lJ5zyt4bpRJf38nFozsxNXCE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "lollyde.online"
  },
  "status": "pending",
  "expires": "2021-01-24T02:05:56Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/MjsBqw",
      "token": "3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/2ORFqQ",
      "token": "3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/zUnBww",
      "token": "3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg"
    }
  ]
}
2021-01-17 03:05:57,102:DEBUG:acme.client:Storing nonce: 0004S4APoah_x7vWNHgUW7-lJ5zyt4bpRJf38nFozsxNXCE
2021-01-17 03:05:57,102:INFO:certbot._internal.auth_handler:Performing the following challenges:
2021-01-17 03:05:57,102:INFO:certbot._internal.auth_handler:http-01 challenge for lollyde.online
2021-01-17 03:05:57,121:DEBUG:certbot_nginx._internal.http_01:Generated server block:
[]
2021-01-17 03:05:57,121:DEBUG:certbot.reverter:Creating backup of /etc/nginx/nginx.conf
2021-01-17 03:05:57,121:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/services.site
2021-01-17 03:05:57,121:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/jfa-go.site
2021-01-17 03:05:57,122:DEBUG:certbot.reverter:Creating backup of /etc/letsencrypt/options-ssl-nginx.conf
2021-01-17 03:05:57,122:DEBUG:certbot.reverter:Creating backup of /etc/nginx/mime.types
2021-01-17 03:05:57,122:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/default.site
2021-01-17 03:05:57,122:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:

#user html;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
include /etc/letsencrypt/le_http_01_cert_challenge.conf;
server_names_hash_bucket_size 128;
    include       mime.types;
    default_type  application/octet-stream;
    types_hash_max_size 2048;
    types_hash_bucket_size 128;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

    include /etc/nginx/sites-enabled/*.site;

}

2021-01-17 03:05:57,140:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/sites-enabled/default.site:
server {
        root /data/personal-site;
        index index.html;
        server_name lollyde.online;
        error_log logs/default.error;
        access_log logs/default.access;

        location / {
                try_files $uri $uri/ =404; 
                add_header X-Easteregg "You're a curious one, arent you?";
        }


    listen 443 ssl; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/services.lollyde.online/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/services.lollyde.online/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


    if ($host = lollyde.online) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        server_name lollyde.online;
    listen 80;
    return 404; # managed by Certbot


location = /.well-known/acme-challenge/3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg{default_type text/plain;return 200 3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg.cQWlFwMR2AodvQBCVrbH6uusAspOQpUbIA_di1KaH78;} # managed by Certbot

}

2021-01-17 03:05:58,154:INFO:certbot._internal.auth_handler:Waiting for verification...
2021-01-17 03:05:58,155:DEBUG:certbot.display.util:Notifying user: Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.
2021-01-17 03:06:09,348:DEBUG:acme.client:JWS payload:
b'{}'
2021-01-17 03:06:09,352:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/MjsBqw:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDA0UzRBUG9haF94N3ZXTkhnVVc3LWxKNXp5dDRicFJKZjM4bkZvenN4TlhDRSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbC12My8xOTMwMDE4ODUvTWpzQnF3In0",
  "signature": "fCqkrkLRBWoNqPKGdsY-Ef1D4q4J6XkrF4Ugr1YDrSFwS5-sO79SizeMB3BZxSv0bRf5QIrmBGDkXltk-3v53hd22r32mMwdJhwETd4GTz8JNkqXSaEdv3iMoIOOvw02Q8aR7_QbdLzhk3Vc1bsi-ltDBVPbFy7t8zCeSeaVdlSA0618z91SERD0jIWJnzkGaAyk_poS-n3id5A1K3rIJ_efAEWrwG7frf6SACadhE8ReJQUPCkn7kIVSgOGTYrpjmDKGDxtWxJQqW3N-y9CMnpTaRclHCxNcNXup_tLHxIyZ_RLrZYmV3dQS0yPNNBLwA88BUMdAZLnlh8PaHrTBw",
  "payload": "e30"
}
2021-01-17 03:06:09,530:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/193001885/MjsBqw HTTP/1.1" 200 192
2021-01-17 03:06:09,532:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 02:06:09 GMT
Content-Type: application/json
Content-Length: 192
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193001885>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/MjsBqw
Replay-Nonce: 0004EegGpBtVV2BuKKQIqqBOCCkiCeBUgmv4mdWmIhFby9U
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/MjsBqw",
  "token": "3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg"
}
2021-01-17 03:06:09,532:DEBUG:acme.client:Storing nonce: 0004EegGpBtVV2BuKKQIqqBOCCkiCeBUgmv4mdWmIhFby9U
2021-01-17 03:06:10,535:DEBUG:acme.client:JWS payload:
b''
2021-01-17 03:06:10,537:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193001885:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDA0RWVnR3BCdFZWMkJ1S0tRSXFxQk9DQ2tpQ2VCVWdtdjRtZFdtSWhGYnk5VSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8xOTMwMDE4ODUifQ",
  "signature": "FHLag79hcNO5AbdzlO0dJuBlTHwG7OQTvIIsrCAzpswHZ5tG0ZeR7dQmVVZY60UCxF0ZyBwg5siJ9dE1ovUoNzfGM7ZxX1cDr2s1WTunEn8B2IjKqJ3rFs9sYnjRdHKcicwSqrKZZGXP9thHvWgIzLe6mT8G5hzbYnzTD5wmAKV-qMWY8d3QKz-iobHfiE88xUdGhZ2lJfbmoa5OYbKwZFSe-FU90eTXr6c8fFZrL8s_IJTm1Q_eqrsRcVBXrlGNYkXtCtkFuvEtad53gofAJgZOkUBECUvnBp32oeutBzy0-3yvff4a2Pl0mLygdYJMdU2K0gVplnpefNgH9imDVA",
  "payload": ""
}
2021-01-17 03:06:10,709:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/193001885 HTTP/1.1" 200 1823
2021-01-17 03:06:10,710:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 02:06:10 GMT
Content-Type: application/json
Content-Length: 1823
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0003PasC2JtM2sSz3pXQs3d-08EeVMkpswrURFxj2aB91o8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "lollyde.online"
  },
  "status": "invalid",
  "expires": "2021-01-24T02:05:56Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from https://lollyde.online/.well-known/acme-challenge/3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg [2606:4700:3033::6815:3b4b]: \"\u003chtml\u003e\\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\\n\u003cbody\u003e\\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\\n\u003chr\u003e\u003ccenter\u003enginx/1.19.6\u003c/center\"",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193001885/MjsBqw",
      "token": "3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg",
      "validationRecord": [
        {
          "url": "http://lollyde.online/.well-known/acme-challenge/3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg",
          "hostname": "lollyde.online",
          "port": "80",
          "addressesResolved": [
            "104.21.59.75",
            "172.67.218.107",
            "2606:4700:3030::ac43:da6b",
            "2606:4700:3033::6815:3b4b"
          ],
          "addressUsed": "2606:4700:3030::ac43:da6b"
        },
        {
          "url": "https://lollyde.online/.well-known/acme-challenge/3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg",
          "hostname": "lollyde.online",
          "port": "443",
          "addressesResolved": [
            "172.67.218.107",
            "104.21.59.75",
            "2606:4700:3033::6815:3b4b",
            "2606:4700:3030::ac43:da6b"
          ],
          "addressUsed": "2606:4700:3033::6815:3b4b"
        }
      ]
    }
  ]
}
2021-01-17 03:06:10,711:DEBUG:acme.client:Storing nonce: 0003PasC2JtM2sSz3pXQs3d-08EeVMkpswrURFxj2aB91o8
2021-01-17 03:06:10,712:WARNING:certbot._internal.auth_handler:Challenge failed for domain lollyde.online
2021-01-17 03:06:10,713:INFO:certbot._internal.auth_handler:http-01 challenge for lollyde.online
2021-01-17 03:06:10,713:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:

Domain: lollyde.online
Type:   unauthorized
Detail: Invalid response from https://lollyde.online/.well-known/acme-challenge/3sp8Fnqd6WHS7i6u_-6sdCydKBrmAIKK4aB7twq4jgg [2606:4700:3033::6815:3b4b]: "<html>\n<head><title>404 Not Found</title></head>\n<body>\n<center><h1>404 Not Found</h1></center>\n<hr><center>nginx/1.19.6</center"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2021-01-17 03:06:10,715:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-01-17 03:06:10,715:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-01-17 03:06:10,715:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-01-17 03:06:12,266:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/889/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/main.py", line 1421, in main
    return config.func(config, plugins)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/main.py", line 1294, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/main.py", line 135, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2021-01-17 03:06:12,267:ERROR:certbot._internal.log:Some challenges have failed.

I'm trying to think of any changes I've made since the last certificate renewal, but I cant think of any that are directly connected to that specific domain.

1 Like
6

I think that your nginx configuration might not be properly handling your IPv6 addresses (AAAA records). Try removing your AAAA records from your DNS zone then running certbot again.

1 Like
7

That is very odd.
I do not have any AAAA records set, nor have I ever had any, as far as I am aware. Either way, I just double checked to make sure there arent any, ran the certbot again and got the same error (invalid response 404), which makes sense since nothing changed.

1 Like
8

Better think again...

Screenshot_20210116-215238_Samsung Internet
Screenshot_20210116-215238_Samsung Internet1440Ă—1101 141 KB

1 Like
9

When using CloudFlare CDN, it is OK for them to publish IPv4 and IPv6 addresses.
They make no difference on the backend channel CloudFlare uses to reach your actual server.

1 Like
10

@rg305

I wasn't sure if the 404 response was from her configured server blocks or the default. My next avenue was to test for the correct webroot. I'll be away for a bit. Hopefully you can get her moved forward.

11

I would suggest to try option #4

Which can be verified before actually running certbot.

I would also suggest doing only --dry-run testing until this is corrected and working.

1 Like
12

I was not aware cloudflare publishes AAAA records even when I havent actually set any myself, that is very good to know though.

Option 4 works as it should!

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Could not find ssl_module; not disabling session tickets.

How would you like to authenticate with the ACME CA?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: Apache Web Server plugin (apache)
2: Nginx Web Server plugin (nginx)
3: Spin up a temporary webserver (standalone)
4: Place files in webroot directory (webroot)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-4] then [enter] (press 'c' to cancel): 4
Plugins selected: Authenticator webroot, Installer None
Simulating a certificate request for lollyde.online
Performing the following challenges:
http-01 challenge for lollyde.online
Input the webroot for lollyde.online: (Enter 'c' to cancel): /data/personal-site
Waiting for verification...
Cleaning up challenges

IMPORTANT NOTES:
 - The dry run was successful.

I can use this as a workaround, however I'd love to actually fix whatever problem might be going on

1 Like
13

Add this line:

listen [::]:80 ipv6only=on;

to the server block in your nginx configuration that contains this:

listen 80;

Add this line:

listen [::]:443 ssl ipv6only=on;

to the server block in your nginx configuration that contains this:

listen 443 ssl;

Run this:

nginx -s reload

Run this:

sudo certbot certonly --nginx -d "lollyde.online" --dry-run

Run this:

sudo certbot --nginx -d "lollyde.online"

Run this:

sudo certbot renew --dry-run

:partying_face:

14

New nginx config:

server {
        root /data/personal-site;
        index index.html;
        server_name lollyde.online;
        error_log logs/default.error debug;
        access_log logs/default.access;

        location / {
                try_files $uri $uri/ =404; 
                add_header X-Easteregg "You're a curious one, arent you?";
        }


    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl ipv6only=on;
    ssl_certificate /etc/letsencrypt/live/services.lollyde.online/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/services.lollyde.online/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {
    if ($host = lollyde.online) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        server_name lollyde.online;
    listen 80;
    listen [::]:80 ipv6only=on;
    return 404; # managed by Certbot
}

sudo nginx -s reload runs without problems
sudo certbot certonly --nginx -d "lollyde.online" --dry-run results in

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Simulating a certificate request for lollyde.online
Performing the following challenges:
http-01 challenge for lollyde.online
Waiting for verification...
Challenge failed for domain lollyde.online
http-01 challenge for lollyde.online
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: lollyde.online
   Type:   unauthorized
   Detail: Invalid response from
   https://lollyde.online/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0
   [2606:4700:3030::ac43:da6b]: "<html>\n<head><title>404 Not
   Found</title></head>\n<body>\n<center><h1>404 Not
   Found</h1></center>\n<hr><center>nginx/1.19.6</center"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.

same as before.

Here's the letsencrypt.log:

2021-01-17 17:36:43,123:DEBUG:urllib3.connectionpool:http://localhost:None "GET /v2/connections?snap=certbot&interface=content HTTP/1.1" 200 97
2021-01-17 17:36:43,268:DEBUG:certbot._internal.main:certbot version: 1.11.0
2021-01-17 17:36:43,269:DEBUG:certbot._internal.main:Location of certbot entry point: /snap/certbot/889/bin/certbot
2021-01-17 17:36:43,269:DEBUG:certbot._internal.main:Arguments: ['--nginx', '-d', 'lollyde.online', '--dry-run', '--preconfigured-renewal']
2021-01-17 17:36:43,269:DEBUG:certbot._internal.main:Discovered plugins: PluginsRegistry(PluginEntryPoint#apache,PluginEntryPoint#manual,PluginEntryPoint#nginx,PluginEntryPoint#null,PluginEntryPoint#standalone,PluginEntryPoint#webroot)
2021-01-17 17:36:43,282:DEBUG:certbot._internal.log:Root logging level set at 20
2021-01-17 17:36:43,282:INFO:certbot._internal.log:Saving debug log to /var/log/letsencrypt/letsencrypt.log
2021-01-17 17:36:43,282:DEBUG:certbot._internal.plugins.selection:Requested authenticator nginx and installer nginx
2021-01-17 17:36:43,792:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7faf5f9831c0>
Prep: True
2021-01-17 17:36:43,792:DEBUG:certbot._internal.plugins.selection:Single candidate plugin: * nginx
Description: Nginx Web Server plugin
Interfaces: IAuthenticator, IInstaller, IPlugin
Entry point: nginx = certbot_nginx._internal.configurator:NginxConfigurator
Initialized: <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7faf5f9831c0>
Prep: True
2021-01-17 17:36:43,792:DEBUG:certbot._internal.plugins.selection:Selected authenticator <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7faf5f9831c0> and installer <certbot_nginx._internal.configurator.NginxConfigurator object at 0x7faf5f9831c0>
2021-01-17 17:36:43,793:INFO:certbot._internal.plugins.selection:Plugins selected: Authenticator nginx, Installer nginx
2021-01-17 17:36:43,795:DEBUG:certbot._internal.main:Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/16945623', new_authzr_uri=None, terms_of_service=None), 1a1b549c66ad1ad7a4d61cb9012e1dc1, Meta(creation_dt=datetime.datetime(2020, 12, 4, 18, 20, 59, tzinfo=<UTC>), creation_host='lollyde-rootserver', register_to_eff=None))>
2021-01-17 17:36:43,796:DEBUG:acme.client:Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
2021-01-17 17:36:43,796:DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
2021-01-17 17:36:44,488:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 724
2021-01-17 17:36:44,488:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 16:36:44 GMT
Content-Type: application/json
Content-Length: 724
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "MUq9IS9XKD4": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
2021-01-17 17:36:44,491:DEBUG:certbot.display.util:Notifying user: Simulating a certificate request for lollyde.online
2021-01-17 17:36:44,523:DEBUG:acme.client:Requesting fresh nonce
2021-01-17 17:36:44,523:DEBUG:acme.client:Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
2021-01-17 17:36:44,694:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
2021-01-17 17:36:44,694:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 16:36:44 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0004CFpC8vEzqzOLhOqq4xft9NrR_20ixc791W_KcJEjyYw
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


2021-01-17 17:36:44,694:DEBUG:acme.client:Storing nonce: 0004CFpC8vEzqzOLhOqq4xft9NrR_20ixc791W_KcJEjyYw
2021-01-17 17:36:44,694:DEBUG:acme.client:JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "lollyde.online"\n    }\n  ]\n}'
2021-01-17 17:36:44,696:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDA0Q0ZwQzh2RXpxek9MaE9xcTR4ZnQ5TnJSXzIwaXhjNzkxV19LY0pFanlZdyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
  "signature": "nVdgFsPhr9d0EbL5D3T1PNRHljh-4ivxeLeKPdctFrpKCwxZDvhvhizdEfNveJHcKVvRW5pago00AvWgLlhpzlr-Umuc85TaEIK_eDjjCaqNZCEjUrLpiTQ9JuVQ3y1lH-G1AwT7JWDLFdTB1Hns4tJOkg7qHhKgIyxziDmR_3yp6_saO3ld6EAgib1JNEnSVwgxOLb9FJn2P5nJZkoYH8hr1HrZHVdiP2pqOnbSKkusUnt_02yCjEYNSizPaeZJiblZqGM56lECc-8-SJVmwkVGwVCkb6FYWg4EQw0Tg6jqT-lN95h__-9YqoZ46eHSRSrqYbHL8ZlGO-OpXHf7qA",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImxvbGx5ZGUub25saW5lIgogICAgfQogIF0KfQ"
}
2021-01-17 17:36:44,898:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 348
2021-01-17 17:36:44,899:DEBUG:acme.client:Received response:
HTTP 201
Server: nginx
Date: Sun, 17 Jan 2021 16:36:44 GMT
Content-Type: application/json
Content-Length: 348
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/16945623/223211728
Replay-Nonce: 0004VEypc2TOB4cEkZ8TIJAOKEDmcI-5Syk7PzD6hbgNr0Y
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2021-01-24T16:36:44Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "lollyde.online"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193433848"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/16945623/223211728"
}
2021-01-17 17:36:44,900:DEBUG:acme.client:Storing nonce: 0004VEypc2TOB4cEkZ8TIJAOKEDmcI-5Syk7PzD6hbgNr0Y
2021-01-17 17:36:44,900:DEBUG:acme.client:JWS payload:
b''
2021-01-17 17:36:44,901:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193433848:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDA0VkV5cGMyVE9CNGNFa1o4VElKQU9LRURtY0ktNVN5azdQekQ2aGJnTnIwWSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8xOTM0MzM4NDgifQ",
  "signature": "Z2OMMKaFSap9aVMFFqBOnKnFB9dsbJK03hWEnA_uSUEzagTTBfsgPnU2uS_S_c-U5Pv7bMNTZ52l2tI4sAd-Vd07hDYm98DmgszLRpDHOVu0XlMC0DQbjuNvbVYSh0tdGQBgm1zmCwSeYcqx9jf9qac51FXnfS4xCAWeZLbqneUD_sx0Xka4UAU9SM7HNdfKTWlc8AHII0cSkQROi46_O2GEY_KKgY9weTj3M6KK86nE4dyOybe_KgEYK0OTX-7b9QFBaJEN_y7b6IZlTNFuv2o_wiztrzLvlmF_xOdoFdlat_7IN4SBAP4WdKBfVf4F7KLAaiJ775bgIiUVYrtSYg",
  "payload": ""
}
2021-01-17 17:36:45,078:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/193433848 HTTP/1.1" 200 813
2021-01-17 17:36:45,079:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 16:36:44 GMT
Content-Type: application/json
Content-Length: 813
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0004dSG4fuLhBe8WS5kVP0r_1_Kd0nusDS0xuHEm2ZwXJrE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "lollyde.online"
  },
  "status": "pending",
  "expires": "2021-01-24T16:36:44Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/CxTlkg",
      "token": "a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/8V93eA",
      "token": "a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/8phYOQ",
      "token": "a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
    }
  ]
}
2021-01-17 17:36:45,079:DEBUG:acme.client:Storing nonce: 0004dSG4fuLhBe8WS5kVP0r_1_Kd0nusDS0xuHEm2ZwXJrE
2021-01-17 17:36:45,079:INFO:certbot._internal.auth_handler:Performing the following challenges:
2021-01-17 17:36:45,080:INFO:certbot._internal.auth_handler:http-01 challenge for lollyde.online
2021-01-17 17:36:45,107:DEBUG:certbot_nginx._internal.http_01:Generated server block:
[]
2021-01-17 17:36:45,108:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/default.site
2021-01-17 17:36:45,108:DEBUG:certbot.reverter:Creating backup of /etc/nginx/nginx.conf
2021-01-17 17:36:45,108:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/jfa-go.site
2021-01-17 17:36:45,108:DEBUG:certbot.reverter:Creating backup of /etc/letsencrypt/options-ssl-nginx.conf
2021-01-17 17:36:45,108:DEBUG:certbot.reverter:Creating backup of /etc/nginx/mime.types
2021-01-17 17:36:45,108:DEBUG:certbot.reverter:Creating backup of /etc/nginx/sites-enabled/services.site
2021-01-17 17:36:45,109:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/nginx.conf:

#user html;
worker_processes  1;

#error_log  logs/error.log;
#error_log  logs/error.log  notice;
#error_log  logs/error.log  info;

#pid        logs/nginx.pid;


events {
    worker_connections  1024;
}


http {
include /etc/letsencrypt/le_http_01_cert_challenge.conf;
server_names_hash_bucket_size 128;
    include       mime.types;
    default_type  application/octet-stream;
    types_hash_max_size 2048;
    types_hash_bucket_size 128;

    log_format  main  '$remote_addr - $remote_user [$time_local] "$request" '
                      '$status $body_bytes_sent "$http_referer" '
                      '"$http_user_agent" "$http_x_forwarded_for"';

    access_log  logs/access.log  main;

    sendfile        on;
    #tcp_nopush     on;

    #keepalive_timeout  0;
    keepalive_timeout  65;

    #gzip  on;

    # another virtual host using mix of IP-, name-, and port-based configuration
    #
    #server {
    #    listen       8000;
    #    listen       somename:8080;
    #    server_name  somename  alias  another.alias;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}


    # HTTPS server
    #
    #server {
    #    listen       443 ssl;
    #    server_name  localhost;

    #    ssl_certificate      cert.pem;
    #    ssl_certificate_key  cert.key;

    #    ssl_session_cache    shared:SSL:1m;
    #    ssl_session_timeout  5m;

    #    ssl_ciphers  HIGH:!aNULL:!MD5;
    #    ssl_prefer_server_ciphers  on;

    #    location / {
    #        root   html;
    #        index  index.html index.htm;
    #    }
    #}

    include /etc/nginx/sites-enabled/*.site;

}

2021-01-17 17:36:45,110:DEBUG:certbot_nginx._internal.parser:Writing nginx conf tree to /etc/nginx/sites-enabled/default.site:
server {
        root /data/personal-site;
        index index.html;
        server_name lollyde.online;
        error_log logs/default.error debug;
        access_log logs/default.access;

        location / {
                try_files $uri $uri/ =404; 
                add_header X-Easteregg "You're a curious one, arent you?";
        }


    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl ipv6only=on;
    ssl_certificate /etc/letsencrypt/live/services.lollyde.online/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/services.lollyde.online/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


    if ($host = lollyde.online) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        server_name lollyde.online;
    listen 80;
    listen [::]:80 ipv6only=on;
    return 404; # managed by Certbot


location = /.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0{default_type text/plain;return 200 a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0.cQWlFwMR2AodvQBCVrbH6uusAspOQpUbIA_di1KaH78;} # managed by Certbot

}

2021-01-17 17:36:46,125:INFO:certbot._internal.auth_handler:Waiting for verification...
2021-01-17 17:36:46,126:DEBUG:acme.client:JWS payload:
b'{}'
2021-01-17 17:36:46,129:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/CxTlkg:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDA0ZFNHNGZ1TGhCZThXUzVrVlAwcl8xX0tkMG51c0RTMHh1SEVtMlp3WEpyRSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbC12My8xOTM0MzM4NDgvQ3hUbGtnIn0",
  "signature": "PnnLwBLUPv-Ih9vVzEkwiwUXn7NOheueTokDhR3Y56nAzNl5SDkGWLIoh302J0o17zD2X97mwpy9OCRjHEc-OWtJCcabh66PIXfJCiiz_j5hm3sFfJBEbrKYIlLSHlO9XJCnjbsi7g7X6YhrvLv7J5hNsS9uaCuOP5tYTQopqARQsk-QAmcwazvRwJkWYoZG56Fw4RB46tvg6uxp0A5X54fLFYoWIe7FR8qGF-FJJ75r9fkGmU7ysyyLct45e67MKepvK0zeRPE_bZMR4cbmY-7I00U1wf5FUC7KLPkFNircuOAVFCxrHXdbCl5zVVf2tLsPNynVuOFn8qxXdxInMw",
  "payload": "e30"
}
2021-01-17 17:36:46,308:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/193433848/CxTlkg HTTP/1.1" 200 192
2021-01-17 17:36:46,309:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 16:36:46 GMT
Content-Type: application/json
Content-Length: 192
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193433848>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/CxTlkg
Replay-Nonce: 0003fHaWGmqFIGDYk5gKKmeQwKLluO9-yEMnoXB7IG6F56E
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "http-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/CxTlkg",
  "token": "a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
}
2021-01-17 17:36:46,309:DEBUG:acme.client:Storing nonce: 0003fHaWGmqFIGDYk5gKKmeQwKLluO9-yEMnoXB7IG6F56E
2021-01-17 17:36:47,311:DEBUG:acme.client:JWS payload:
b''
2021-01-17 17:36:47,315:DEBUG:acme.client:Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/193433848:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC8xNjk0NTYyMyIsICJub25jZSI6ICIwMDAzZkhhV0dtcUZJR0RZazVnS0ttZVF3S0xsdU85LXlFTW5vWEI3SUc2RjU2RSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My8xOTM0MzM4NDgifQ",
  "signature": "v7XCSCWSGHYFoyi5Zp8Go7jo6xtELdmQ5xe6K6UB0sNQOdtKvjoJe4KcN669eaWWvQKNJQqwu9Csq-ZvBQzgAfChxCYqWwMUPls9SBkw0kOxG771O4idxDkS3WsMiS4DJOnhKtS29dzhv5cbJctf2EOll4oQBe7a8mkycFg0Hce7j4QIk_8K-WgUP3rwqjbpaSsRvnrdpsy6VjpDGN-hxp51Dx0CqtGnWKtVjahluf-tIyCGlk9WZTBtLXrbXaps8_n8bsvg9HjT4xgvkcly0948d5b6nh1jCA4fIOlkzIn7zNk50F4-WrzgqSAFnD_lilmAZ8nBvBg_h3R3xZJ5_g",
  "payload": ""
}
2021-01-17 17:36:47,492:DEBUG:urllib3.connectionpool:https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/193433848 HTTP/1.1" 200 1823
2021-01-17 17:36:47,493:DEBUG:acme.client:Received response:
HTTP 200
Server: nginx
Date: Sun, 17 Jan 2021 16:36:47 GMT
Content-Type: application/json
Content-Length: 1823
Connection: keep-alive
Boulder-Requester: 16945623
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: 0003ZImKcQSnp51AzC9Esod5a9Lf_1VI_Zgpu-uafaWwtcE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "lollyde.online"
  },
  "status": "invalid",
  "expires": "2021-01-24T16:36:44Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:unauthorized",
        "detail": "Invalid response from https://lollyde.online/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0 [2606:4700:3030::ac43:da6b]: \"\u003chtml\u003e\\n\u003chead\u003e\u003ctitle\u003e404 Not Found\u003c/title\u003e\u003c/head\u003e\\n\u003cbody\u003e\\n\u003ccenter\u003e\u003ch1\u003e404 Not Found\u003c/h1\u003e\u003c/center\u003e\\n\u003chr\u003e\u003ccenter\u003enginx/1.19.6\u003c/center\"",
        "status": 403
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/193433848/CxTlkg",
      "token": "a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0",
      "validationRecord": [
        {
          "url": "http://lollyde.online/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0",
          "hostname": "lollyde.online",
          "port": "80",
          "addressesResolved": [
            "104.21.59.75",
            "172.67.218.107",
            "2606:4700:3033::6815:3b4b",
            "2606:4700:3030::ac43:da6b"
          ],
          "addressUsed": "2606:4700:3033::6815:3b4b"
        },
        {
          "url": "https://lollyde.online/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0",
          "hostname": "lollyde.online",
          "port": "443",
          "addressesResolved": [
            "172.67.218.107",
            "104.21.59.75",
            "2606:4700:3030::ac43:da6b",
            "2606:4700:3033::6815:3b4b"
          ],
          "addressUsed": "2606:4700:3030::ac43:da6b"
        }
      ]
    }
  ]
}
2021-01-17 17:36:47,494:DEBUG:acme.client:Storing nonce: 0003ZImKcQSnp51AzC9Esod5a9Lf_1VI_Zgpu-uafaWwtcE
2021-01-17 17:36:47,495:WARNING:certbot._internal.auth_handler:Challenge failed for domain lollyde.online
2021-01-17 17:36:47,495:INFO:certbot._internal.auth_handler:http-01 challenge for lollyde.online
2021-01-17 17:36:47,496:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:

Domain: lollyde.online
Type:   unauthorized
Detail: Invalid response from https://lollyde.online/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0 [2606:4700:3030::ac43:da6b]: "<html>\n<head><title>404 Not Found</title></head>\n<body>\n<center><h1>404 Not Found</h1></center>\n<hr><center>nginx/1.19.6</center"

To fix these errors, please make sure that your domain name was entered correctly and the DNS A/AAAA record(s) for that domain contain(s) the right IP address.
2021-01-17 17:36:47,497:DEBUG:certbot._internal.error_handler:Encountered exception:
Traceback (most recent call last):
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

2021-01-17 17:36:47,497:DEBUG:certbot._internal.error_handler:Calling registered functions
2021-01-17 17:36:47,498:INFO:certbot._internal.auth_handler:Cleaning up challenges
2021-01-17 17:36:49,015:DEBUG:certbot._internal.log:Exiting abnormally:
Traceback (most recent call last):
  File "/snap/certbot/889/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/main.py", line 15, in main
    return internal_main.main(cli_args)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/main.py", line 1421, in main
    return config.func(config, plugins)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/main.py", line 1294, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/main.py", line 135, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/client.py", line 441, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/client.py", line 374, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/client.py", line 421, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, best_effort)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 91, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, best_effort)
  File "/var/lib/snapd/snap/certbot/889/lib/python3.8/site-packages/certbot/_internal/auth_handler.py", line 180, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
2021-01-17 17:36:49,016:ERROR:certbot._internal.log:Some challenges have failed.

This section from the nginx error log leads me to believe there might be some sort of issue with certbot not reloading nginx:

2021/01/17 17:36:46 [debug] 36414#36414: *141216 try files handler
2021/01/17 17:36:46 [debug] 36414#36414: *141216 http script var: "/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
2021/01/17 17:36:46 [debug] 36414#36414: *141216 trying to use file: "/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0" "/data/personal-site/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
2021/01/17 17:36:46 [debug] 36414#36414: *141216 http script var: "/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
2021/01/17 17:36:46 [debug] 36414#36414: *141216 trying to use dir: "/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0" "/data/personal-site/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0"
2021/01/17 17:36:46 [debug] 36414#36414: *141216 trying to use file: "=404" "/data/personal-site=404"
2021/01/17 17:36:46 [debug] 36414#36414: *141216 http finalize request: 404, "/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0?" a:1, c:1
2021/01/17 17:36:46 [debug] 36414#36414: *141216 http special response: 404, "/.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0?"
2021/01/17 17:36:46 [debug] 36414#36414: *141216 http set discard body
2021/01/17 17:36:46 [debug] 36414#36414: *141216 HTTP/1.1 404 Not Found

since it looks like it hits the try_files handler in the / location from my own nginx config, instead of the .well-known location in the config that certbot tries to load (notice how it does not have a try_files handler in the /.well-known/acme-challenge location):

server {
        root /data/personal-site;
        index index.html;
        server_name lollyde.online;
        error_log logs/default.error debug;
        access_log logs/default.access;

        location / {
                try_files $uri $uri/ =404; 
                add_header X-Easteregg "You're a curious one, arent you?";
        }


    listen 443 ssl; # managed by Certbot
    listen [::]:443 ssl ipv6only=on;
    ssl_certificate /etc/letsencrypt/live/services.lollyde.online/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/services.lollyde.online/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

}
server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


    if ($host = lollyde.online) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        server_name lollyde.online;
    listen 80;
    listen [::]:80 ipv6only=on;
    return 404; # managed by Certbot


location = /.well-known/acme-challenge/a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0{default_type text/plain;return 200 a01SN_0mpRrnwkYYE4Ld1Si8YKX7LT_fcKvfY8lOSu0.cQWlFwMR2AodvQBCVrbH6uusAspOQpUbIA_di1KaH78;} # managed by Certbot

}

I've tried again with inotifywait -m . in the directory in which my nginx configurations are stored, and it looks like certbot has no trouble writing to the config files.

Running nginx with a debug level error log and retrying reveals that it looks like the process does get restarted, however I am just guessing here since I do not have deep knowledge on how to read a nginx debug log.
Heres the output when running sudo nginx -s reload and then sudo certbot certonly --nginx -d "lollyde.online" --dry-run

2021/01/17 17:59:11 [debug] 44948#44948: epoll add event: fd:7 op:1 ev:00002001    <- manual nginx reload
2021/01/17 17:59:11 [debug] 44948#44948: epoll add event: fd:18 op:1 ev:00002001
2021/01/17 17:59:11 [debug] 44832#44832: epoll del event: fd:7 op:2 ev:00000000
2021/01/17 17:59:11 [debug] 44832#44832: epoll del event: fd:18 op:2 ev:00000000    <- manual nginx reload done
2021/01/17 17:59:23 [debug] 45047#45047: epoll add event: fd:7 op:1 ev:00002001    <- starting certbot
2021/01/17 17:59:23 [debug] 45047#45047: epoll add event: fd:18 op:1 ev:00002001
2021/01/17 17:59:23 [debug] 44948#44948: epoll del event: fd:7 op:2 ev:00000000
2021/01/17 17:59:23 [debug] 44948#44948: epoll del event: fd:18 op:2 ev:00000000
2021/01/17 17:59:26 [debug] 45054#45054: epoll add event: fd:7 op:1 ev:00002001
2021/01/17 17:59:26 [debug] 45054#45054: epoll add event: fd:18 op:1 ev:00002001
2021/01/17 17:59:26 [debug] 45047#45047: epoll del event: fd:7 op:2 ev:00000000
2021/01/17 17:59:26 [debug] 45047#45047: epoll del event: fd:18 op:2 ev:00000000    <- certbot done

The output of the certbot command is

sudo certbot certonly --nginx -d "lollyde.online" --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Simulating a certificate request for lollyde.online
Performing the following challenges:
http-01 challenge for lollyde.online
Waiting for verification...
Challenge failed for domain lollyde.online
http-01 challenge for lollyde.online
Cleaning up challenges
Some challenges have failed.

IMPORTANT NOTES:
 - The following errors were reported by the server:

   Domain: lollyde.online
   Type:   unauthorized
   Detail: Invalid response from
   https://lollyde.online/.well-known/acme-challenge/8owxV1bJXmqEGIZlsxs0INU39UtolOcUHLHsCoC3uAQ
   [2606:4700:3030::ac43:da6b]: "<html>\n<head><title>404 Not
   Found</title></head>\n<body>\n<center><h1>404 Not
   Found</h1></center>\n<hr><center>nginx/1.19.6</center"

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address.
1 Like
15

I'm starting to think that you might have some incompatible (invisible) characters in your configuration file. I'm going to tag someone who can likely take one look at this and know exactly what the issue is.

@_az

I'm almost certain you know exactly what's going on here. I believe I've been involved in a topic with you once before where an issue very similar to this has arisen, but I can't remember where. I think you only need to look at Lollyde's post right above this one.

False alarm. @JuergenAuer set things straight.

1 Like
16

If it is an invisible character in the configuration file, wouldnt just retyping the configuration file fix the issue? I'll give it a shot, worst case nothing happens.

1 Like
17

Yep. There's probably a sed that will find it/them too.

1 Like
18

Hi @Lollyde

please read some required basics:

If you use --nginx, your http config file is temporary modified - /.well-known/acme-challenge/random-filename -> required content.

So https isn't changed.

But if you use Cloudflare, there is a redirect http -> https

Detail: Invalid response from
   https://lollyde.online/.well-known/acme-challenge/8owxV1bJXmqEGIZlsxs0INU39UtolOcUHLHsCoC3uAQ
   [2606:4700:3030::ac43:da6b]: "<

so that check can't see the validation content.

That's a fundamental misconception and has nothing to do with UTF-8 characters.

Use webroot.

2 Likes
19

OH! :woman_facepalming: of course.

I did see that the request was coming in as https and I did see that the change by certbot was made in the http section, but I did not put two and two together.

Thank you!

2 Likes
20

Thanks @JuergenAuer. I think you mentioned that somewhere before and I missed that part. I did find it curious that her server blocks didn't have IPv6 listens though.

1 Like