Certbot nginx seems to be looking for the challenge in the wrong place

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
solr.iges.jp

I ran this command:
sudo certbot --nginx -d solr1.iges.jp

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for solr1.iges.jp
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. solr1.iges.jp (http-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Invalid response from https://www.iges.or.jp/.well-known/acme-challenge/ex_UfY-isZ8-0PAvAD6ulw4YDi-IBeuwmr-mHsQ49HY? [2606:4700:10::ac43:a1a]: "\n404 Not Found\n<body bgcolor="white">\n

404 Not Found

\n
ngin"

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: solr1.iges.jp
    Type: unauthorized
    Detail: Invalid response from
    https://www.iges.or.jp/.well-known/acme-challenge/ex_UfY-isZ8-0PAvAD6ulw4YDi-IBeuwmr-mHsQ49HY?
    [2606:4700:10::ac43:a1a]: "\n404 Not
    Found\n<body bgcolor="white">\n

    404 Not
    Found

    \n
    ngin"

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address.

My web server is (include version):
nginx version: nginx/1.14.0 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 18.04.5 LTS

My hosting provider, if applicable, is:
Linode

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.31.0

Note that the Invalid response seems to be coming from another of our domains, and I can't figure out where it might be getting redirected. How can I debug this?

Well the message shows that the original request has been redirected to another FQDN.
You will need to review the vhost config and likely make an exception to not redirect the challenge requests.

And confirmed with:

curl -Iki solr1.iges.jp/.well-known/acme-challenge/Test-File-1234

HTTP/1.1 301 Moved Permanently
Server: nginx/1.14.0 (Ubuntu)
Date: Wed, 25 Aug 2021 10:06:20 GMT
Content-Type: text/html
Content-Length: 194
Connection: keep-alive
Location: https://www.iges.or.jp/.well-known/acme-challenge/Test-File-1234?

Start with the output of:
sudo nginx -T

Thanks @rg305

There must be some weird dns cache somewhere or something. When I run the same command, I get:

 ~  curl -Iki solr1.iges.jp/.well-known/acme-challenge/Test-File-1234
HTTP/1.1 404 Not Found
Server: nginx/1.14.0 (Ubuntu)
Date: Wed, 25 Aug 2021 10:07:47 GMT
Content-Type: text/html; charset=utf-8
Content-Length: 178
Connection: keep-alive

Also I noticed that somehow I had a prehistoric version of certbot, so I reinstalled it, but get much the same output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for solr1.iges.jp

Certbot failed to authenticate some domains (authenticator: nginx). The Certificate Authority reported these problems:
  Domain: solr1.iges.jp
  Type:   unauthorized
  Detail: Invalid response from https://www.iges.or.jp/.well-known/acme-challenge/-murAGIIRzUBu2w4fIz_xHdZ6M3H_2bjbH4O8f8K1IE? [2606:4700:10::6816:b1b]: "<html>\n<head><title>404 Not Found</title></head>\n<body bgcolor=\"white\">\n<center><h1>404 Not Found</h1></center>\n<hr><center>ngin"

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
1 Like

The version of certbot is not an issue here.
[Although it was somewhat prehistoric - LOL]

DNS cache isn't to blame.
I've never visited the site and it gave me the same results.

Let's have a look at the output of:
sudo nginx -T
and
sudo ps -ef | grep -i nginx

Was that from the same system or from the Internet?
[what the server "sees" doesn't always match what the world "sees"]

Sorry, I meant to include my virtual host config from nginx, but I forgot. I'll nclude that here. The previous curl output was run on my laptop terminal, not from the server.

solr1.iges.jp.conf:

server {
    server_name    solr1.iges.jp;
    root           /var/www/my-iges-solr.iges.jp/testbed;
    index          index.html;
    autoindex      off;
    charset        utf-8;
    error_page     404 /404.html;
    error_log      /var/log/nginx/error.solr1.iges.jp.log;
    access_log     /var/log/nginx/access.solr1.iges.jp.log;
#    include        global/restrictions.conf;
#    auth_basic    "Authorized access only.";
#    auth_basic_user_file /var/www/my-iges-solr.iges.jp/.htpasswd;
    # Very rarely should these ever be accessed outside of your lan
    location ~* \.(txt|log)$ {
        allow 192.168.0.0/16;
        deny all;
    }

    location ~ \..*/.*\.php$ {
        return 403;
    }

    # Block access to scripts in site files directory
    location ~ ^/sites/[^/]+/files/.*\.php$ {
        deny all;
    }

    # Allow "Well-Known URIs" as per RFC 5785
    location ~* ^/.well-known/acme-challenge/ {
        allow all;
        auth_basic off;
    }

    # Block access to "hidden" files and directories whose names begin with a
    # period. This includes directories used by version control systems such
    # as Subversion or Git to store control files.
#    location ~ (^|/)\. {
#        return 403;
#    }

    location /solr {
             proxy_set_header Host $host;
             proxy_set_header X-Real-IP $remote_addr;
             proxy_pass "http://127.0.0.1:8983";
    }

    location /data {
         return 403;
    }

    location /conf {
        return 403;
    }

    location /logs {
        return 403;
    }

}

 sudo ps -ef | grep -i nginx                                (master|✚1…84)
root     23283     1  0 17:50 ?        00:00:00 nginx: master process /usr/sbin/nginx -g daemon on; master_process on;
www-data 31686 23283  0 19:05 ?        00:00:00 nginx: worker process
www-data 31687 23283  0 19:05 ?        00:00:00 nginx: worker process
bob      32260  4975  0 19:18 pts/2    00:00:00 grep --color=auto --exclude-dir=.bzr --exclude-dir=CVS --exclude-dir=.git --exclude-dir=.hg --exclude-dir=.svn --exclude-dir=.idea --exclude-dir=.tox -i nginx

Would that connect locally to the server (same private network) or over the Internet?

It would connect over the internet the server is in a data center, not on the same network as me. I also tried the command from a different linode server and got the same result

Well the ps output looks good.

The provided piece of the nginx config doesn't show the problem - it must be elsewhere.
If you have trust issues/security concerns (I know I sure do), you can PM the output to me - so the entire Internet doesn't see it.
But I'm certain the problem can be found therein.

Well that throws a monkey-wrench in this fix plan...

Please show the output from your laptop:
ping solr1.iges.jp

You can also find a test file in place at http://solr1.iges.jp/.well-known/acme-challenge/ok.html

html test files are not best suited for testing that directory.

curl -Iki http://solr1.iges.jp/.well-known/acme-challenge/ok.html
HTTP/1.1 301 Moved Permanently
Location: https://www.iges.or.jp/.well-known/acme-challenge/ok.html?

OK this might be to blame:

Name:      solr1.iges.jp
Addresses: 2400:8902::f03c:92ff:fe7a:8e69
           139.162.123.237

Yes it is!

curl -Iki4 http://solr1.iges.jp/.well-known/acme-challenge/ok.html
HTTP/1.1 200 OK

curl -Iki6 http://solr1.iges.jp/.well-known/acme-challenge/ok.html
HTTP/1.1 301 Moved Permanently
Location: https://www.iges.or.jp/.well-known/acme-challenge/ok.html?
1 Like

OK, just a second, I'll remove the extension

ping output from a different server:

ping solr1.iges.jp
PING solr1.iges.jp (139.162.123.237) 56(84) bytes of data.
64 bytes from beryllium.iges.jp (139.162.123.237): icmp_seq=1 ttl=63 time=0.393 ms
64 bytes from beryllium.iges.jp (139.162.123.237): icmp_seq=2 ttl=63 time=0.452 ms
64 bytes from beryllium.iges.jp (139.162.123.237): icmp_seq=3 ttl=63 time=0.457 ms
64 bytes from beryllium.iges.jp (139.162.123.237): icmp_seq=4 ttl=63 time=0.421 ms
^C
--- solr1.iges.jp ping statistics ---
47 packets transmitted, 47 received, 0% packet loss, time 49697ms
rtt min/avg/max/mdev = 0.311/0.476/0.619/0.063 ms

ping output from laptop:

~  ping solr1.iges.jp
PING solr1.iges.jp (139.162.123.237): 56 data bytes
64 bytes from 139.162.123.237: icmp_seq=0 ttl=55 time=12.763 ms
64 bytes from 139.162.123.237: icmp_seq=1 ttl=55 time=12.419 ms
64 bytes from 139.162.123.237: icmp_seq=2 ttl=55 time=17.871 ms
64 bytes from 139.162.123.237: icmp_seq=3 ttl=55 time=19.560 ms
64 bytes from 139.162.123.237: icmp_seq=4 ttl=55 time=18.658 ms
^C
--- solr1.iges.jp ping statistics ---
5 packets transmitted, 5 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 12.419/16.254/19.560/3.040 ms

hmmm, that should havebeen the right ipv6 address, bt let me double check, thank you!

1 Like

Even if it is the right IP address, make sure that nginx is listening on both IPv4 and IPv6.
There are no listen statements shown in the file: solr1.iges.jp.conf

@rg305 thank you, I'm definitely learning some curl-foo

I added a listen directive in there, could you see if that command still gives you the same error?

1 Like
curl -Iki6 http://solr1.iges.jp/.well-known/acme-challenge/ok.html
HTTP/1.1 200 OK

Back in business!

Great, let me try that certbot command again!