Invalid Response 404 on Simple Nginx Config

[mrhamel]

My domain is: autismkarate.org

I ran this command: certbot renew -v --dry-run --test-cert

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Processing /etc/letsencrypt/renewal/as21805.net-0002.conf
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Certificate not due for renewal, but simulating renewal for dry run
Plugins selected: Authenticator nginx, Installer nginx
Simulating renewal of an existing certificate for as21805.net and 22 more domains
Performing the following challenges:
(snip)
http-01 challenge for autismkarate.org
(snip)
Waiting for verification...
(snip)
Challenge failed for domain autismkarate.org
(snip)
http-01 challenge for autismkarate.org

(snip)
  Domain: autismkarate.org
  Type:   unauthorized
  Detail: During secondary validation: (v6 IP): Invalid response from http://autismkarate.org/.well-known/acme-challenge/(challenge): 404

  Domain: www.autismkarate.org
  Type:   unauthorized
  Detail: During secondary validation: (v6 IP): Invalid response from http://www.autismkarate.org/.well-known/acme-challenge/(challenge): 404
(snip)

Hint: The Certificate Authority failed to verify the temporary nginx configuration changes made by Certbot. Ensure the listed domains point to this nginx server and that it is accessible from the internet.

Cleaning up challenges
Failed to renew certificate as21805.net-0002 with error: Some challenges have failed.

- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
All simulated renewals failed. The following certificates could not be renewed:
  /etc/letsencrypt/live/as21805.net-0002/fullchain.pem (failure)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1 renew failure(s), 0 parse failure(s)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): nginx version: nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version): Ubuntu 22.04.5 LTS

My hosting provider, if applicable, is: (myself)

I can login to a root shell on my machine (yes or no, or I don't know): Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 5.0.0

---

Generated Nginx config:

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot

rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


    server_name autismkarate.org www.autismkarate.org;
    root /var/www/autismkarate.org/web;
    index index.html index.htm index.php;
    #charset utf-8;

    listen [::]:443 ssl http2; # managed by Certbot
    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/as21805.net-0002/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/as21805.net-0002/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot

location = /.well-known/acme-challenge/(challenge){default_type text/plain;return 200 (data);} # managed by Certbot

location = /.well-known/acme-challenge/(challenge){default_type text/plain;return 200 (data);} # managed by Certbot

}

server {rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot

rewrite ^(/.well-known/acme-challenge/.*) $1 break; # managed by Certbot


    if ($host = www.autismkarate.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


    if ($host = autismkarate.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot


        listen 80;
        listen [::]:80;

        server_name autismkarate.org www.autismkarate.org;
    #return 404; # managed by Certbot
location = /.well-known/acme-challenge/(challenge){default_type text/plain;return 200 (data);} # managed by Certbot

location = /.well-known/acme-challenge/(challenge){default_type text/plain;return 200 (data);} # managed by Certbot

}

[griffin]

Welcome back to the Let's Encrypt Community!

The reported errors indicate that secondary validation has failed, which usual indicates some type of network geoblocking (of IPv6 in this case). The primary validations succeeded.

@mrhamel

During secondary validation: (v6 IP): Invalid response from http://autismkarate.org/.well-known/acme-challenge/(challenge): 404

@mrhamel

During secondary validation: (v6 IP): Invalid response from http://www.autismkarate.org/.well-known/acme-challenge/(challenge): 404

Also (and likely unrelated to the primary problem), I'm not sure why there are permanent, duplicate rewrites in your server blocks from certbot. That seems very odd to me. There are more than a few odd things in there to my eyes.

[mrhamel]

The v6 address is my own (from ARIN) 2602:f9ed:103::19 announced from AS21805, and there are no firewalls in the way.

I also do not believe it's any firewall as for the other domains, it picks up on the hard return 301 (URL), which is another issue I'll deal with separately.

The Nginx config is just perfect timing while certbot was running, to see what it looked like.

[griffin]

Please post the config when certbot is not running, if you would.

[griffin]

I'm still thinking this is related to some type of global reachability problem since the failures aren't seeing the HTTP->HTTPS redirect (because "http://" is in the address in the failures). Exactly what is returning the 404s I am not sure. It could be your nginx server, but that would seem odd given that the primary validations succeeded for the same content. I don't readily have servers configured to stimulate the multi-perspective global validation locations.

[mrhamel]

The Nginx config:

server {
    server_name autismkarate.org www.autismkarate.org;
    root /var/www/autismkarate.org/web;
    index index.html index.htm index.php;
    #charset utf-8;

    listen [::]:443 ssl http2; # managed by Certbot
    listen 443 ssl http2; # managed by Certbot
    ssl_certificate /etc/letsencrypt/live/as21805.net-0002/fullchain.pem; # managed by Certbot
    ssl_certificate_key /etc/letsencrypt/live/as21805.net-0002/privkey.pem; # managed by Certbot
    include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
    ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

server {
    if ($host = www.autismkarate.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

    if ($host = autismkarate.org) {
        return 301 https://$host$request_uri;
    } # managed by Certbot

        listen 80;
        listen [::]:80;

        server_name autismkarate.org www.autismkarate.org;
    #return 404; # managed by Certbot
}

[griffin]

Seems straightforward. One (or more) of LE's global validation points is getting a 404 though.

I'll run a test...

[griffin]

These results don't indicate much IMO, but they aren't great. They mean that your IPv4 and IPv6 responses differ, but they're not as conclusive as LE's global validation process. LE prefers IPv6 over IPv4, but will fallback to IPv4 if IPv6 isn't present, which IPv6 is present in your case.

Test result for autismkarate.org: Warning

Test result for www.autismkarate.org: Warning

[griffin]

Is there some type of caching/CDN present that might be interfering with the responses? Just throwing out ideas here. If you have other domains on the same webserver that succeed, this is possible. Also, but less likely, you could have orphaned nginx processes with a previous config still loaded that happen to (incorrectly) serve the challenge content.

[griffin]

I'm suspecting the orphaned nginx process problem now that I think on it more. It just seems the most likely to me.

Added detail from @MikeMcQ for solution:

@MikeMcQ

Was nginx stopped when you did the certbot renew by any chance?

Because Certbot's --nginx option pretty much requires nginx to already be running. Certbot will actually start nginx but does not with systemd which can cause symptoms like the grief you saw.

[mrhamel]

That was it! I'm seriously scratching my head to figure out how that happened, because I did a full service nginx restart to be sure that wasn't the case, but checking the process list with ps confirmed it was exactly that.

Thank you!

[MikeMcQ]

@mrhamel

I'm seriously scratching my head to figure out how that happened

Was nginx stopped when you did the certbot renew by any chance?

Because Certbot's --nginx option pretty much requires nginx to already be running. Certbot will actually start nginx but does not with systemd which can cause symptoms like the grief you saw.

[griffin]

Excellent!

[griffin]

Regarding the changes certbot makes, since it's one specific challenge per specific domain name (which is not as evident from the data sanitization you performed in your posts), there will be two locations and two rewrites, so all is well there.