Mixed-case challenge filenames + lowercase-only requests from letsencrypt = 404, I can't issue any certificates

Help
1

I'm encountering this problem for the first time after many years of successful use. On a new server built out like all the rest, I'm getting 404's on all of my "secondary validation" challenge checks.

I've traced this to challenge files being mixed case, while the URLs letsencrypt is requesting are all lower case, but otherwise the same.

If I didn't know any better, I would say the problem is simple... someone shipped a change that only works on a case-insensitive system, like a Mac or Windows box, and it's busted for Linux!

Except, certbot 2.11.0 came out on June 5th! And there is nothing newer. So... surely I wouldn't be anywhere near the only one... right????

Thanks for your help, I'm truly mystified and behind the 8-ball here.

My domain is:

I ran this command:

sudo podman run -it --rm -v "/var/www/certbot:/var/www/certbot" -v "/etc/letsencrypt:/etc/letsencrypt" -v "/var/lib/letsencrypt:/var/lib/letsencrypt" certbot/certbot certonly -n --agree-tos --email ops@apostrophecms.com --domains feedkidsma.org --webroot --webroot-path /var/www/certbot/ -vvv

It produced this output:

...
Creating root challenges validation dir at /var/www/certbot/.well-known/acme-challenge
Attempting to save validation to /var/www/certbot/.well-known/acme-challenge/ZRWGCMvYTTLeKUD3oN5SIsVEm5YzqDOEozbdn-7eOgU

(NOTE: MIXED CASE FILENAME)

Certbot failed to authenticate some domains (authenticator: webroot). The Certificate Authority reported these problems:
Domain: www.feedkidsma.org
Type: unauthorized
Detail: 35.168.251.141: Invalid response from https://feedkidsma.org/.well-known/acme-challenge/zrwgcmvyttlekud3on5sisvem5yzqdoeozbdn-7eogu: 404

NOTE: ALL LOWERCASE URL, otherwise the same

My web server is (include version):

nginx 1.18.0, but note I'm using --webroot directly with the docker image here, so probably not an nginx issue per se.

I manually created a test file in /var/www/certbot and was able to access it at the expected URL using this location block:

location /.well-known/acme-challenge/ {
  root /var/www/certbot;
}

The operating system my web server runs on is (include version):

ubuntu 22.04

My hosting provider, if applicable, is:

AWS (EC2)

I can login to a root shell on my machine (yes or no, or I don't know):

Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 2.11.0 (via the official container image)

If someone suspects another explanation I have the full verbose output (-vvv), I'm not sure if that contains any secrets I shouldn't divulge off the machine, let me know.

2

I agree the problem is related to the case but there is no way it started at Let's Encrypt that way. Keep in mind LE issues like 5 million certs / day.

Seeing HTTPS in the URL of the error means something redirected the HTTP request from LE to HTTPS. Whatever did that is what needs fixing as it also modified the URI

See the sample test below. Issued from my own test server. The mixed-case URL got changed by something on your end to lower case. See the "Location" value compared to the original URL

curl -i http://feedkidsma.org/TEst404

HTTP/1.1 302 Found
Server: nginx/1.18.0 (Ubuntu)
X-Powered-By: Express
Set-Cookie: project-bread-usms.csrf=cm1feua8m01fj0bo3mj0crt2f; Path=/
Set-Cookie: project-bread-usms.sid=s%3AUwnarOIGZSQmUyM6NgpwSdujeaEWRmuc.Sr2l2YZDHEAC4fSiiCnBWE1rK2ZabiWObFMajW54EYA; Path=/; Expires=Tue, 24 Sep 2024 19:38:32 GMT; HttpOnly
Location: /test404
5 Likes
3

Thanks for giving that a try, but the URL you're accessing is not in the challenges directory so it's outside the location block entirely. You're getting through to Express which is not involved when the URL matches the location block.

So how about this test:

# cd /var/www/certbot
# mkdir -p .well-known/acme-challenge
# vi .well-known/acme-challenge/teST.html

[puts "hi2" in file]

# curl -i http://feedkidsma.org/.well-known/acme-challenge/teST.html

HTTP/1.1 200 OK
Server: nginx/1.18.0 (Ubuntu)
Date: Mon, 23 Sep 2024 19:59:32 GMT
Content-Type: text/html
Content-Length: 4
Last-Modified: Mon, 23 Sep 2024 19:55:56 GMT
Connection: keep-alive
ETag: "66f1c7cc-4"
Accept-Ranges: bytes

hi2

Case-sensitive filename successfully fetched, no redirect took place, Express never involved. Express app has never heard of /var/www/certbot so it's not resolving that on its own.

And if I change the URL to one that doesn't match a file in /var/www/certbot/.well-known/acme-challenge, I get a proper 404 with no hint of a redirect:

boutell@Thomass-MacBook-Pro:~/tmp$ curl -i http://feedkidsma.org/.well-known/acme-challenge/teSTish.html

HTTP/1.1 404 Not Found
Server: nginx/1.18.0 (Ubuntu)
Date: Mon, 23 Sep 2024 20:03:58 GMT
Content-Type: text/html
Content-Length: 162
Connection: keep-alive

<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.18.0 (Ubuntu)</center>
</body>
</html>

So I don't think we've found our culprit yet.

I realize this just... can't be letsencrypt (unless it just started today on the server side, but even then somebody else should be asking by now)

But... WTF?

Thanks!

1 Like
4

The symptom is now very different. Would you try issuing a cert again and report back?

I had previously also run other tests and they confirmed the faulty redirect. I just don't see one now.

Given your "express" system does redirect to all lower case it was probably involved somehow.

2 Likes
5

I think it's DNS. letsencrypt is somehow still getting a very old IP for this server although all records were updated this morning. The TTL now is a somewhat reasonable 1800 (30 minutes), but I can't rule out that it had some crazy value before it was changed. I now suspect that is what happened. I will try to do port forwarding from the old server and see if that helps.

6

You should review your DNS update procedure then. Because Let's Encrypt walks the DNS tree of the authoritative DNS Servers. It does not rely on TTL propagation like DNS resolvers. The authoritative DNS servers must sync world-wide but if this is taking many hours you should switch DNS providers :slight_smile: A good provider's servers sync in less than a minute and the worst I ever saw was 1-2 hours (this is rare).

You can see this IP value using a tool like this: https://unboundtest.com/

If 35.168.251.141 is the wrong IP then I think you might have updated the info in the wrong place. Check with your hosting or DNS provider about that.

5 Likes
7

Thanks. The IP is right now, but I now see from logs that certbot was getting the wrong one long after it should not have been. Network Solutions... cough... was offering up mixed results from their authoritative servers well into the afternoon. (We don't control our clients' choices on that, and to be fair many have had their domains since the dawn times.)

We'll be making more use of port forwarding from legacy servers to try to mitigate this kind of nonsense.

Sorry for the noise.

2 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.