(got \"\") 403 error directory seems accessible with curl and files are being created

Help
1

TL:DR, this was a caching problem - Apache's internal data cache was jamming up the ephemeral test file creation/removal process that dehydrated/Lets encrypt use to validate the domain, serving a null file which was cached before the the validation text was inserted. Blocking caching in the permissions stanza for the .well-known directory in httpd.conf solved this issue. The successful incantation is in the responses below. It could be inserted in httpd.conf, httpd-vhosts.conf, or a .htaccess file, depending on how one's servers are configured.

I migrated my server from one box to another, updating some of the config files, so there's definitely a risk of misconfiguration, [spoiler: it was] but I've checked against other servers already migrated and collecting certs correctly. This one seems to be having a problem validating despite carefully "melding" the .conf files as best I can. It does not seem to be a problem with :80 access from outside (that works) nor with dehydrated having write permissions as the authorization files are being created and deleted with the correct ownership, group, and permissions. I'm stumped...

My domain is: cube.blackrosetech.com

I ran this command: # su -m _letsencrypt -c 'dehydrated -c''

It produced this output:

# INFO: Using main config file /usr/local/etc/dehydrated/config
Processing cube.blackrosetech.com with alternative names: webmail.blackrosetech.com red.blackrosetech.com git.blackrosetech.com cameras.blackrosetech.com
 + Checking domain name(s) of existing cert... unchanged.
 + Checking expire date of existing cert...
 + Valid till Jul  5 10:16:25 2024 GMT (Less than 30 days). Renewing!
 + Signing domains...
 + Generating private key...
 + Generating signing request...
 + Requesting new certificate order from CA...
 + Received 5 authorizations URLs from the CA
 + Handling authorization for cube.blackrosetech.com
 + Handling authorization for git.blackrosetech.com
 + Handling authorization for red.blackrosetech.com
 + Handling authorization for webmail.blackrosetech.com
 + Handling authorization for cameras.blackrosetech.com
 + 5 pending challenge(s)
 + Deploying challenge tokens...
 + Responding to challenge for cube.blackrosetech.com authorization...
 + Cleaning challenge tokens...
 + Challenge validation has failed :(
ERROR: Challenge is invalid! (returned: invalid) (result: ["type"]      "http-01"
["url"] "https://acme-v02.api.letsencrypt.org/acme/chall-v3/404149904996/uC9XdA"
["status"]      "invalid"
["validated"]   "2024-09-16T00:07:30Z"
["error","type"]        "urn:ietf:params:acme:error:unauthorized"
["error","detail"]      "During secondary validation: The key authorization file from the server did not match this challenge. Expected \"--30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts.soCaRvo-o7b26zonEQR2-fzRoG0fVrYbUoYpR1P6abQ\" (got \"\")"
["error","status"]      403
["error"]       {"type":"urn:ietf:params:acme:error:unauthorized","detail":"During secondary validation: The key authorization file from the server did not match this challenge. Expected \"--30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts.soCaRvo-o7b26zonEQR2-fzRoG0fVrYbUoYpR1P6abQ\" (got \"\")","status":403}
["token"]       "--30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts"
["validationRecord",0,"url"]    "http://cube.blackrosetech.com/.well-known/acme-challenge/--30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts"
["validationRecord",0,"hostname"]       "cube.blackrosetech.com"
["validationRecord",0,"port"]   "80"
["validationRecord",0,"addressesResolved",0]    "23.114.97.246"
["validationRecord",0,"addressesResolved"]      ["23.114.97.246"]
["validationRecord",0,"addressUsed"]    "23.114.97.246"
["validationRecord",0]  {"url":"http://cube.blackrosetech.com/.well-known/acme-challenge/--30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts","hostname":"cube.blackrosetech.com","port":"80","addressesResolved":["23.114.97.246"],"addressUsed":"23.114.97.246"}
["validationRecord"]    [{"url":"http://cube.blackrosetech.com/.well-known/acme-challenge/--30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts","hostname":"cube.blackrosetech.com","port":"80","addressesResolved":["23.114.97.246"],"addressUsed":"23.114.97.246"}])

My web server is (include version): apache24-2.4.62

The operating system my web server runs on is (include version): FreeBSD 14.1-RELEASE-p3

My hosting provider, if applicable, is: self

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

# INFO: Using main config file /usr/local/etc/dehydrated/config
# INFO: Running /usr/local/bin/dehydrated as _letsencrypt/_letsencrypt
# INFO: Using main config file /usr/local/etc/dehydrated/config
Dehydrated by Lukas Schauer
https://dehydrated.io

Dehydrated version: 0.7.2
GIT-Revision: unknown

OS: FreeBSD 14.1-RELEASE-p3
Used software:
 bash: 5.2.32(0)-release
 curl: 8.9.1
 awk, sed, mktemp, grep, diff: BSD base system versions
 openssl: OpenSSL 3.2.3 3 Sep 2024 (Library: OpenSSL 3.2.3 3 Sep 2024)

if I create an index.php file in /usr/local/www/.well-known/acme-challenge/index.htm, with the text "test" I can curl it from outside the LAN (e.g. from a machine not VPN'd in) with the same permissions/ownership as the challenge files.

# ls -lah
-rw-r--r--  1 _letsencrypt www    5B Sep 15 17:27 index.htm

$ curl  http://cube.blackrosetech.com/.well-known/acme-challenge/
test

This made me suspiecious that the challenge files weren't being created, and as they're ephemeral, curling them as they appear from a remote host isn't trivial, but I did run the following which shows the files were created successfully.

# /usr/local/www/.well-known/acme-challenge # fswatch -m kqueue_monitor . |  xargs -n 1 -I {} ls -lah {}
total 38
drwxrwxr-x  2 _letsencrypt www    2B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
total 39
-rw-------  1 _letsencrypt www   87B Sep 15 17:07 --30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts
drwxrwxr-x  2 _letsencrypt www    3B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
total 39
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 --30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts
drwxrwxr-x  2 _letsencrypt www    4B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-------  1 _letsencrypt www   87B Sep 15 17:07 TMDgK--NQ5Aa3oMXuNo4Rp8cZtxtCgp1XwQ66vcLKh4
total 40
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 --30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts
drwxrwxr-x  2 _letsencrypt www    5B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-------  1 _letsencrypt www   87B Sep 15 17:07 hUIN_2LqifS2sKBhAFYk0ixotx-RxFTtGovU8bBOJQg
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 TMDgK--NQ5Aa3oMXuNo4Rp8cZtxtCgp1XwQ66vcLKh4
total 40
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 --30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts
drwxrwxr-x  2 _letsencrypt www    6B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 hUIN_2LqifS2sKBhAFYk0ixotx-RxFTtGovU8bBOJQg
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 jguvPlz9gBkIwRCwOfucjw4oqweglL-B4m5q6VO7OA4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 TMDgK--NQ5Aa3oMXuNo4Rp8cZtxtCgp1XwQ66vcLKh4
total 41
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 --30dcmwgxO0LTDtwMV96Oe5DcrlC-laI1A2q8l8Tts
drwxrwxr-x  2 _letsencrypt www    7B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 hUIN_2LqifS2sKBhAFYk0ixotx-RxFTtGovU8bBOJQg
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 jguvPlz9gBkIwRCwOfucjw4oqweglL-B4m5q6VO7OA4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 TMDgK--NQ5Aa3oMXuNo4Rp8cZtxtCgp1XwQ66vcLKh4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 vD4aQ9F61bMTTJSYWf9IpecOj3UAbq-q29n8dt4WsD0
total 76
drwxrwxr-x  2 _letsencrypt www    6B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 hUIN_2LqifS2sKBhAFYk0ixotx-RxFTtGovU8bBOJQg
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 jguvPlz9gBkIwRCwOfucjw4oqweglL-B4m5q6VO7OA4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 TMDgK--NQ5Aa3oMXuNo4Rp8cZtxtCgp1XwQ66vcLKh4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 vD4aQ9F61bMTTJSYWf9IpecOj3UAbq-q29n8dt4WsD0
total 67
drwxrwxr-x  2 _letsencrypt www    5B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 hUIN_2LqifS2sKBhAFYk0ixotx-RxFTtGovU8bBOJQg
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 jguvPlz9gBkIwRCwOfucjw4oqweglL-B4m5q6VO7OA4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 vD4aQ9F61bMTTJSYWf9IpecOj3UAbq-q29n8dt4WsD0
total 57
drwxrwxr-x  2 _letsencrypt www    4B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 jguvPlz9gBkIwRCwOfucjw4oqweglL-B4m5q6VO7OA4
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 vD4aQ9F61bMTTJSYWf9IpecOj3UAbq-q29n8dt4WsD0
total 48
drwxrwxr-x  2 _letsencrypt www    3B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
-rw-r--r--  1 _letsencrypt www   87B Sep 15 17:07 vD4aQ9F61bMTTJSYWf9IpecOj3UAbq-q29n8dt4WsD0
total 38
drwxrwxr-x  2 _letsencrypt www    2B Sep 15 17:07 .
drwxrwxr-x  3 _letsencrypt www    3B Dec  7  2017 ..
2

During secondary validation:

do you have any geoblocking firewall or anycast seup?

2 Likes
3

An HTTP challenge will likely be 5 (unlikely but could be 4) HTTP requests to your server for the same URI. Do you know what those other files that show in that folder are?

The problem reported is that after the first HTTP request that worked the LE Server then got returned a mostly empty reply instead of the challenge data it expected.

What does your Apache access log look like for an actual cert HTTP challenge? How many do you see with the same URI and what were the HTTP response codes (all "200"?).

Interesting, if I try your test file multiple times there is some sort of caching affecting the replies. Note the second is a HIT. Could this be affecting the HTTP challenge somehow?

curl -i http://cube.blackrosetech.com/.well-known/acme-challenge/index.htm
HTTP/1.1 200 OK
Date: Mon, 16 Sep 2024 02:28:56 GMT
Server: Apache
Last-Modified: Mon, 16 Sep 2024 00:27:54 GMT
...
X-Cache: MISS from cube.blackrosetech.com
test

curl -i http://cube.blackrosetech.com/.well-known/acme-challenge/index.htm
HTTP/1.1 200 OK
Date: Mon, 16 Sep 2024 02:29:10 GMT
Server: Apache
Last-Modified: Mon, 16 Sep 2024 00:27:54 GMT
...
Age: 14
X-Cache: HIT from cube.blackrosetech.com
test
2 Likes
4

No geoblocking, there is a firewall buy the same extIP->NAT location was working for years previous. Further, pfSense dynamic blocking is WAN-wide or destination type-wide, meaning all the instances with 1:1 nat IPs serving HTTP would get the same rules and any dynamic rules are full WAN - that is if the letsencrypt challenge server ended up on the emerging threats IP list somehow, say, none of the instances could update and I just successfully forced an update on another jail (on the same host).

I moved the instance to new hardware and restarted the FreeBSD jail on the new server at the old IP. Is there some magic number or other ID that ties a letsencrypt account an instance - it is possible there's a file I didn't move if it is outside the usual folders on /etc, /usr/local/etc and /var/dehydrated which is blocking the move.

I haven't tried simply deleting everything dehydrated and starting over, but as I was successful in previous migrations, that shouldn't be necessary.

5

No, there is no connection to the underlying hardware that previously setup the ACME account and requested a cert.

2 Likes
6

YAH, that could be... Lemmie except the directory from caching, the ETag should disappear in the curl get... I'll test and report. I didn't try the -i directive... Thanks!

2 Likes
7

et. walla!!!

I modified httpd.conf well-known directory directive stanza to include a few "don't cache pls" type directives found on the interwebs, more better, probably redundant, but it looks like this now:

<Directory "/usr/local/www/.well-known/">
    Options None
    AllowOverride None
    Require all granted
    Header add Content-Type text/plain
    FileETag None
    Header unset ETag
    Header set Cache-Control "max-age=0, no-cache, no-store, must-revalidate"
    Header set Pragma "no-cache"
</Directory>

Tested it with curl from an off-lan machine and sure 'nough, no caching

$ curl -i http://cube.blackrosetech.com/.well-known/acme-challenge/
HTTP/1.1 200 OK
Date: Mon, 16 Sep 2024 13:17:06 GMT
Server: Apache
Upgrade: h2,h2c
Connection: Upgrade
Last-Modified: Mon, 16 Sep 2024 00:27:54 GMT
Accept-Ranges: bytes
Content-Length: 5
Vary: Accept-Encoding,User-Agent
Content-Type: text/html
Cache-Control: max-age=0, no-cache, no-store, must-revalidate
Pragma: no-cache
X-Cache: MISS from cube.blackrosetech.com

reran # su -m _letsencrypt -c 'dehydrated -c'

And the magic happens:

 + 5 pending challenge(s)
 + Deploying challenge tokens...
 + Responding to challenge for cube.blackrosetech.com authorization...
 + Challenge is valid!
 + Responding to challenge for git.blackrosetech.com authorization...
 + Challenge is valid!
 + Responding to challenge for red.blackrosetech.com authorization...
 + Challenge is valid!
 + Responding to challenge for webmail.blackrosetech.com authorization...
 + Challenge is valid!
 + Responding to challenge for cameras.blackrosetech.com authorization...
 + Challenge is valid!
 + Cleaning challenge tokens...
 + Requesting certificate...
Warning: Will read cert request from stdin since no -in option is given
 + Checking certificate...
 + Done!
 + Creating fullchain.pem...
 + Done!

Thank you, excellent catch with the caching. I'll be using the no cache parameters from now on just in case.

2 Likes
8

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.