Having some trouble with TrueNAS jail

Hey all, so I'm following a tutorial and I think it's possible I've got some mundane detail of my configuration wrong, but it's hard to tell. I'll fill out the questionaire, but in plain english I'm running a box in my house with TrueNAS, and I'm pointing to a jail running nginx with the hope of using it as a reverse proxy for a nextcloud instance.

My domain is: spaceraser.dev

I ran this command: certbot certonly -v --standalone -d 'spaceraser.dev'

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Requesting a certificate for spaceraser.dev
Performing the following challenges:
http-01 challenge for spaceraser.dev
Waiting for verification...
Challenge failed for domain spaceraser.dev
http-01 challenge for spaceraser.dev

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: spaceraser.dev
Type: unauthorized
Detail: 172.13.9.105: Invalid response from http://spaceraser.dev/.well-known/acme-challenge/o8WcSvcfnoSuXrYvVUZ_SY7M2gNviRsYqSsb9dFyW34: 404

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80. Ensure that the listed domains point to this machine and that it can accept inbound connections from the internet.

Cleaning up challenges
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version):

nginx version: nginx/1.22.0
built with OpenSSL 1.1.1l-freebsd 24 Aug 2021
TLS SNI support enabled
configure arguments: --prefix=/usr/local/etc/nginx --with-cc-opt='-I /usr/local/include' --with-ld-opt='-L /usr/local/lib' --conf-path=/usr/local/etc/nginx/nginx.conf --sbin-path=/usr/local/sbin/nginx --pid-path=/var/run/nginx.pid --error-log-path=/var/log/nginx/error.log --user=www --group=www --with-compat --with-pcre --modules-path=/usr/local/libexec/nginx --with-file-aio --http-client-body-temp-path=/var/tmp/nginx/client_body_temp --http-fastcgi-temp-path=/var/tmp/nginx/fastcgi_temp --http-proxy-temp-path=/var/tmp/nginx/proxy_temp --http-scgi-temp-path=/var/tmp/nginx/scgi_temp --http-uwsgi-temp-path=/var/tmp/nginx/uwsgi_temp --http-log-path=/var/log/nginx/access.log --with-http_v2_module --with-http_addition_module --with-http_auth_request_module --with-http_dav_module --with-http_flv_module --with-http_gunzip_module --with-http_gzip_static_module --with-http_mp4_module --with-http_random_index_module --with-http_realip_module --with-http_secure_link_module --with-http_slice_module --with-http_ssl_module --with-http_stub_status_module --with-http_sub_module --without-mail_imap_module --without-mail_pop3_module --without-mail_smtp_module --with-mail_ssl_module --without-pcre2 --with-stream_ssl_module --with-stream_ssl_preread_module --with-threads --with-mail=dynamic --with-stream=dynamic

The operating system my web server runs on is (include version): FreeBSD 12.3-RELEASE-p9 running in a jail on TrueNAS (TrueNAS-13.0-U3.1)

My hosting provider, if applicable, is: self-hosted

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no, not yet.

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.31.0

Hello @real-spaceraser, welcome to the Let's Encrypt community. :slightly_smiling_face:

Using https://letsdebug.net/ the HTTP-01 Challenge has an error
Let's Debug


Your IPv6 Address needs to have Port 80 open and responding as well.
Best Practice - Keep Port 80 Open

$ curl -I http://spaceraser.dev/.well-known/acme-challenge/o8WcSvcfnoSuXrYvVUZ_SY7M2gNviRsYqSsb9dFyW34
HTTP/1.1 404 Not Found
Server: nginx/1.22.0
Date: Fri, 18 Nov 2022 16:41:24 GMT
Content-Type: text/html
Content-Length: 153
Connection: keep-alive

$ curl http://spaceraser.dev/.well-known/acme-challenge/o8WcSvcfnoSuXrYvVUZ_SY7M2gNviRsYqSsb9dFyW34
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.22.0</center>
</body>
</html>

To be specific for IPv4 Address your server is NOT responding to the HTTP-01 Challenge properly

1 Like

The pertinent information from nslookup

$ nslookup
> spaceraser.dev
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   spaceraser.dev
Address: 172.13.9.105
Name:   spaceraser.dev
Address: 2600:1700:6494:a810::40
>

The long supporting back up slide information from nslookup

$ nslookup
> spaceraser.dev
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   spaceraser.dev
Address: 172.13.9.105
Name:   spaceraser.dev
Address: 2600:1700:6494:a810::40
> set q=aaaa
> spaceraser.dev
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
Name:   spaceraser.dev
Address: 2600:1700:6494:a810::40
> set q=cname
> spaceraser.dev
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
*** Can't find spaceraser.dev: No answer

Authoritative answers can be found from:
spaceraser.dev
        origin = ns1.hover.com
        mail addr = dnsmaster.hover.com
        serial = 1668465489
        refresh = 10800
        retry = 3600
        expire = 604800
        minimum = 900
> set q=soa
> spaceraser.dev
Server:         127.0.0.53
Address:        127.0.0.53#53

Non-authoritative answer:
spaceraser.dev
        origin = ns1.hover.com
        mail addr = dnsmaster.hover.com
        serial = 1668465489
        refresh = 10800
        retry = 3600
        expire = 604800
        minimum = 900

Authoritative answers can be found from:
> server ns1.hover.com
Default server: ns1.hover.com
Address: 216.40.47.26#53
> spaceraser.dev
Server:         ns1.hover.com
Address:        216.40.47.26#53

spaceraser.dev
        origin = ns1.hover.com
        mail addr = dnsmaster.hover.com
        serial = 1668465489
        refresh = 10800
        retry = 3600
        expire = 604800
        minimum = 900
> set q=ns
> spaceraser.dev
Server:         ns1.hover.com
Address:        216.40.47.26#53

spaceraser.dev  nameserver = ns1.hover.com.
spaceraser.dev  nameserver = ns2.hover.com.
> set q=cname
> spaceraser.dev
Server:         ns1.hover.com
Address:        216.40.47.26#53

*** Can't find spaceraser.dev: No answer
> set q=aaaa
> spaceraser.dev
Server:         ns1.hover.com
Address:        216.40.47.26#53

Name:   spaceraser.dev
Address: 2600:1700:6494:a810::40
> set q=a
> spaceraser.dev
Server:         ns1.hover.com
Address:        216.40.47.26#53

Name:   spaceraser.dev
Address: 172.13.9.105
>
1 Like

That ("--standalone") implies there is nothing listening on port 80.

But then there is:

Please show:
netstat -pant | grep -Ei '\:80|nginx'
curl -4 ifconfig.co

2 Likes

netstat: ant: unknown or uninstrumented protocol

then

172.13.9.105

Originally, curl wasn't installed so I ran the command "pkg install curl". I tried the same for netstat, "pkg install netstat", but that came up empty.

Seems to be linuxizm that *BSD doesn't do, I get this on OpenBSD 7.2

$ netstat -pant
netstat: ant: unknown protocol
1 Like

Ok, that makes sense. I have port 80 open and redirecting to the static ip in my router, and confirmed that my isp (ATT, in the US) doesn't block inbound port 80 traffic. Is there something that needs to be configured in the server, or is it a problem with how I set up the DNS records with my domain provider (Hover)?

I'll expose myself as a complete novice, I don't know what the difference is (other than the IPs look like internal IPS) between that first chunk of output and the second, labeled "non authoritative answer". The "non authoritative" info looks like the info I configured with my domain provider.

1 Like

Was just wanting to show the IP Addresses that your domain maps to being both IPv4 and IPv6.

Just means that nslookup was not using the Authoritative Name Servers for getting its answers.

Is fine, you are willing to learn. :slightly_smiling_face:

2 Likes

Yeah, I was getting both outputs from the whatismyip.com website and since I was already in there, I figured I'd add them both.

Is the IPv4/IPv6 kind of a one or the other kind of thing? I figured that having them both in there couldn't be a bad thing.

1 Like

@real-spaceraser Your first post showed a 404 error using an IPv4 address. That points to a particular problem. You then seemed to add an IPv6 address to your DNS but no connection with that IP works. Let's Encrypt will favor IPv6 so that explains the timeout Bruce showed in post #2

You can fix your router to handle IPv6 too. Then we can try to sort out the 404 error.

Or, remove the IPv6 for now and go back to your original problem

3 Likes

Is usually considered a good thing.

1 Like

hmm...
Seems to NOT like netstat -pant
Try showing us:
netstat -?

2 Likes

This might be the safer bet netstat(1)

1 Like

Not sure about the safety aspect - but it was a lot quicker!!!

Try:
netstat -Pan | grep -Ei '\:80|nginx'

2 Likes

no output at all. No error, though!

What about?:
netstat -Pan
netstat -an

2 Likes

On the FreeBSD remote machine I have access to, netstat -Pan seems to work.
(no nginx on that box)

>netstat -Pan | head -5
Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)     Log ID
tcp4       0      0 147.28.0.62.80         18.220.197.210.27821   SYN_RCVD    -
tcp4       0      0 147.28.0.62.80         18.220.197.210.31796   SYN_RCVD    -
tcp4       0      0 147.28.0.62.80         18.220.197.210.40174   SYN_RCVD    -

And it seems weird that syntax of 147.28.0.62.80 (dot) instead of 147.28.0.62:80 (colon)

1 Like

so
\:80
should be
\.80

netstat -Pan | grep -Ei '\.80|nginx'

2 Likes

So I went looking in the router's firewall and saw that, by default, there's a lot of filter rules that sort of seem like they might be automatically dropping some/all IPv6 traffic. Specifically some stuff like

image

As an example of what I see on that said box.
This is likely the interesting line:
tcp4 0 0 *.80 *.* LISTEN -

netstat -Pan | grep -Ei '.80|nginx'
tcp4 160 0 147.28.0.62.80 18.220.197.210.13224 ESTABLISHED -
tcp4 160 0 147.28.0.62.80 18.220.197.210.13883 ESTABLISHED -
tcp4 160 0 147.28.0.62.80 18.220.197.210.12727 ESTABLISHED -
tcp4 160 0 147.28.0.62.80 18.220.197.210.13880 ESTABLISHED -
tcp4 160 0 147.28.0.62.80 18.220.197.210.46710 ESTABLISHED -
tcp4 160 0 147.28.0.62.80 18.220.197.210.65284 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.23332 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.64723 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.50938 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.21552 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.51234 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.22374 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.24823 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.61093 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.38127 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.39663 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.8175 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.62283 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.53982 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.5778 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.32911 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.29217 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.58338 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.43440 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.6587 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.4453 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.17528 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.41362 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.58387 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.19069 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.17817 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.44185 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.31623 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.58437 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.4376 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.2609 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.32063 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.42990 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.1191 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.6686 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.30743 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.14423 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.29816 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.56389 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.53496 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.18894 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.29455 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 18.220.197.210.55205 ESTABLISHED -
tcp4 0 0 147.28.0.62.80 148.153.97.211.43676 FIN_WAIT_2 -
tcp4 0 0 147.28.0.62.80 148.153.97.211.42380 FIN_WAIT_2 -
tcp4 0 0 147.28.0.62.80 217.198.191.208.53862 FIN_WAIT_2 -
tcp4 0 0 147.28.0.62.80 13.89.35.115.57531 FIN_WAIT_2 -
tcp4 0 0 147.28.0.62.80 13.89.35.115.57517 TIME_WAIT -
tcp4 0 0 147.28.0.62.80 37.120.153.229.55406 TIME_WAIT -
tcp4 0 0 147.28.0.62.80 13.89.35.115.57510 TIME_WAIT -
tcp4 0 0 147.28.0.62.80 148.153.97.211.56870 FIN_WAIT_2 -
tcp4 0 0 147.28.0.62.80 148.153.97.211.46514 TIME_WAIT -
tcp4 0 0 147.28.0.62.80 148.153.97.211.37110 TIME_WAIT -
tcp4 0 0 *.80 . LISTEN -

1 Like