I am using pfsense + acme + stunnel to secury route traffic through the firewall to specific ports. My certificate recently expired and a new certificate was issued with the ACME plugin using Let's encrypt. The new ceritificate is using R11 intermediate the old was using R3. Synce the update to R11 stunnel does not route traffic, but fails with an error:
Jun 26 08:47:38 stunnel 8581 LOG3[293]: Error resolving r10.o.lencr.org: Address family for nodename not supported (EAI_ADDRFAMILY)
Jun 26 08:47:38 stunnel 8581 LOG3[293]: OCSP: Failed to resolve the OCSP responder address
Jun 26 08:47:38 stunnel 8581 LOG3[293]: SSL_accept: /var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/crypto/openssl/ssl/record/rec_layer_s3.c:304: error:0A000126:SSL routines::unexpected eof while reading
Jun 26 08:47:38 stunnel 8581 LOG5[293]: Connection reset/closed: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
I have the simillar installation with a still certificate using R3 intermediate and it works fine. On the Netgate support I was told that this is not a bug, but an issue with my installation. However starting from scratch with a clean pfsense installation I ended up with the same result.
I am using pfsense 2.7.2 with latest ACME and stunnel packages.
This is the log of the issued certificate from the ACME plugin:
schnee-ver.no-ip.org-pfsense
Renewing certificate
account: schnee-ver.no-ip.org
server: letsencrypt-production-2
/usr/local/pkg/acme/acme.sh --issue --domain 'schnee-ver.no-ip.org' --standalone --listen-v4 --httpport '8989' --home '/tmp/acme/schnee-ver.no-ip.org-pfsense/' --accountconf '/tmp/acme/schnee-ver.no-ip.org-pfsense/accountconf.conf' --force --always-force-new-domain-key --reloadCmd '/tmp/acme/schnee-ver.no-ip.org-pfsense/reloadcmd.sh' --log-level 3 --log '/tmp/acme/schnee-ver.no-ip.org-pfsense/acme_issuecert.log'
Array
(
[path] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
[PATH] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
[SSL_CERT_DIR] => /etc/ssl/certs/
[port] => 9é989
[ipv6] =>
)
[Wed Jun 26 08:36:26 CEST 2024] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Wed Jun 26 08:36:26 CEST 2024] Standalone mode.
[Wed Jun 26 08:36:26 CEST 2024] Using pre generated key: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key.next
[Wed Jun 26 08:36:26 CEST 2024] Generate next pre-generate key.
[Wed Jun 26 08:36:29 CEST 2024] Single domain='schnee-ver.no-ip.org'
[Wed Jun 26 08:36:34 CEST 2024] Getting webroot for domain='schnee-ver.no-ip.org'
[Wed Jun 26 08:36:34 CEST 2024] Verifying: schnee-ver.no-ip.org
[Wed Jun 26 08:36:34 CEST 2024] Standalone mode server
[Wed Jun 26 08:36:36 CEST 2024] Pending, The CA is processing your order, please just wait. (1/30)
[Wed Jun 26 08:36:39 CEST 2024] Pending, The CA is processing your order, please just wait. (2/30)
[Wed Jun 26 08:36:42 CEST 2024] Success
[Wed Jun 26 08:36:42 CEST 2024] Verify finished, start to sign.
[Wed Jun 26 08:36:42 CEST 2024] Lets finalize the order.
[Wed Jun 26 08:36:42 CEST 2024] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/1801934517/281806720357'
[Wed Jun 26 08:36:43 CEST 2024] Downloading cert.
[Wed Jun 26 08:36:43 CEST 2024] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/0334347e4c62dbd1ded7f033faef487249c0'
[Wed Jun 26 08:36:44 CEST 2024] Cert success.
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
[Wed Jun 26 08:36:44 CEST 2024] Your cert is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.cerschnee-ver.no-ip.org-pfsense
Renewing certificate
account: schnee-ver.no-ip.org
server: letsencrypt-production-2
/usr/local/pkg/acme/acme.sh --issue --domain 'schnee-ver.no-ip.org' --standalone --listen-v4 --httpport '8989' --home '/tmp/acme/schnee-ver.no-ip.org-pfsense/' --accountconf '/tmp/acme/schnee-ver.no-ip.org-pfsense/accountconf.conf' --force --always-force-new-domain-key --reloadCmd '/tmp/acme/schnee-ver.no-ip.org-pfsense/reloadcmd.sh' --log-level 3 --log '/tmp/acme/schnee-ver.no-ip.org-pfsense/acme_issuecert.log'
Array
(
[path] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
[PATH] => /etc:/bin:/sbin:/usr/bin:/usr/sbin:/usr/local/bin/
[SSL_CERT_DIR] => /etc/ssl/certs/
[port] => 9é989
[ipv6] =>
)
[Wed Jun 26 08:36:26 CEST 2024] Using CA: https://acme-v02.api.letsencrypt.org/directory
[Wed Jun 26 08:36:26 CEST 2024] Standalone mode.
[Wed Jun 26 08:36:26 CEST 2024] Using pre generated key: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key.next
[Wed Jun 26 08:36:26 CEST 2024] Generate next pre-generate key.
[Wed Jun 26 08:36:29 CEST 2024] Single domain='schnee-ver.no-ip.org'
[Wed Jun 26 08:36:34 CEST 2024] Getting webroot for domain='schnee-ver.no-ip.org'
[Wed Jun 26 08:36:34 CEST 2024] Verifying: schnee-ver.no-ip.org
[Wed Jun 26 08:36:34 CEST 2024] Standalone mode server
[Wed Jun 26 08:36:36 CEST 2024] Pending, The CA is processing your order, please just wait. (1/30)
[Wed Jun 26 08:36:39 CEST 2024] Pending, The CA is processing your order, please just wait. (2/30)
[Wed Jun 26 08:36:42 CEST 2024] Success
[Wed Jun 26 08:36:42 CEST 2024] Verify finished, start to sign.
[Wed Jun 26 08:36:42 CEST 2024] Lets finalize the order.
[Wed Jun 26 08:36:42 CEST 2024] Le_OrderFinalize='https://acme-v02.api.letsencrypt.org/acme/finalize/1801934517/281806720357'
[Wed Jun 26 08:36:43 CEST 2024] Downloading cert.
[Wed Jun 26 08:36:43 CEST 2024] Le_LinkCert='https://acme-v02.api.letsencrypt.org/acme/cert/0334347e4c62dbd1ded7f033faef487249c0'
[Wed Jun 26 08:36:44 CEST 2024] Cert success.
-----BEGIN CERTIFICATE-----
-----END CERTIFICATE-----
[Wed Jun 26 08:36:44 CEST 2024] Your cert is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.cer
[Wed Jun 26 08:36:44 CEST 2024] Your cert key is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key
[Wed Jun 26 08:36:44 CEST 2024] The intermediate CA cert is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/ca.cer
[Wed Jun 26 08:36:44 CEST 2024] And the full chain certs is there: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/fullchain.cer
[Wed Jun 289896 08:36:44 CEST 2024] Your pre-generated next key for future cert key change is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key.next
[Wed Jun 26 08:36:45 CEST 2024] Run reload cmd: /tmp/acme/schnee-ver.no-ip.org-pfsense/reloadcmd.sh
IMPORT CERT schnee-ver.no-ip.org-pfsense, /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key, /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.cer
update cert![Wed Jun 26 08:36:45 CEST 2024] Reload success
[Wed Jun 26 08:36:44 CEST 2024] Your cert key is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key
[Wed Jun 26 08:36:44 CEST 2024] The intermediate CA cert is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/ca.cer
[Wed Jun 26 08:36:44 CEST 2024] And the full chain certs is there: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/fullchain.cer
[Wed Jun 289896 08:36:44 CEST 2024] Your pre-generated next key for future cert key change is in: /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key.next
[Wed Jun 26 08:36:45 CEST 2024] Run reload cmd: /tmp/acme/schnee-ver.no-ip.org-pfsense/reloadcmd.sh
IMPORT CERT schnee-ver.no-ip.org-pfsense, /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.key, /tmp/acme/schnee-ver.no-ip.org-pfsense/schnee-ver.no-ip.org/schnee-ver.no-ip.org.cer
update cert![Wed Jun 26 08:36:45 CEST 2024] Reload success
Thank you in advance for any suggestions what to do next.
The certificate works for the admin panel in pfsense, but does not work with stunnel. The only change happened originally is that the cert was renewed, before that the config worked fine for 2+ years.
I still see the error Error resolving r10.o.lencr.org: Address family for nodename not supported (EAI_ADDRFAMILY) whenever I try to connect via stunnel
You have gotten certs issued by the R10 and R11 intermediates lately. This is now completely normal.
HTTPS requests on port 443 to your domain work fine (with an R10 intermediate right now). I don't know enough about stunnel to help. You may need to wait for someone to reply to your post on their forum. Or perhaps someone else here.
Do you have any trouble resolving r10.o.lencr.org (or r11.o.lencr.org) in general on that system (Using dig, nslookup, host, or whatever you've got), or get different kinds of answers than doing so with r3.o.lencr.org? Do you have both IPv4 and IPv6 access? (I would expect it to work with either one available, but the error message I think is indicating it's having a problem resolving for one or the other somehow.)
I remember people having issue accessing the R3 certificates OCSP resolver in the past. Some security device/software blacklisted the domain name/URL because it was heavily accessed from multiple locations, so they considered it being malicious.
This is one of those situations where it would be worth being a customer of pfsense so you can escalate the issue to their technical support.
Their software is broken but you need to find someone in their organizations that cares about fixing it for you, and that's probably harder if you're not a customer.
Installing stunnel 5.68 on a Debian 12.5 the same certificate (pem file compied from pfsense) works wihtout issues.
Just curious what system is failing. Maybe one of us will recognize something then.
Mostly though I wanted to suggest trying a different Certificate Authority. Maybe BuyPass or even ZeroSSL. Your ACME client acme.sh supports both and probably others (see their docs)
So I tried ZeroSSL and the problem is the same what I have with Let's encrypt:
Jun 27 20:45:35 stunnel 33074 LOG5[4]: Service [SerHomeCTRL1] accepted connection from 192.168.3.243:56492
Jun 27 20:45:35 stunnel 33074 LOG5[4]: OCSP: Connecting the AIA responder "http://zerossl.ocsp.sectigo.com"
Jun 27 20:46:37 stunnel 33074 LOG3[0]: Error resolving "r10.o.lencr.org": Address family for nodename not supported (EAI_ADDRFAMILY)
Jun 27 20:46:37 stunnel 33074 LOG3[0]: OCSP: Failed to resolve the OCSP responder address
Jun 27 20:46:37 stunnel 33074 LOG3[0]: SSL_accept: /var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/crypto/openssl/ssl/record/rec_layer_s3.c:304: error:0A000126:SSL routines::unexpected eof while reading
Jun 27 20:46:37 stunnel 33074 LOG5[0]: Connection reset/closed: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
Jun 27 20:46:41 stunnel 33074 LOG3[1]: Error resolving "zerossl.ocsp.sectigo.com": Address family for nodename not supported (EAI_ADDRFAMILY)
Jun 27 20:46:41 stunnel 33074 LOG3[1]: OCSP: Failed to resolve the OCSP responder address
Jun 27 20:46:41 stunnel 33074 LOG3[1]: SSL_accept: /var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/crypto/openssl/ssl/record/rec_layer_s3.c:304: error:0A000126:SSL routines::unexpected eof while reading
Jun 27 20:46:41 stunnel 33074 LOG5[1]: Connection reset/closed: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
That is bizarre. The r10 OCSP URL has no relation to ZeroSSL.
I cannot even imagine ... except that maybe something from your LE cert got placed in the stunnel config somewhere and you did not replace it with the ZeroSSL cert fully. I don't know.
Could you be blocking all HTTP requests maybe in stunnel? OCSP queries are done with that but that error message is not helpful.
It looks more like blocking DNS queries, or otherwise whatever DNS resolution that it's trying to do is failing.
I'm guessing that it's broken for any "new" hostname, but the r3 hostname is still in some cache or allowlist or the like somewhere. (To be clear, I'm speculating wildly here.)
Yes, that's a better guess Maybe there is something like a hosts file that is the only way their stunnel resolves name. And that it only has the r3 name in it?
My use of stunnel is limited to Debian and DoT. I stopped using stunnel once BIND added native DoT in an update (Bookworm, probably). I never used it with an LE cert, only a Sectigo wildcard. The only complication I ran into was IPv6 binding always failed at boot resulting in the daemon needing to be manualy started after every reboot.
None of that probably helps you, but it's all I can offer.
So it is still not working for me. I will have to do some more investigation when I have some time. I think the most important thing I have to figure out is the bug is in stunnel (upstream from pfsense) or in pfsense This is not going it be easy ride....
Finally had some time and do some more investigation. At the end it seems to be an installation issue, but I have no idea how to fix it. I tested the whole thing on my other pfsense box, and there a new certificate with R10 works fine:
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: Service [XXXXX] accepted connection from zzzzz:35076
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: OCSP: Connecting the AIA responder "http://r10.o.lencr.org"
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: s_connect: connected 104.76.210.70:80
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: OCSP: Certificate accepted
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: s_connect: connected yyyyy:8080
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: Service [XXXXX] connected remote server from 192.168.10.1:58518
Jul 18 20:29:52 router2 stunnel[68978]: LOG3[0]: transfer: s_poll_wait: TIMEOUTclose exceeded: closing
Jul 18 20:29:52 router2 stunnel[68978]: LOG5[0]: Connection closed: 3241 byte(s) sent to TLS, 802 byte(s) sent to socket
On the broken router after enabling stunnel debug level 6:
Jul 19 00:53:08 router1 stunnel[2933]: LOG5[6]: Service [XXXX] accepted connection from xxxxxx:46415
Jul 19 00:53:08 router1 stunnel[2933]: LOG6[6]: Peer certificate not required
Jul 19 00:53:08 router1 stunnel[2933]: LOG6[6]: OCSP: The root CA certificate was not found
Jul 19 00:53:08 router1 stunnel[2933]: LOG5[6]: OCSP: Connecting the AIA responder "http://r10.o.lencr.org"
Jul 19 00:56:05 router1 stunnel[2933]: LOG3[6]: Error resolving "r10.o.lencr.org": Address family for nodename not supported (EAI_ADDRFAMILY)
Jul 19 00:56:05 router1 stunnel[2933]: LOG3[6]: OCSP: Failed to resolve the OCSP responder address
Jul 19 00:56:05 router1 stunnel[2933]: LOG6[6]: OCSP: No OCSP stapling response to send
Jul 19 00:56:05 router1 stunnel[2933]: LOG3[6]: SSL_accept: /var/jenkins/workspace/pfSense-CE-snapshots-2_7_2-main/sources/FreeBSD-src-RELENG_2_7_2/crypto/openssl/ssl/record/rec_layer_s3.c:304: error:0A000126:SSL routines::unexpected eof while reading
Jul 19 00:56:05 router1 stunnel[2933]: LOG5[6]: Connection reset/closed: 0 byte(s) sent to TLS, 0 byte(s) sent to socket
The service logs a OCSP: The root CA certificate was not found message, and I am not sure why,
As the certificate is working for the web interface of the pfsense router, and under System>Certificate>Authorities I can see R10 on both routers I have no idea where to look for a fix.
:
Maybe there is something like a hosts file that is the only way their stunnel resolves name. And that it only has the r3 name in it?
This is not going it be easy ride....