I don’t think I’m understanding the nuances of SSL Cert verification.
My setup
Internet ------>Apache Reverse Proxy-----LAN----->Apache Web Server
I have LE certs installed that make the SSL connection between Internet and Proxy Server, however I’m trying to encrypt the backend connection between the Reverse Proxy and Local Apache Web Server
Within my LE cert – I have it setup kind of a SAN or with multiple domains. The reverse proxy has one domain, and the internal Web Server is known as another domain. My pfSense local Domain Name Server or DNS resolver has the local IP address of the Apache Web Server mapped to the domain name of the Web Server. Both the reverse proxy and internal web server have copies of the LE certificates that were distributed by other means described on these forums: Automated deployment of key/cert from reverse proxy to internal systems
I can get the backend encrypted with the following Apache directives run on the proxy server:
SSLProxyEngine On
SSLProxyVerify none #SSLProxyVerifyDepth 5
SSLProxyCheckPeerCN Off
SSLProxyCheckPeerName Off
I’m no expert on SSL --> However I think these statements tell the client to encrypt but don’t very authenticity of the server.
When I enable these options (for example):
SSLProxyEngine On
SSLProxyVerify require
SSLProxyVerifyDepth 5
SSLProxyCheckPeerCN Off
SSLProxyCheckPeerName On
the error logs suggest I then need a SSLCACertificate file. I was hoping to use the LE fullchain.pem file for this however this clearly didn’t work.
Ive seen others suggest use of an internal CA and selfsigned certs to get around this problem. I’m not sure how to do this unfortunately.
I just wanted to confirm that I can not do the backend SSL encryption/verification using LE certs.
As long as your backend server is using a FQDN from a real (purchased from a registrar as opposed to something like .local) domain you control, it’s possible to get an LE certificate for it. However, since it is not likely accessible directly from the Internet with that name, you’ll need to use DNS based validation rather than HTTP.
How hard that will be depends on the DNS provider hosting the internet-facing zone for the FQDN and whether you have a client that supports automated TXT record creation against it.
The way I read the OP’s post is that they already have Let’s Encrypt certificates installed on both the proxy and the internal backend - so I’ll respond in that way.
You can absolutely use Let’s Encrypt certificates to protect the internal link. Client certificates (mutual authentication) are a different beast entirely.
A quick test shows that this seems to be the minimum config to get that internal link secured:
<VirtualHost *:443>
# ... Put your usual SSL configuration for presenting SSL to
# the outside world.
# Then we configure the secured reverse proxy
# You will need to ensure that the domain resolves to the internal backend
# host, even if you need to /etc/hosts it
ProxyPass / https://internal-backend.example.com
SSLProxyEngine on
SSLProxyVerify require
SSLProxyVerifyDepth 3
# This path is suitable for Debian/Ubuntu.
SSLProxyCACertificatePath /etc/ssl/certs
# For EL-based distros, install ca-certificates and use:
# SSLProxyCACertificateFile /etc/ssl/certs/ca-bundle.crt
# Disabling OCSP checking here because it was producing "bad nonce" errors
# when verifying the remote certificate. Ideally you would leave it on.
SSLOCSPEnable off
</VirtualHost>
Just for clarification – I have FQDN on the proxy and internal server. Both domain names are specified in on LE cert. I renew the cert on proxy server and then distribute the cert to internal server. The internal server is not accessible to outside other than proxied through proxied server. DNS records are at cloudflare. Certs are renewed using dns_cloudflare module.
Theoretically this should work I think, however its not
Here is my relevant apache config;
SSLEngine on
SSLCertificateFile /usr/local/etc/letsencrypt/live/xxx.com/fullchain.pem
SSLCertificateKeyFile /usr/local/etc/letsencrypt/live/xxx.com/privkey.pem
# Uncomment the following directive when using client certificate authentication
#SSLCACertificateFile /usr/local/etc/letsencrypt/live/xxxx.com/fullchain.pem
SSLProxyCACertificatePath /usr/local/etc/letsencrypt/live/xxxx.com
SSLOCSPEnable off
SSLProtocol all -SSLv3 -TLSv1 -TLSv1.1
SSLCipherSuite ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256
SSLHonorCipherOrder on SSLCompression off
SSLSessionTickets off
# HSTS (mod_headers is required) (15768000 seconds = 6 months)
Header always set Strict-Transport-Security "max-age=15552000; includeSubDomains"
# Encoded slashes need to be allowed
AllowEncodedSlashes NoDecode
# Container uses a unique non-signed certificate
SSLProxyEngine On
SSLProxyVerify require
SSLProxyVerifyDepth 3
SSLProxyCheckPeerCN Off
SSLProxyCheckPeerName On
# keep the host
ProxyPreserveHost On
ProxyPass / https://office.xxx.com
ProxyPassReverse / https://office.xxx.com
Here are log files when I tried to access page:
[Wed Oct 02 16:35:44.206818 2019] [ssl:error] [pid 7754] [remote 10.0.1.162:443] AH02039: Certificate Verification: Error (20): unable to get local issuer certificate
[Wed Oct 02 16:35:44.207085 2019] [proxy:error] [pid 7754] (20014)Internal error (specific information not available): [client 2600:1008:b042:89d3:a1a1:8b24:7cde:21c0:0] AH01084: pass request body failed to 10.0.1.162:443 (office.xxxx.com)
[Wed Oct 02 16:35:44.207106 2019] [proxy:error] [pid 7754] [client 2600:1008:b042:89d3:a1a1:8b24:7cde:21c0:0] AH00898: Error during SSL Handshake with remote server returned by /
[Wed Oct 02 16:35:44.207113 2019] [proxy_http:error] [pid 7754] [client 2600:1008:b042:89d3:a1a1:8b24:7cde:21c0:0] AH01097: pass request body failed to 10.0.1.162:443 (office.xxxx.com) from 162.158.62.160 ()
I’m open to try new things – have to be missing something here
Take note of what values I suggest for SSLProxyCACertificatePath in my example.
I think if you point it to the OS-provided CA certificates directory on your system, that unable to get local issuer certificate will disappear.
The above working depends on you sending a correct chain from office.xxx.com. You should be sending both the leaf and intermediate certificate, which you can check with:
Proxy server running Freebsd, internal apache web server on Linux
The CA Certs for freebsd are a little bit different -- they are all concatenated into on big file on FreeBSD rather than individually separated. File is /usr/local/share/certs/ca-root-nss.crt
I looked up apache docs on SSLProxyCACertificate path:
The files in this directory have to be PEM-encoded and are accessed through hash filenames. So usually you can't just place the Certificate files there: you also have to create symbolic links named hash-value.N . And you should always make sure this directory contains the appropriate symbolic links.
I'm not sure my one file is PEM-encoded and clearly there is no hash filenames in play here.
So I changed the following with virtual host file:
SSLProxyCACertificatePath /usr/local/share/certs
Unfortunately I didn't see any change -- Relevant log files:
[Wed Oct 02 21:09:46.887593 2019] [ssl:error] [pid 31945] [remote 10.0.1.162:443] AH02039: Certificate Verification: Error (20): unable to get local issuer certificate
[Wed Oct 02 21:09:46.887818 2019] [proxy:error] [pid 31945] (20014)Internal error (specific information not available): [client 2600:1008:b042:89d3:4120:ae3d:fba1:6868:0] AH01084: pass request body failed to 10.0.1.162:443 (office.xxx.com)
[Wed Oct 02 21:09:46.887855 2019] [proxy:error] [pid 31945] [client 2600:1008:b042:89d3:4120:ae3d:fba1:6868:0] AH00898: Error during SSL Handshake with remote server returned by /
[Wed Oct 02 21:09:46.887861 2019] [proxy_http:error] [pid 31945] [client 2600:1008:b042:89d3:4120:ae3d:fba1:6868:0] AH01097: pass request body failed to 10.0.1.162:443 (office.xxx.com) from 162.158.63.227 ()
[Wed Oct 02 21:09:47.563267 2019] [proxy:error] [pid 31946] [client 2600:1008:b042:89d3:4120:ae3d:fba1:6868:0] AH00898: DNS lookup failure for: office.xxx.comfavicon.ico returned by /favicon.ico
In terms of the chain file being passed -- its the lets encrypt fullchain.pem file -- I'd post it but its really long and probably the contents aren't useful.
From inside the LAN I can access the internal server directly, and I can access web pages using Chrome/Mozilla so I can verify if I take the proxy out of the equation things tend to work.
Thanks for guiding me through this – hot dog!!!
Things work. Wow!!
I should probably write this up somewhere since there isn’t a lot of information on internet how to do this with these types of certificates. Awesome.
I’m just curious however – just for my knowledge
The proxy server is essentially a MITM. Correct me if I’m wrong but doesn’t the proxy server encrypt in two places – one from client to proxy server, and the other from proxy to internal server. Isn’t this essentially a “classic MITM attack” where header could be injected during the communication process?
And yeah, it seems to me that any reverse proxy is by definition a MITM. What makes it not an attack is that the reverse proxy is a trusted peer - you are pointing your domain at it, after all.
Most HTTP CDNs also fit that description.
If you wanted true end-to-end encryption, you could do something like pre-read the SNI part of the TLS ClientHello packet on the proxy, extract domain name is being asked for, and then just route the raw TCP byte stream to the intended destination without decrypting it. Web servers like haproxy and nginx can do this quite trivially (e.g. https://nginx.org/en/docs/stream/ngx_stream_ssl_preread_module.html), but Apache httpd cannot, because it lacks the architecture to deal with non-HTTP protocols.
Hey my enthusiasm may have been too soon. (And perhaps this question isn’t relative to this discussion.
I dont think I have my reverse proxies setup appropriately. The web client can appropriately display the index.html file located within the DocumentRoot from office.xxx.com. (This is why I was so excited previously). Any other subdirectory such as https://office.xxx.com/data or https://office.xxx.com/lool —> I get a 500 error. I’m not sure where to actually start with this one.
Depending on who actually produced the 500 (the backend server or the reverse proxy), the reason should be reported in one or both of the Apache error logs.
I can confirm these statements actually work as intended if you don’t connect through the first proxy.
If connecting through first proxy however (double proxy scenario) – things dont work however and I get a error similar to this:
No protocol handler was valid for the URL /lool/https:%2F%2Fnextcloud.domain.com%2Findex.php%2Fapps%2Frichdocuments%2Fwopi%2Ffiles%2F707_ocny42d5quk3?access_token=DkhSTgh76XDb9xCoGtqQZ9Bx7Z9lOvVW&access_token_ttl=0&reuse_cookies=__cfduid%3Dd077ca775440b9eb4eb79df3926adc9101570283873&permission=edit/ws (scheme 'ws'). If you are using a DSO version of mod_proxy, make sure the proxy submodules are included in the configuration using LoadModule.
I’m guessing the ws information is stripped somehow in the process of all these reverse proxies. (mod_wstunnel is activated on both proxies, so this doesn’t seem to be the problem).