Advice for a Newby

Hello,
I am not totally new to encrypting, certificates, and keys. But so far my exposure has been limited. I apologise up front for asking these questions... I know some gurus get frustrated with us novices asking dumb questions.

What am I trying to achieve... I have a number of internal systems accessible via HTTP that are not secured, and when accessed via a web browser the browser complains about the site not being secure. These sites are systems such as Home Assistant, Node Red, Pi-Hole, etc. All of them run on Linux hosts.

Yes, these sites are not internet facing, so why should I bother securing them? With some friends experiencing recent issues around security I believe it is well and truly time I got my act together and secured each server that is accessible via web browser.

I have done loads of reading, but my results have been... ummm, varied. I'd really like to come up with an overall approach that allows me to learn, and secure each system. I am also using a pfsense firewall, which I had hoped may be able to provide me a way to have a private certificate authority, and then use keys on various hosts. That has failed so far.

I have a primary ubuntu workstation that I use to access the systems, but also have a windows machine. So... I am wondering, is there an easy way to secure each of the systems I identified above, and access them via web browser (I use Brave) possibly using the same key to access?

Is there some documentation I could read? I know my way around Linux, so can easily follow instructions. I think I just need a nudge in the right direction, so I can understand the concepts. I am also wondering, is it easier for me to use self signed certificates, or utilise a service such as Let's Encrypt?

Appreciate any advice.

Thanks, Mark

2 Likes

Your question covers quite a range of topics, which are all relatively common questions - they're difficult. Yet I couldn't readily find a nice point-to topic that covers most basics in one go, so I'm writing this slightly longer post instead.

Let us start with a few basics: What do certificates do?

In short, they provide authenticity. When you are browsing the web, you want transferred data to remain secure in-transit - no one listening on the wire should be able to see what you are sending or receiving. On a technical level, this requires three different things: Confidentiality, Authentication, and Integrity. Confidentiality and Integrity are provided by what we usually call "encryption algorithms" or "ciphers". I'm skipping over details here, but essentially we just encrypt the data, so no one can read them. However, in order for the data to be secure, we also need to authenticate it. Essentially, we want to know that the other side of the wire is who we believe it to be. If we establish an encrypted connection to an attacker, there's really no point in encrypting at all. We need assurance that the party we are connected to is what it claims to be. This authentication is provided by certificates. They're essentially proofs of identity, notarized by a trustworthy party - the Certificate Authority (CA). Let's Encrypt is one of such CAs.

A protocol providing all of this is called Transport Layer Security, or TLS in short. It's also known as Secure Socket Layer (SSL). This protocol is what the "S" in HTTPS stands for: HTTPS is HTTP over TLS.

An important misconception is, that certificates also provide trustworthiness - this is not the case. Certificates provide authenticity, but that does not assure that the authenticated party is honest. A sentence you commonly hear is "you could have a private, authenticated conversation with satan". A Domain Validated (DV) certificate assures you that you are really talking to example.com, but it doesn't tell you "the people running example.com are good people" (they probably are, but that's not something the certificate tells you). Most importantly, a certificate does not protect against phishing - phishing websites can have certificates, just like everyone else.


When you want to protect a service against an attacker, you always (even if implicitly) need to define a threat model: What do you want to protect against?

For the internet or web, this answer is usually always the same: Against an active or passive attacker having access to your data packets transferred over the internet. TLS, or HTTPS, provides protection against this type of attacker (if properly configured).

For a local (private) network, this threat model might differ. If you consider your private network to be secure, with no attackers present on it, you actually don't need HTTPS/TLS at all: It gives you absolutely no value, if your threat model does not include such an attacker. Browsers primarily warn about insecure (HTTP) websites, because the browser can't possibly know whether the entire data path taken by the network packet was already secure. It is common security practice to always assume that network packets are transmitted over insecured networks, so browsers warn you about this.

Best practice is to also consider your private network to be insecure, as this could give you an additional layer of defense in depth: Of course you will try to secure your local network, but if your data is still secure even if were not, that's better. This is why it can make sense to also employ TLS/HTTPS in your local network.


Once you've decided that you want TLS on your services, you need a certificate - to provide the authenticity we discussed earlier.

Now, we need to understand that Let's Encrypt is a publicly trusted CA. That means, almost every computer in the world trusts certificates issued by Let's Encrypt. This comes with a high level of responsibility: Let's Encrypt needs to validate that you actually control the identities you want to issue certificates for.

Let's Encrypt performs so called Domain-Validation (DV). It works by checking that you control the domains you are issuing certificates for. For a local network, you might use raw IP addresses, or use private domain names (.local, .lan...). Let's Encrypt cannot issue certificates for such domains, as it is impossible to validate global, unique control over them - anyone can use them in their own networks.

Therefore, if you want a local network secured with Let's Encrypt certificates, you need globally unique domain names for all of your machines you want to secure. In addition to this, you also need to be able to perform a challenge from Let's Encrypt to perform the proof of domain control. There are three different types of challenges, which I don't want to cover in more detail here. A short overview of the process is in the documentation. I have written a brief description of the challenge types in the past. However, for internal networks we need to know that we will need a mechanism that allows Let's Encrypt to connect either to our machines via HTTP(S), or we will need (automated) DNS access to perform domain validation via DNS.

Especially the HTTP challenge - which is normally the simplest to perform - can get difficult in an isolated network. This makes Let's Encrypt often unfavorable for these networks, as its difficult to obtain publicly trusted certificates for them. It is generally possible though, but it varies from setup to setup.


A different option is to obtain a certificate not issued by a publicly trusted CA. Certificates can be generated by absolutely everyone. The value of having them signed by a CA is that they obtain trust through this process: A public CA is trusted, and so everything signed by them is too. If you create your own CA, no one will trust it. For a public website, this is a problem, but for a network where you own all of the devices it might not be. You can configure your own machines to trust your own CA, and then issue certificates however you like, without being bound to restrictions.

This approach is often taken by enterprise networks, but setting up your own CA can be a complex and difficult task - especially if you want it automated, manual methods may be easier here. There are a few systems available that I know of: Plain OpenSSL (manual), XCA (manual), Smallstep (automated, partially commercial), EJBCA (automated/manual, community + commercial), and Dogtag (automated, requires RedHat products). Neither of these are particularly easy I'm afraid. Smallstep is probably the easiest, though some features are either not available at all or only available as part of a commercial closed-source product.

You can also manually configure trust exceptions in your browsers - generate one-time certificates for your services, write down their fingerprints, then set exceptions in the browser - or better, add the certificate as trusted to the browser by hand. This is easy and fast, and also fairly secure, as the browser remembers the exception (but only for this exact certificate).


So overall, what's the recommended solution? It varies for everyone. If you can perform one of the three Let's Encrypt challenges in an automated way, getting certificates from Let's Encrypt can be easy & secure. Remember that the lifetime of Let's Encrypt is only 90 days, so you want to do this automated.

If you can't get that to work, setting up your own CA or using trust exceptions is another perfectly valid way to go.

8 Likes

Another way to possibly simplifying the situation, is to put all your "sites" behind a single proxy.
The proxy can be completely internal and made to handle all the FQDNs.
It could also use an internally signed CA cert OR a globally signed CA cert [same as already mentioned].
The difference here being that you could limit everything CA related to just one single system.
[note: the pfSense could be made to serve as the proxy]

3 Likes

Greetings @Nummer378

Firstly I wanted to thank you. That has to be one of the most comprehensive replies I've ever received on a forum. Not only did I learn from your reply, but it also helped me contemplate options for moving forward.

Regarding the use of Let's Encrypt certificates, I figured a large amount of the domain validation could be done via DNS, through the pfsense firewall?

Mind you, I also managed to setup a root CA on my pfsense box... and following a process on the pfsense forum was able to secure the connection between my ubuntu workstation and pfsense firewall.

I guess for me it boils down to this... what is the least complex way of securing connections between a handful of clients and a handful of servers/systems on my internal network? I'd ideally prefer a cookie-cutter approach i.e. a process I can follow and repeat for various systems/servers and the workstation/client connections.

From what I've understood from what you've written... I need to have a Certificate Authority, either internally/private, or a public authority like Let's Encrypt. Once I have a CA, I then need to configure each of the systems/servers to use the CA, and each client to use a key that can validate to the CA. Hope I've got that correct?

I do like the idea presented by Rudy @rg305 using a proxy, which I figure the pfsense shouldn't have any problem handling. But like I said, for me the least complex approach is going to be the best, purely because this is an internal network and the only reason I am going to the trouble is paranoia.

Thanks again for the replies folks... please know your help is very much appreciated.

4 Likes

hmm...
Does any part of that "internal network" use WiFi?
Are there any zones/VLANs involved?
Exactly how paranoid are you? [we'll be the judge of that by the answers to the first two questions - LOL]

2 Likes

LOL... thanks Rudy. For now, the network has WiFi, and no VLAN's. Longer term I am looking at separating the WiFi via a VLAN. Very little 'admin' work is done via WiFi, so hopefully it shouldn't be a big move.

I am not super paranoid, but would like to work towards each of my internal systems/servers using certs. Some of my systems already use MFA, so the cert side of it is not a super high priority.

Thanks, Mark

1 Like

That would be my biggest security concern.

Also of concern.

If the servers where in a separate VLAN, it would force all connections to pass through the firewall that connects them. And it would also play into doing HTTPS inspection there [reverse proxy].

2 Likes

I do have plans for implementing VLAN's... just finding the time is an issue. But if implementing a reverse proxy is a better first step then I guess I can start there. I just figured understanding the basics of certificates was more important, and after that implementing VLAN's. I've recently upgraded my switches to enable implementing VLAN's, so will look at that for WiFi first.

I've been looking at this video which looks good... How To Setup ACME, Let's Encrypt, and HAProxy HTTPS offloading on pfsense - YouTube

I've done some reading about reverse proxy, and it appears most people are saying the application of reverse proxy is for external traffic coming in, not internal http sites being used by internal clients? So is reverse proxy still a worthy application? Also, regarding the use of ACME / Let's Encrypt on pfsense, I could be wrong, but it appears to use Let's Encrypt I need to sign up to accounts that enable an external DNS provider?

Sorry for all the questions... just trying to get my head around the best way to go forward. HA Proxy on pfsense looks good, just want to be sure it's a suitable solution for my internal http sites.

Do you understand "the benefits" of such a proxy?
Are they still worth the effort in your case?

You will need to use real domain names to get a cert from any globally trusted CA.
[which is how you can avoid the sefl-signed certificate security pop-up/warnings]

2 Likes

Hi Rudy,

I have a real domain name... check. So in your experience, if I have a 'real' domain name, do I need to setup anything external for a DNS? Just trying to get my head around what I need or dont need. As I mentioned, several of the videos I've watched, they refer to using reverse proxy for incoming connections to websites etc.

Here's one I watched... Quick and easy secured reverse-proxy endpoints via HAProxy + ACME on your pfSense machine - YouTube

And to answer your question... do I understand the benefits of such a proxy? I do, but not 100% which is why I am asking questions. I apologise if I am frustrating you, it is not my intention.

I am really interested to trial HAproxy on my pfsense, so yes I think it is worth it. As I said above, I am just asking questions to gain more knowledge. I am a super computer guy... so I know heaps about super computers and AI... but reverse proxy, not so much :laughing:

Maybe.
There are three ways to authenticate:

  • HTTP
    Requires an HTTP response from the FQDN
  • DNS
    Requires TXT zone record update for _acme-challenge.[FQDN]
  • TLS-ALPN
    Requires an HTTPS response from the FQDN [not all web servers support TLS-ALPN]

So, depending on which method(s) pfSense supports, any of your exposed web servers support, any newly added reverse proxy may support, if your DSP (DNS Service Provider) supports automated zone updates via API [for automated DNS updates] will likely dictate which option you may take.

The greatest benefit IMHO is reducing the need to implement TLS in many systems down to just one single system.
But it's not yet certain which method fits best, nor at which system will such a proxy be configured.
I'm leaning towards the pfSense being able to handle all your cert needs [but I'm abnormally optimistic today - lol]

A reverse proxy is a very simplistic thing.
A "normal proxy" is maybe easier to understand: You need something from the Internet, you ask the proxy to get it for you, it does, and you are happy.
A "reverse proxy" works in the other direction: The Internet needs something from your internal server, it asks the "reverse proxy" to get it for them, it does, and the Internet client is happy.
In this sense anyone (including you) that is not directly connected to the internal system, is treated like an Internet user - who must ask the proxy for all things on your internal server(s).
[that is where VLAN separation comes in handy - forcing everyone to cross the proxy (or a firewall)]

3 Likes

Hi Rudy,

So as I wrote above, I've installed ACME and HAProxy. Here's the ACME setup so far...

So as far as ACME goes I think I am ok?

The only area I am a bit confused about is the HAProxy, and the config... mostly around the Front End and the Back End. Any advice greatly appreciated.

Will only let me put a picture per post... so here's some more ACME info.

And some more...

I'm not too familiar with that screen.
I'd try the "Issue" button [once] (if it's an action button).

I'd also defer to the pfSense docs for guidance with HAproxy:
https://docs.netgate.com/pfsense/en/latest/packages/haproxy.html

1 Like