Round Robin DNS

[LogicEthos]

@BenSjoberg

It's possible to use the same private key on multiple hardware security modules. The mechanism for doing so depends on the manufacturer.

OMG, I hope not!!

[BenSjoberg]

@LogicEthos

It's just less secure, do to a policy decision by LE.

What security issue are you worried about? With SSH you can copy the keys from server to server without any significant security risk. Double-check the SSH fingerprint the first time you make the connection and you should be good.

[LogicEthos]

@BenSjoberg

What security issue are you worried about? With SSH you can copy the keys from server to server without any significant security risk. Double-check the SSH fingerprint the first time you make the connection and you should be good.

If one server gets compromised, then they all do. I didn't say it was insecure as such, just less so.

[aarongable]

@LogicEthos

Instead of pulling a single A record, you pull them all.

We do. But as mentioned above, we do not have and do not want to have the request origination IP address information to compare the list against.

@LogicEthos

So how do you return the request?

For security reasons, the system performing validation is not the same as the system which received the HTTP request that validation be performed. And as you might be aware, ACME validation is an asyncrhonous process: the client makes an HTTP request saying "I have fulfilled the challenge", and the server immediately responds with "okay, we'll get started on checking that". Some time later, the client makes a separate HTTP request asking "have you validated my challenge yet?" and the server responds with yes or no. So during the validation process itself, there is no open connection to the client, and of course the two requests that I described may come from different IP addresses.

@LogicEthos

Same content. Different private key.

Sorry, no, that's what I'm saying: they don't have the same content, despite your assertion that they do. One of the servers has a specific value at the /.well-known/acme-challenge/<token> path, while another server does not. Therefore they do not have the same content. If you replicate that content (which is not the same thing as replicating the private key) then they will be serving the same content, and this issue will go away.

[LogicEthos]

@aarongable

For security reasons, the system performing validation is not the same as the system which received the HTTP request that validation be performed. And as you might be aware, ACME validation is an asyncrhonous process: the client makes an HTTP request saying "I have fulfilled the challenge", and the server immediately responds with "okay, we'll get started on checking that". Some time later, the client makes a separate HTTP request asking "have you validated my challenge yet?" and the server responds with yes or no. So during the validation process itself, there is no open connection to the client, and of course the two requests that I described may come from different IP addresses.

Thank you. This was the answer I wanted.

[BenSjoberg]

@LogicEthos

If one server gets compromised, then they all do. I didn't say it was insecure as such, just less so.

If you have multiple certificates for the same domain, and the private key for one of them is stolen, then you're no better off than if the keys for all of them were stolen, because the attacker now has a valid certificate they can use to impersonate your site. All of the certificates are equally valid to users' browsers.

[petercooperjr]

You may want to look at the "Storing and Reusing Certificates and Keys" section of the "Client and Large Provider Integration Guide":

Integration Guide - Let's Encrypt

While having each server have its own private key does have some advantages over having one shared key that gets securely distributed to all the servers, each additional certificate adds a cost for Let's Encrypt. (Certificates from Let's Encrypt aren't free, they're just free to you the subscriber because donations pay for the cost of the infrastructure.) So if you're looking at how to scale out beyond just a couple servers, you're going to want to figure out how to manage your certificate acquisition a bit more carefully anyway so you don't run afoul of the rate limits that Let's Encrypt has put in place to help ensure their resources aren't abused and in general just to be a "good user" respecting the costs of what you're requesting.

I think that most people scaling up beyond a couple of servers find the tradeoff of having the private key shared between the servers to be worth doing. If not, they probably have the TLS connection terminated at their load balancer rather than the individual servers, so that it's just the load balancer that needs the private key (and have different keys, probably not even from a public CA, if needed to secure the connections between the load balancer and the web servers).

[LogicEthos]

@BenSjoberg

If you have multiple certificates for the same domain, and the private key for one of them is stolen, then you're no better off than if the keys for all of them were stolen, because the attacker now has a valid certificate they can use to impersonate your site. All of the certificates are equally valid to users' browsers.

Point taken, but I could invalidate that certificate. I assume the others would be ok.... admittedly I am not read up on that.

[jvanasco]

@LogicEthos

Copying the cert private key is the way to get around it. It just makes it less secure.

Reusing a Certificate/KeyPairing across multiple servers does not make a system less secure. It is also the recommended solution for clustered environments and autoscaling systems.

Obtaining a duplicate Certificate/Key for each node is almost always an anti-pattern that has historically caused problems for users.

A recap of the typical solutions for multiple nodes, many of which were mentioned above:

• Deployment:
  • Share a Certificate/Key across your network. This can be handled as a push or pull command.
• Procurement:
  • Redirect all requests to /well-known/acme-challenge/ to a specific FQDN that is only handled by a single server
  • Proxy Pass all requests to /well-known/acme-challenge/ to a specific backend within your network
  • Utilize DNS-01 challenges, preferably with a delegated DNS solution like "acme-dns"
  • Serve the acme-challenge directory from a networked directory that is mounted onto all servers

There are a handful of other solutions for these types of environments, these are just the most popular.

[rg305]

@LogicEthos

LE pulls the full A records list, and checks that the 4th server's IP is in the list.

@LogicEthos

LE checks DNS records. IP 123.123.123.123 is in the A record list.

@LogicEthos

Instead of pulling a single A record, you pull them all. It's just a different DNS request. One line of code.

How [exactly]?

As a DNS person, I'm just trying to understand exactly how this can be done.