Completing A Challenge when DNS is Globally Load Balanced (Multiple DNS Entries Per Domain)


#1

At work am managing a dozen globally balanced DNS A records that are then round robinn-ed at the datacenter level.

Common enough setup. I was able to get the cert in place by stupid DNS tricks to briefly point the domain at an admin server to generate the cert and then propogate it to the lbs. This requires a brief maintenance window every 3 months so is not ideal.

Another idea is that at the DNS layer, I can filter responses by prefix (ie first 3 octets of the IP) or by ASN. So I could figure out letsencrypts ASN or prefix - another hacky fix and against the spirit of:

thank you

Michael


#2

@mbubb, Let’s Encrypt does not want to publish source addresses from which validation will occur, in order to use unpredictable ones in the future to thwart some network-based attacks against validation.

People in your situation have had good luck with two approaches:

① You can use the DNS-01 challenge type, where instead of having Let’s Encrypt connect directly to your site, you prove control of your domain by setting a DNS entry.

② If each host has an HTTP listener, they can all return a 301 redirect for http://thathost.example.com/.well-known/acme-challenge/ to http://authserver.example.com/.well-known/acme-challenge/, which the CA will follow regardless of which particular round-robin host it hits. The redirect can point to a centralized server which is where you actually run the ACME client (with HTTP-01 validation); then the certificate authority will hit a random host and get redirected to that server to complete the validation process.


#3

Thank you @schoen - I want to do this the right way so completely understand why lE wants to have anonymized IPs

I like #1 and thought of it as a first choice. I think I misunderstand how this works though.

Lets say I have 6 A RECs for test.example.com set up in 2 regions:

EUROPE
1.1.1.1
1.1.1.2
1.1.1.3

ASIA
1.1.1.4
1.1.1.5
1.1.1.6

and lets stay the LE server that answers the CSR ends up closer to Europe. It will only ‘see’ these addresses as valid:

1.1.1.1
1.1.1.2
1.1.1.3

How would it be able to cert the ASIA based IPs?

(edit)

I should qualify and say this is more of a concern for future implementations of LE. In the FAQ it says in the future the challenges would come from multiple servers. I am wondering if a challenge came from an Asian based server and a European based server on the above geobalanced url - would this likely break?


#4

Hi @mbubb,

Let’s Encrypt certificates only certify a domain name, and don’t mention an IP address, regardless of whether that domain name is pointed at 0, 1, or 10000 physical hosts or IP addresses. In the current web PKI, the same certificate, once issued, can be deployed on any IP address as long as the browser believes that it accessed that IP address as a result of trying to connect to a specified hostname that matches the certificate content. So the browser doesn’t care if example.com resolved to 1.1.1.6 or 1.1.1.3 as long as the certificate it got back includes “example.com” as a subject name.

Therefore, you don’t have to worry about this aspect if you’re using the DNS validation challenge, assuming that your DNS zone is globally consistent with regard to extra stuff that you add to it. That is, when you add the extra validation record, that particular record does have to be visible from anywhere for the validation process to complete.

Edit: some CAs are willing to issue a certificate for an IP address, but this is still only checked “permissively” (or we could say “disjunctively”) if the browser is trying to access the site by IP address, so it says “it’s OK if you think the site was called this”, not “the site is required to be served from this IP address, and only this IP address”.


#5

Thank you - am testing this now


#6

@mbubb

there are a couple of other ways of achieving this

if you use a tool like Puppet or Ansible you can deploy the challenge to all your servers so it really doesn’t which server LetsEncrypt hits they will all validate the challenge

if you are using a global load balancer (F5, Citrix Netscaler) you can also have the redirection on that (rather than at each web server)

Andrei


#7

@ahaw021

The DNS-01 way has worked reasonably well. I use NS1 so need to get the api call to update the TXT record.

I like your idea of using Ansible to first pull the .well_known file from the one server and disseminate it but in this case I am using a SAN cert on haproxy so I have 4 parallel challenges to keep track of. (ie 4 domains on the SAN). DNS is much simpler and is a little more future proof. If a challenge was issued from a server in Singapore and Amsterdam at the same time and something went wrong it would be easier to troubleshoot vis-a-vis DNS.

By future proof I mean that LE is saying they will issue challenges from multiple servers

I have an Ansible playbook that is handling some of this - the initial cerbot-auto run is still manual though.

Since you mentioned Ansible have you gotten the letsencrypt module to work?
https://docs.ansible.com/ansible/letsencrypt_module.html
It looks promising but a little beta atm. I am a little unclear as how to use this for a SAN cert and how to update the cert with the module. Promising start though.


#8

hi @mbubb

NS1 looks good and they have a python library

there is also a more generic library called DNS lexicon https://github.com/AnalogJ/lexicon which allows manipulations of various DNS providers

Andrei


#9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.