Route53 & Certbot using DNS record linked to multiple IP addresses (latency routed)

Help
#1

Hi all,

I have a route53 DNS record that is linked to 2 IP addresses. One IP resides in the US and on in the EU. Based on the latency of the client, route53 can decide which IP is the best choice. It works pretty cool, much cheaper as compared to their load balancer. If you fill in your DNS in https://www.whatsmydns.net/ and have it checked, you can nicely see that the US regions all pick the IP from the US server and the EU regions pick the IP from the EU server.

Anyway, I am using this script https://github.com/tomyates/letsencrypt-install-elasticbeanstalk-single-instance to install autocertbot on my servers and it works perfectly for the US server.

However, when I deploy it on my EU server it fails because it obtains the US IP instead of the EU IP. This itself is strange, because when I perform:

dig dns-of-my-server

On my US machine, I get the US IP and when I perform this task on my EU machine, I get the EU machine. Just what you would expect.

I would have expected that certbot also would get the EU IP when performing the autocertbot task on a EU server but it probably does a different check which results in the US server IP. This then results in IP mismatch and certbot fails.

Is there a way to inform certbot to check for multiple IP addresses linked to this DNS name? Is there a way to do this task? I can login to these machines and have root access so I could perform the manual DNS acme check but I would like to employ the autorenewal feature of autocertbot.

#2

Hi @ejblom

your main idea is wrong.

It's not your local Certbot that checks your dns entry. These are

  • some Letsencrypt servers (US-hosted) and
  • some additional servers (worldwide)

If your dns isn't synchronized, that's bad.

And why doesn't it work with the US-ip?

Looks like your dns synchronization doesn't work.

2 Likes
#3

Hi JuergenAuer,

Thanks for the quick reply! I didn't know Letsencrypt uses US hosted server, but that at least explains the result. Are you familiar with the latency based DNS result from Amazon Route53? It works perfect. When I enter the concerned DNS in https://www.whatsmydns.net/, the US based DNS servers all get the US IP, and the EU based servers get the European IP. So I am pretty sure the DNS works!

Here is another check which I did. When I go the concerned address with my browser, I get an SSL error because I am located in the EU --> I get therefore get the EU IP and that doesn't have a valid SSL certificate.

However, when I start my VPN and use an US server, and repeat the same analysis, it works! Because I then get a better latency with the US based server.

Is there a way to specify which DNS server to use? For instance, if I could tell certbot to use a EU based DNS server, I probably would get the EU IP if I would do the certbot auto on that IP.

#4

If you're using HTTP authentication, then you need to have the challenge file on all servers that might receive the request.

But your setup is really ripe for using DNS authentication instead. Route 53 has a comprehensive API (like the rest of AWS's services), and you can just do that to validate your domain name ownership and not worry about which IP is responding. I'm not familiar with "autocertbot" specifically, but if you use a Route 53 DNS plugin for Certbot it should be pretty straightforward to integrate.

2 Likes
#5

We validate from multiple different regions (simultaneously), so if you're using anycast or DNS load balancing, every one of your front-end points of presence will need to serve the ACME challenge in order for validation to succeed.

3 Likes
#6

No, I don't use that.

But it's general: If you use different name servers, all name servers must send a valid result.

So it looks that your zone isn't consistent.

If you use dns validation, the TXT entry is always the same.

If you use http validation, every ip address must work. So it would be unrelevant if your dns sends the european or the US-ip.

Your informations must work - worldwide.

2 Likes
#7

Hi Peter

I am not using HTTP authentication, I think AWS Route53 checks what the latency is and then decides to provide a certain IP. You are right that there is an API that is also available for Route53, that was on my todo list if I couldn't get it to work.

#8

Hi James

I did the ACME challenge from my Linux server at home. But I read somewhere that if you run (auto)certbot with a DNS name that resolves to the IP address that is making the request, it will run without the ACME challenge? Because autocertbot works perfect without the ACME challenge on my US server.

#9

Hi JuergenAuer,

I am not sure what you mean by: "If you use http validation, every ip address must work. So it would be unrelevant if your dns sends the european or the US-ip."

Because I clearly see in the log that autocertbot is failing on my EU server because it received the US IP (and mentions that IP in the log) when it did a DNS call.

I apologize in advance for these questions. I only gotten subdomains to work this weekend with Amazon route53 and then found out about the latency check based DNS. Here is the actual address, it does seem to work doesn't it? https://dnschecker.org/#A/members2.geneticaffairs.com

#10

If IP addresses of your servers are relevant, you're probably using the http-01 challenge. Please read the basics on how Let's Encrypt (or more specifically, the ACME protocol) works:

and

The script indeed uses the webroot plugin and thus the http-01 challenge type:

Also note that the certbot-auto script is being deprecated and might stop functioning in the future. Also note that the script uses --renew-by-default which is an option best to be avoided before you burn through a rate limit due to some bug in the script.

2 Likes
#11

Your setup is completely unknown.

Please answer all of the following questions:

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:

I ran this command:

It produced this output:

My web server is (include version):

The operating system my web server runs on is (include version):

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

If your dns sends a wrong answer, validation fails, that's expected.

#12

There are two different ip addresses reported:

52.24.114.200
54.170.182.216

So you have two ip addresses.

If this is relevant, you use http validation.

So your ip addresses don't send the same result.

Must check that - wait a minute.

Ah - one is in Dublin - https://check-your-website.server-daten.de/?q=54.170.182.216&h=members2.geneticaffairs.com

Host Type IP-Address is auth. ∑ Queries ∑ Timeout
54.170.182.216 A 54.170.182.216
Dublin/Leinster/Ireland (IE) - Amazon.com, Inc. Hostname: ec2-54-170-182-216.eu-west-1.compute.amazonaws.com

The other is in US - https://check-your-website.server-daten.de/?q=52.24.114.200&h=members2.geneticaffairs.com

Host Type IP-Address is auth. ∑ Queries ∑ Timeout
52.24.114.200 A 52.24.114.200
Portland/Oregon/United States (US) - Amazon.com, Inc. Hostname: ec2-52-24-114-200.us-west-2.compute.amazonaws.com yes

And you see: The first has a result N, the second a result H.

So both ip addresses send different results. So http validation may not work. One check has a screenshot, the other not.

Completely different answers.

PS: If you want to use http validation, both servers must have the same configuration. Minimal the same answer /.well-known/acme-challenge/random-filename-to-validate-your-domain - that file is created via Certbot, Letsencrypt checks that file.

1 Like
#13

Juergen and Osiris, thanks a lot for your comments! I need to read everything first and learn some more. Appreciate the fast replies!

2 Likes
#14

Hi Peter,

I did some reading and found this page:


That seems really doable. I think this is my best approach. Although I can't seem to install certbot on my AWS machine (Elastic Beanstalk - 64bit Amazon Linux/2.9.12). Snapd is not supported in the yum installer. Will investigate.
#15

I had a lot of issues with trying to get both the Certbot Route 53 plugin installed at the same time as the AWS CLI on Amazon Linux 2, since they both seem to use different versions of Python or something. (I don't really have a ton of experience with Python myself.) What I personally ended up doing was actually to do the renewals from a Node script within Lambda which saved the certs to S3 and then separately have the hosts load the certificates from S3. But that's just my hacked-together hobby project; it may not be exactly what you're looking for.

If you want to run a client and get the cert directly from the EC2 machines, and keep having trouble getting certbot installed, you may want to try one of the other clients.

1 Like
#16

The other option you could explore is that AWS fairly recently announced this "enclave" concept where you could have a (free) certificate from AWS Certificate Manager inside an enclave of your EC2 instance such that your EC2 code doesn't actually have access to the private key directly but can use it through a PKCS11 interface (kind of like a hardware HSM would be, except it's actually in software, just software that's highly restricted and you don't have direct access to). They have an example tutorial of using it with nginx.

I love Let's Encrypt, and I think integrating an ACME client using Let's Encrypt is probably simpler. But sometimes it's easier to stay within the ecosystem of one cloud provider, so I wanted to point this out in case you wanted to look into it. [The other option staying within AWS is just not securing the EC2 layer with a public certificate and using CloudFront (with a free ACM Certificate Manager certificate) in front of it to serve encrypted pages to the public Internet. But I'm assuming there's a reason you need encryption on the EC2 instances themselves sending you down this route in the first place.]

1 Like
#17

Thanks Peter!

Your older comment concerning the Route 53 DNS plugin for Certbot made me investigate this further. And I must say, I am quite impressed. Good tip! Here is the page:

As you can see, you only need to add some additional rights to the role policy. Next, I modified the script from https://github.com/tomyates/letsencrypt-install-elasticbeanstalk-single-instance
such that it didn't use the certbotauto but a good installation. Like you said, finding a good installation tutorial for certbot on EC2 is quite a challenge, but I managed. Here is the additional code I implemented:

echo "Removing existing certbot installation"
sudo rm -rf /opt/eff.org/*
echo "Installing python and certbot"
sudo yum -y install python36 python36-pip python36-libs python36-tools python36-virtualenv
sudo /usr/bin/pip-3.6 install -U certbot
sudo /usr/bin/pip-3.6 install certbot-apache
sudo /usr/bin/pip-3.6 install certbot-dns-route53

# Assign value to DOCUMENT_ROOT
DOCUMENT_ROOT=$(sudo /opt/elasticbeanstalk/bin/get-config optionsettings | jq '."aws:elasticbeanstalk:container:php:phpini"."document_root"' -r)
SECONDS=0

# run certbot with complete paths, $LE_SSL_DOMAIN is an env variable
sudo /usr/local/bin/certbot --debug certonly --dns-route53 -d "$LE_SSL_DOMAIN"

# Configure ssl.conf
sudo mv /etc/httpd/conf.d/ssl.conf.template /etc/httpd/conf.d/ssl.conf
sudo sed -i -e "s/{DOMAIN}/$LE_SSL_DOMAIN/g" /etc/httpd/conf.d/ssl.conf

# Install crontab
sudo crontab /tmp/cronjob

# Start apache
sudo service httpd restart

and in the 9-ssl-letsencrypt-single-instance.config I modified:

files:
"/tmp/cronjob" :
mode: "000777"
owner: ec2-user
group: ec2-user
content: |
# renew ssl
0 3 * * 0 /usr/local/bin/certbot renew
# post install ssl s3
0 3 * * 0 /etc/9-post-renew-cp-s3.sh

Hope this is going to work!

Thanks again all for the help. Btw, I did not change anything regarding the Route53 DNS entries, although it was suggested they were wrong.
I deployed the new website on both sites and they both seem to work (tested the US version with a VPN).

#18

Glad you found something that worked for you! (And thanks for reporting back on it so that others can learn if they're trying to do the same thing!) I just wanted to make sure you knew some things:

  1. Once you have it all working you probably don't need the --debug on the certbot command anymore. :slight_smile:
  2. If I understand what this is doing right (while I might sound knowledgable, I've not actually used Elastic Beanstalk before; I've just used EC2 and a 1-system-only "autoscaling"), this is creating a new certificate with each new server. If you spin up servers a lot, you might run into rate limits, especially if the servers are all for the same domain. If you have a lot of servers for one domain, it may be better to share the certificate they all use rather than have each server get its own. (If the servers are all for different domains, you might run into other rate limits but I think you'll probably be fine unless you're at a pretty big scale.) If servers are started up fairly rarely then this probably won't be a problem at all.
  3. I'm not quite sure what you mean by the Route53 DNS entries that "it was suggested they were wrong". The certbot plugin should just update the TXT entries in Route53 as needed and you shouldn't need to touch them.
  4. You say you added "some additional rights to the role policy". I'm assuming that this is the "ChangeResourceRecordSets" permission for your Route 53 Zone. You should be aware that this permissions in your EC2 role allows for anything running on your server to be able to change any record in your Route 53 Zone. In the normal case this is of course exactly what you want to allow certbot to work, but if somehow your server was compromised (hacked due to a bug in your application or Apache or whatever such that a knowledgeable attacker is running code on it) this means that the damage could go further than just that one server. For instance, your A/AAAA/MX/etc. records could be pointed to an attacker's servers, an attacker could get their own certificate with their own private key, and so forth. This may be a perfectly acceptable risk to take, but it's the kind of thing that makes some IT departments nervous. It's a shame that AWS doesn't let one restrict the change permissions to just TXT records or certain names or whatever, they only have the one granularity of allowing changes for a whole zone or not. [Some people mitigate this by creating a separate Route 53 zone just for the acme challenges, give only permissions to that zone to the systems that are making certificates, and have the main zone use a CNAME (or maybe it's NS records or something?) to delegate to the acme challenge zone. That kind of thing can limit your exposure, though of course you're paying for an extra Route 53 zone at $6/year and making the system that much more complicated. It's probably not worth it for most people, I just wanted to let you know about the possibility since some people prefer (or require) doing it that way to keep their permissions using the least privileges possible to get things done.]

Good luck!

1 Like
#19

Hi Peter,

These are excellent suggestions/tips. Let me address some:

  1. indeed! although the --debug is only used during the launch of the server
  2. I am running 2 separate ELB servers, both single instances without the load balancer. The load balancer is really annoying, I couldn't get it to work anymore and I really did not need it with respect to load. So basically only for the DNS certificate, I guess AWS likes to see it that way. Anyway, I currently have 2 servers running under the same DNS, perhaps will expand to 3. Not really a lot right? And I initiate them, and every now and then update the website. Not that often.
  3. I got some DNS feedback regarding Route53. In hindsight, I think it still functions as it should. So this was not linked to certbot. I now use the native solution from certbot which is ofcourse much better.
  4. Excellent suggestion. I am using ELB servers also because they are setup only for serving PHP and nothing else. Next, I only allow port 443/80 from the outside world, will close port 80 probably. But I now understand the risk.

Good points!

1 Like
#20

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.