There have been a few discussions around white-listing outbound IPs for the LetsEncrypt API.
As many have mentioned the use of Akamai and "Cloud Service" model means that fixed IPs are not guaranteed.
An article about this can be found here:
The question that comes to my mind - what is the feasibility, how many IPs should we list and what impacts the change of IPs.
To this end I set up a little experiement where I resolved the acme-v01.api.letsencrypt.org every 15 minutes using 14 different revolvers.
The code I wrote essentially resolves the IP of acme-v01.api.letsencrypt.org, tries to connect over port 443 and establish a TLS handshake. If these succeed then a connection is said to be successful.
The idea here was to see how long each IP was valid for. The result set is currently not usable. This is due to the way Splunk aggregates data.
Lets Say a DNS Resolver used IP 1 on day 1 and then served it up for 4 hours. The run should be 4 Hours. However what I am finding is that the DNS Resolver will reuse the IP at a later date so it looks like a run is 10 days (when in fact there were multiple runs during those 10 days).
Fixing this is tricky in Splunk so I am going to fix it in PowerShell (have a run number that is incremented when there is an IP change and add it to the record). This will allow better understanding of IP behaviour.
If you are using a good DNS Provider connectivity issues to the API are most likely to do with Library and Firewalls. The DNS providers assessed issue valid IP addresses and connectivity using these IP addresses is stable
DNS behavior from provider to provider changes so choosing a DNS provider that doesn’t change as much is a valid tactic. There are challenges on how to update it on the server itself (I am thinking host records) but there is a discrepancy between providers.
Whitelisting IP addresses does seem like a feasible proposition currently. However, LetsEncrypt may make IP changes more frequent in the future.
I am going to make a couple of tweaks to get some better insights in to CDN functioning.
These are reverse IP lookups to see what AKAMAI endpoints are being hit.
Timing more accurately how long each IP “RUN” lasts for.
Select DNS Providers that provide a better geographic pictures (i.e. Oceania, Latin America, India etc)
I wouldn’t even begin to assume that inbound traffic to acme-v01.api.letsencrypt.org and outbound traffic are at all related to the same IP addresses. There is no technical reason why they should. LE is free to setup their infrastructure in a completely different way tomorrow and there is no way to know or assume anything from the client point of view.
The whole premise of this study is flawed.
Edit: Also I find it weird to make a study as if there is nothing known about IP protocols and DNS etc. You’re not discovering anything here that you can’t already explain, because there are standards and there is free choice on LE’s part on how to implement things. Even if this study reached some conclusion, tomorrow it can be invalid on a whim. It’s pointless.
I didn't assume that @ahaw021 was claiming that the inbound and outbound traffic (e.g. for challenge validation) use the same IP addresses, and indeed I can confirm that challenge validation does not use the same IP addresses as the API endpoints.
If I want to filter outgoing traffic on my network based on DNS information, I simply establish a hostname<->address association that gets updated regularly, preferrably right before my infrastructure wants to use that name. I don’t go and scribble down individual IP addresses and make studies about how and when these might change. This would be madness.
If you want to limit access on the HTTP layer, you simply use a proxy and whitelist acme-v01.api.letsencrypt.org.
just because inbound links are proxied through Akami, that doesn’t mean any outbound traffic would come through akamai’s network
I operate an active indexer in the 100M-1B range of links. In terms of dealing with CDNs and edges, it’s a crapshoot. the akamai pools can gradually change over weeks, or instantly change overnight. over a 4 week course a single domain might have 100+ ips on different networks.
if there is a concern about whitelisting IPs, better strategies are around whitelisting entire blocks [e.g. akamai has 30+ assigned networks with ARIN https://whois.arin.net ].
The study was focused on outbound traffic not inbound traffic for validations
The premise was - if I had to give the firewall team one or two IP addresses would I be able to complete the challenge in time (are the IP addresses changing on an hourly, daily or weekly basis)
That was suggested in the article initially. However due to the network mask that would whitelisting approximately 4 Million Address (with a /10 subnet mask).
If things in here are already knowledge to you feel free to chime in on the next white listing request for assistance. It would be good to get feedback from people that have to get outbound IPs white listed if they would prefer a small range or if they are ok with white listing 30 IP ranges.
Sorry, I’m not sure if you understand this point that I’ve tried to make: by the nature of how Akamai operates it is almost guaranteed that the ips for acme-v01.api.letsencrypt.org will be on a completely different network. (some networks run a CDN + Hosting service combo)
In order to whitelist for your desires, you need to monitor where the requests from Boulder are coming from. That might point to a handful of smaller networks (perhaps using colocated servers or cloud providers with dedicated IPs), but it could point to a larger IP space that is likely to be volatile and/or distributed (such as amazon).
Akamai might be fronting the api, but that doesn’t mean the connections terminate within Akamai’s network. They most-likely terminate in one or more other networks, which then trigger a request on a Boulder instance to verify against your domains.
The challenge I see (and many not have thought through) is how would you monitor these validation requests if you can’t submit challenges due to outbound HTTPS to the acme-v01.api.letsencrypt.org being blocked by firewalls?
For outbound traffic, you simply employ a filter that understands “acme-v01.api.letsencrypt.org” and not individual IP addresses, i.e. an HTTP proxy or a packet filter with the ability to periodically resolve host names and use the resulting addresses.
For inbound traffic, there is no point in whitelisting. Any challenge you can serve is sitting on a service that you are running anyway, and simply serving the LE challenge is no bigger an attack surface than serving the regular stuff is.
