Well just having the ip.txt available won’t affect much as only folks who know what they are doing will properly maintain and firewall whitelist letsencrypt ips listed in ip.txt. To other regular folks the existence of ip.txt list has no bearing on how Letsencrypt validation servers operate 
Very easy to do with firewalls like CSF Firewall http://centminmod.com/csf_firewall.html