Note that the IP addresses that Let's Encrypt uses may change over time, on a scale of seconds to months. We've said in various FAQs and forum threads that we don't document the IP addresses and don't intend to support specially whitelisting them.
So if a specific validation IP address is whitelisted by your firewall, you might be seeing intermittent failures for that reason because the validation might sometimes occur from that address and sometimes from a different one.