Currently, we are using a pfSense + HAProxy + the ACME certificates package that handles LetsEncrypt communication.
Right now with discourse.lubuntu.me which I can confirm reaches the pfSense appliance, the Certbot side of CA verifications, etc. says "timeout" despite the connections and DNS being updated globally.
Is there a way to see what the LetsEncrypt servers are seeing for DNS resolution to see if they're extremely stale? Google DNS and most major DNS providers are seeing updated IPs so why these timeout and hard fail for LetsEncrypt (both prod and staging servers tested!) is a head-scratcher and concerning.
Let's Encrypt uses the authoritative DNS servers directly.
Some tools that can help you see how connectivity to your DNS and your servers work from the outside, similarly to how Let's Encrypt checks it, include Unboundtest, Let's Debug, and DNSViz.
Interesting then that it's timing out. I'll have to do some additional research 'cause I see the requests landing but then it shouldn't be getting a timeout.
Another thing to keep in mind when looking through logs of validation attempts is that Let's Encrypt checks from multiple locations simultaneously to help ensure that you actually own the name as seen from everywhere on the Internet, so you should be seeing at least three checks from different IPs each time.
Okay, so something about pfSense was busted with the thing, and not accepting that the backend / site WAS online, so it resulted in 503 responses despite the backend being online.
SO, I went haywire on the environment. The pfSense has been replaced with an nginx reverse proxy that handles handoff to proper backends instead of HAProxy, and after some chaos of misconfiguring a few things myself, I went full network sysadmin in "SCREW THIS" fashion and just made it work. Yay for understanding NGINX and hating NAT.
ANYWHO, I got it working, but it is indeed odd behavior so I'll have to investigate that further. But, I guess it works now. shrugs