LetsEncrypt: “There was an error updating the certificate: Verification timed out”

Can’t renew LetsEncrypt cert on a FreePBX system. Posting here in the hopes that y’all will have some valuable feedback.

outbound1.letsencrypt.org, outbound2.letsencrypt.org, mirror1.freepbx.org, mirror2.freepbx.org are excluded in the Firewall on the system.

Ran a packet capture whilst requesting the cert. It gets all the way to the acme challenge from remote servers. My server sends back a 200 OK.

Then I see a BUNCH of SYNs from the same servers that my server never respond to. The remote servers keep retransmitting the SYNs, but after awhile, seeing that I’m not responding, they give up.

I’m very confused. Can’t run certbot manually in the terminal and can’t find LE logs anywhere T_T. Not sure were to go from here

Where was the packet capture run?

Are those SYNs allowed by the firewall?
If so, where do they NAT to?

And… Welcome to the LE community!

1 Like

This suggests (to me anyway) you’ve got quite a “strict” set of firewall rules set up. Let’s Encrypt recently has set up Multi-Perspective Validation. This might be a problem for your firewall setup. For Let’s Encrypt to validate the challenge properly, it isn’t wise to use a “block all, allow some” kind of firewall setup, so I would recommend change it to an “allow all, block some” setup.

3 Likes

At least for the HTTP validation requests.

I left that excercise to the reader :stuck_out_tongue:

Hi, thanks for responding! Normally people take a day or two to respond on forums like this, I totally didn’t expect a response this fast.

This suggests (to me anyway) you’ve got quite a “strict” set of firewall rules set up. Let’s Encrypt recently has set up Multi-Perspective Validation. This might be a problem for your firewall setup. For Let’s Encrypt to validate the challenge properly, it isn’t wise to use a “block all, allow some” kind of firewall setup, so I would recommend change it to an “allow all, block some” setup.

It’s a VOIP server that’s using a responsive Firewall. Unfortunately, due to the nature of the beast, allow all, block some, isn’t really an option I’m afraid. So many people try to hammer on it to try to setup their offshore phone accounts (or something, I don’t know) that it’s not secure that way. The VOIP server is not behind another firewall either, it has its own and that’s it.

That said, I could try whitelisting every IP I see in the packet capture… It’s DEFINITELY possible that this validation thing is why it’s failing, because it has been working for years and it just started failing this month. I’ll check it out.

Ran a packet capture whilst requesting the cert.
On the FreePBX (VOIP) system - the one requesting the certificate. I SSH’d in and ran tcpdump.

Are those SYNs allowed by the firewall?
This is a DUH question that I need to look into, as I indicated to the other person below.

Thanks for the welcome!

That may be an endless task and unless you are only allowing IPs that try /.well-known/acme-challenge/ requests, your are also allowing the wrong IPs. (not recommended).
In other words, the allowed requests should be URL based NOT IP based.

Is there any way to put this VoIP system behind any firewall?

Sadly no. :frowning: Yeah, I whitelisted the IPs and it worked. Reading up, people over at FreePBX are saying you just have to open port 80 all the time for automatic, or manually open 80 when we renew every 2 months and closing thereafter. Thanks for the help though guys!

1 Like

If you could script the open/[get cert]/close (on port 80), that may be a decent firewalling approach.