Yet another firewall problem

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mrtg.ndtel.com

I ran this command: /usr/local/sbin/certbot-auto --apache

It produced this output: Failed authorization procedure. mrtg.ndtel.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Fetching http://mrtg.ndtel.com/.well-known/acme-challenge/wMIBIAJy3mop6XfSgFfAqpDljGNKVbrcjTCJKflMwKc: Timeout during connect (likely firewall problem)

IMPORTANT NOTES:

  • The following errors were reported by the server:

    Domain: mrtg.ndtel.com
    Type: connection
    Detail: Fetching
    http://mrtg.ndtel.com/.well-known/acme-challenge/wMIBIAJy3mop6XfSgFfAqpDljGNKVbrcjTCJKflMwKc:
    Timeout during connect (likely firewall problem)

    To fix these errors, please make sure that your domain name was
    entered correctly and the DNS A/AAAA record(s) for that domain
    contain(s) the right IP address. Additionally, please check that
    your computer has a publicly routable IP address and that no
    firewalls are preventing the server from communicating with the
    client. If you’re using the webroot plugin, you should also verify
    that you are serving files from the webroot path you provided.

My web server is (include version): Apache/2.2.15

The operating system my web server runs on is (include version): CentOS release 6.9

My hosting provider, if applicable, is: self

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

This is a Centos/Apache server behind a Fortigate firewall, natted. When the certbot program attempts to browse to the web server at the published IP address, from the server itself, the Fortigate (like many other firewalls) does not allow access. So the process fails.

What is an alternate method to perform the site/domain verification? Is this documented anywhere? I have looked through the documentation on both the letsencrypt and the certbot pages, and have not found this particular situation discussed anywhere…

I have about 50 other domains to do this with. We are a hosting company and ISP, and many of our hosted web sites are very small businesses that cannot afford to spend their dollars on certificates. With the upcoming Chrome encryption requirement looming, we want to get our customer’s sites secured prior to the Chrome change. Having the ability to automate all of this is extremely important to me, due to the number of domains I am responsible for.

Any help would be greatly appreciated! TIA!

How is that logical? If a user wants to surf to the site in question, does the firewall block that request too? If not, why should that be different than when the LE server tries to access the token?

Apparently, your whole site is blocked:

This site can’t be reached
mrtg.ndtel.com took too long to respond.

Try:
Checking the connection

ERR_CONNECTION_TIMED_OUT

No AAAA record, so not an IPv6 issue.

How is it logical? Because, the firewall refuses to allow the traffic from the natted ip address to browse to the public address of that nat. That’s pretty typical of a nat firewall, in my experience.

This is a server that is used for our network management. I am using this machine because it doesn’t affect customer sites, during this experiment/test. We have an acl allowing only our netblocks to access the services. So, you won’t be able to browse to the server from your IP address, which is why you get the timeout message. I should have mentioned this in the original post, sorry about that.

We are not yet implementing IPV6, so no AAAA recs.

BTW, we have a ticket opened with Fortinet about this, attacking the problem from that angle as well. I don’t know how much luck we will have with them, as this is a default behavior that may not be able to be changed…

That's not what's happening though. Certbot doesn't try to connect to your server; it connects to Let's Encrypt (the CA) and Let's Encrypt connects back to your server from their own infrastructure.

This would be the problem: Let's Encrypt needs to be able to connect to verify your control of the domain.

If that's not acceptable you can use the DNS challenge instead.

Wait a minute… The way that I understood it, Certbot itself writes a “private” file to the local filesystem, and then attempts to browse to that saved file via the published public IP address from the machine that Certbot is installed on.

If I misunderstood, and it’s the Let’s Encrypt servers that are INFORMED by Certbot to do the browsing, then that’s even better! Because, I can create a firewall rule to allow the Let’s Encrypt ip’s access to port 80! So, all I need is the IP addresses that the Let’s Encrypt validation servers are using.

Anyone know the IP addresses of the Let’s Encrypt validation servers?

Yes, exactly...

...but nope, sorry.

I guess if it's just for a once-off test you could check your firewall logs for the address the last attempt came from - I don't think they change very much yet, but in general you'll need to have port 80 open to the internet (at least during the validation).

Wow… that’s too bad. And pretty short-sighted, really, especially since the whole point of the process is to make things MORE secure…

For most of my customer’s web servers, this won’t be an issue. But for those like this test server, that I want to limit access to the web service for security reasons, it is a big issue. Knowing the process, though, makes a big difference. Maybe I can write a script that opens things up during the validation process, then closes again afterward.

Thanks for all the help here… You’ve made my day!

Alex

You need to be able to prove to Let's Encrypt that you control the domain name. That can be done by having Let's Encrypt connect to your site, or by setting a DNS TXT record. If you don't want to allow inbound connections to your site, you can look into the DNS TXT record option (called the DNS-01 challenge).

1 Like

I totally understand the validation. And I completely agree, it has to happen. I will look into the DNS-01 option. As it is, I changed the ACL on the firewall temporarily to get the server validated and the cert installed, and it worked perfectly. Now I just need to work on the automation to perform the ACL change for updates later, or alternatively work on the DNS-01 challenge instead.

Thanks everyone for the help! It’s all working now! And I understand the process too, so that’s a good thing as well.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.