Firewall Issues

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: darkshado.ca

I ran this command:

certbot certonly --manual --manual-auth-hook /etc/letsencrypt/acme-dns-auth.py --preferred-challenges dns --debug-challenges -d server1.homenet.darkshado.ca

It produced this output:

My web server is (include version):

apache2 -V
[Mon Jul 11 05:37:04.250641 2022] [core:warn] [pid 3868370] AH00111: Config variable ${APACHE_RUN_DIR} is not defined
apache2: Syntax error on line 80 of /etc/apache2/apache2.conf: DefaultRuntimeDir must be a valid directory, absolute or relative to ServerRoot
Server version: Apache/2.4.53 (Debian)
Server built: 2022-03-14T16:28:35
Server's Module Magic Number: 20120211:124
Server loaded: APR 1.7.0, APR-UTIL 1.6.1, PCRE 8.39 2016-06-14
Compiled using: APR 1.7.0, APR-UTIL 1.6.1, PCRE 8.39 2016-06-14
Architecture: 64-bit
Server MPM:
Server compiled with....
-D APR_HAS_SENDFILE
-D APR_HAS_MMAP
-D APR_HAVE_IPV6 (IPv4-mapped addresses enabled)
-D APR_USE_PROC_PTHREAD_SERIALIZE
-D APR_USE_PTHREAD_SERIALIZE
-D SINGLE_LISTEN_UNSERIALIZED_ACCEPT
-D APR_HAS_OTHER_CHILD
-D AP_HAVE_RELIABLE_PIPED_LOGS
-D DYNAMIC_MODULE_LIMIT=256
-D HTTPD_ROOT="/etc/apache2"
-D SUEXEC_BIN="/usr/lib/apache2/suexec"
-D DEFAULT_PIDLOG="/var/run/apache2.pid"
-D DEFAULT_SCOREBOARD="logs/apache_runtime_status"
-D DEFAULT_ERRORLOG="logs/error_log"
-D AP_TYPES_CONFIG_FILE="mime.types"
-D SERVER_CONFIG_FILE="apache2.conf"

The operating system my web server runs on is (include version):

Debian 11
uname -a
Linux server1 5.10.0-14-amd64 #1 SMP Debian 5.10.113-1 (2022-04-29) x86_64 GNU/Linux

My hosting provider, if applicable, is: shaw.ca

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot --version
certbot 1.12.0

Ok I think I know what the issue is this is a firewall issue when I run the certbot when my
firewall is up it just sits there and hangs but if I flush my IP tables using
iptables -F and destroy my IPSETS with ipset destroy then it works.

I think I know what the issue is I have a pretty aggressive firewall which blocks all traffic
from digitalocean.com due to their wide spread abuse that seems to come from their
IPs and their lack of caring about abuse.

I whitelisted the IP block 172.64.0.0/13 which is cloudflare.com but the server is not hosted
on cloudflare I know that and you are using another hosting provider to host the actual
server to get a response from to do the challenge to get and renew certificates. You are just
using cloudflare.com as an anti-ddos protection service.

I believe that you may be using a droplet on digitalocean.com and my server isn't getting
a response from https://acme-v02.api.letsencrypt.org

So I need to be able to whitelist the IP block for the actual server for acme-v02.api.letsencrypt.org

I really don't want to have to drop my firewall every time to renew the certificate.

What I would like to do is be able to white list say the /24 where your server is.

What is the IP block for which the renewals are done please. This way I can
just white list that IP block where server is located. I need the actual IP block
where the server is though not cloudflare.com

If you could give me this information for the IP block where https://acme-v02.api.letsencrypt.org
resides I can white list it and fix the issue.

Thank you,

Jamie (she / her)

2

Hello Jamie,

As you've noticed, our API is behind Cloudflare. The IPv4 block you've posted does seem reasonable, though note that we also have an IPv6 address for the API. You can check DNS to find that if you need the value, or via cloudflare's page https://www.cloudflare.com/ips/. We cannot guarantee that we'll use these IP ranges forever.

You shouldn't need to separately firewall other IP addresses if you just want to talk outbound from your network to our API. Since you're using the DNS challenge, Let's Encrypt also needs to reach your DNS server.

If your client is hanging connecting to the API, it's possible something else is wrong with your iptables rules. Are you allowing established/related traffic? What happens if you try to connect with another tool? Maybe curl -v https://acme-v02.api.letsencrypt.org?

7 Likes
3

Mcperrinm there is nothing wrong with my IP tables rules at all this is because
you are using cloudflare.com as a anti-ddos service and that is just a pass though service
if you are on digitalocean.com the response
will get dropped by my rules because it refuses any incoming from digitalocean.com

again I am asking what is the IP block or IP range for the server where acme-v02.api.letsencrypt.org
is located if you give me the CIDR then I can easily fix this issue.

Again there is nothing wrong with my IPTABLES Rules. It isn't just outbound either
your server sends a response as well which will get dropped by my firewall.

Again this is NOT just outbound it is both inbound and outbound as your server
sends a response as well which the script isn't getting and which is why it is failing.

Again please give me the IP range which I am asking for and this will be easily
fixed.

Also if I do a
curl -v https://acme-v02.api.letsencrypt.org

curl -v https://acme-v02.api.letsencrypt.org

  • Trying 172.65.32.248:443...
  • Connected to acme-v02.api.letsencrypt.org (172.65.32.248) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt
  • CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: CN=acme-v02.api.letsencrypt.org
  • start date: Jul 10 19:29:45 2022 GMT
  • expire date: Oct 8 19:29:44 2022 GMT
  • subjectAltName: host "acme-v02.api.letsencrypt.org" matched cert's "acme-v02.api.letsencrypt.org"
  • issuer: C=US; O=Let's Encrypt; CN=R3
  • SSL certificate verify ok.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x55766e31d2c0)

GET / HTTP/2
Host: acme-v02.api.letsencrypt.org
user-agent: curl/7.74.0
accept: /

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
    < HTTP/2 200
    < server: nginx
    < date: Mon, 11 Jul 2022 15:09:45 GMT
    < content-type: text/html
    < content-length: 1540
    < last-modified: Thu, 23 Jun 2022 21:25:28 GMT
    < etag: "62b4da48-604"
    < x-frame-options: DENY
    < strict-transport-security: max-age=604800
    <
Boulder: The Let's Encrypt CA header { display: flex; max-height: 30vh; flex-wrap: wrap; margin-bottom: 10vh; } header img { display: flex; max-height: 20vh; align-content: flex-end; margin-right: 20px; }

Boulder
The Let's Encrypt CA

This is an ACME Certificate Authority running Boulder.

This is a programmatic endpoint, an API for a computer to talk to. You should probably be using a specialized client to utilize the service, and not your web browser. See https://letsencrypt.org/docs for help.

If you're trying to use this service, note that the starting point, the directory, is available at this URL: https://acme-v02.api.letsencrypt.org/directory.

Service Status (letsencrypt.status.io) | Let's Encrypt Twitter

* Connection #0 to host acme-v02.api.letsencrypt.org left intact if I do a curl to there it seems to work.

Thank you,

Jamie (she / her)

4

@jamieb9999 please see the Let's Encrypt FAQ: What IP addresses does Let’s Encrypt use to validate my web server?

10 Likes
5

What is the actual problem you are having? Because I see you were issued a wildcard cert today.

Also, your most recent post shows you successfully connecting to the Let's Encrypt API endpoint. So, I don't see any firewall interference with the API requests.

10 Likes
6

Completely unrelated, but you should correct this:

9 Likes
7

In your first post, you showed --preferred-challenges dns, so we'll do DNS lookups to your authoritative DNS server for your domain to prove control. Those requests will come from a handful of places, which somewhat frequently change. Because we change the vantage points routinely, we don't publish a list of those.

Your post indicated hanging, which isn't what I'd expect if you're blocking the DNS lookups -- you should get an error message back from the API that validation failed.

12 Likes
8

The Apache2 isn't the problem here i am doing a DNS verification issue that isn't the problem
rg305

9

MikeMcQ I got the wild card cert when I dropped the firewall rules it would not work
when I had the firewall rules up. It only worked once I dropped the firewall rules.

Again I am asking what IP block do I whitelist for
https://acme-v02.api.letsencrypt.org

10

Again @jamieb9999 please see the Let's Encrypt FAQ: What IP addresses does Let’s Encrypt use to validate my web server?

3 Likes
11

And yet you said:

If so, you don't need to whitelist any IPs.

Maybe you could show us the renewal config file.

8 Likes
12

I do when the firewall is up when I got that wildcard record the firewall was down
and the rules removed when it is up it doesn't work. I give up no one here is going to help me.

" We don’t publish a list of IP addresses we use to validate, and these IP addresses may change at any time. Note that we now validate from multiple IP addresses."

Ya well I am not asking for an exact IP just an IP range to whitelist or CIDR that you
should be able to provide some one. But I guess not so well goodbye. This will be my last response.

Jamie (she / her)

13

Should require a TXT record at:
_acme-challenge.server1.homenet.darkshado.ca
OR a wildcard to cover it at:
_acme-challenge.homenet.darkshado.ca

But the only TXT record I can find is at:
_acme-challenge.darkshado.ca text = "F6dd3vx0LmYXT4q0SZbRL3dVKx9uR0MuUuijBFRzdcM"

9 Likes
14

You are completely misunderstand what we are saying.

DNS authentications would only "touch" your domain at your authoritative DNS servers:

darkshado.ca    nameserver = ns-cloud-a1.googledomains.com
darkshado.ca    nameserver = ns-cloud-a3.googledomains.com
darkshado.ca    nameserver = ns-cloud-a2.googledomains.com
darkshado.ca    nameserver = ns-cloud-a4.googledomains.com

So, unless you have a firewall in front of "GoogleDomains.com", there is nothing you can firewall to allow, nor prevent, those DNS requests from completing.

11 Likes
15

I guess you could just turn off your firewall for the duration of the validation cycle.
(not that I am recommending it, just a suggestion for a possible solution)

1 Like
16

And, that was answered in post #2. You are mistaken that you will see the IP address of the Let's Encrypt origin servers on the "far side" of the Cloudflare CDN. You do not.

Any client (in this case certbot) connects to the Cloudflare CDN. The Cloudflare Edge repackages that and sends it to an Origin Server (in this case Let's Encrypt server). The Origin's response is returned to the Edge which repackages it and presents it to you. You really have no visibility to what happens behind that Edge. It could be forwarded multiple times to any number of back-end servers and you would not know.

You have presented a diagnosis but I am interested in the symptoms. I don't agree with your diagnosis because it does not fit all the facts of the underlying architecture.

We may not be able to get past this and it would be a shame. So far the only thing we agree on is that something is amiss.

10 Likes
17

That's the whole point. Your site should be available to the whole internet not just a "block of ip's"
It sounds like you are blocking for tons of stuff. Not really best practice if you want to be found on the world wide web.
My 2 cents

8 Likes
18

The tone of this thread is going a bit off the rails; let's all try to be constructive here. The question asker has indicated they aren’t going replying any more, so I’m closing this thread.

5 Likes