Acme-dns not reachable, possible DNS problems, "couldn't get address for"

[CoUsT]

My domain is: coust.cloud

I ran this command: dig +trace +additional auth.coust.cloud

It produced this output:

auth.coust.cloud.       600     IN      NS      auth.coust.cloud.
auth.coust.cloud.       600     IN      A       35.211.229.210
couldn't get address for 'auth.coust.cloud': failure
dig: couldn't get address for 'auth.coust.cloud': no more

My web server is (include version): Latest acme-dns commit 68bb6ab from 11.01.2021

The operating system my web server runs on is (include version): Ubuntu 20.04.2 LTS

My hosting provider, if applicable, is: tried on both Google Cloud Compute Instance and on Oracle Cloud Compute Instance.

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): (irrelevant?) the final outcome is managing nginx-proxy-manager on my local network

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): -

Basically I can reach the local acme-dns stuff by running dig @0.0.0.0 auth.coust.cloud and get the answer:

;; ANSWER SECTION:
auth.coust.cloud.       3600    IN      A       130.61.122.175

the problem is reaching the auth.coust.cloud from the outside. All DNS checkers can't resolve the IP address and the dig +trace +additional auth.coust.cloud always returns couldn't get address for 'auth.coust.cloud': failure and dig: couldn't get address for 'auth.coust.cloud': no more

My primary use of the domain is for Mail-in-a-box instance where I set up custom DNS records. I tried to reverting back to porkbun default nameservers and adding records on their web page but the outcome is the same. I checked two machines, one on Google Cloud and one on Oracle Cloud. Firewall allows 53, 80 and 443 ports. Security lists/firewalls on the cloud providers are changed to allow communication on mentioned ports too.

I'm at loss and maybe some of you can quickly catch the mistake or point me in a good directions.

What I'm trying to achieve is running nginx-proxy-manager with added wildcard *.coust.cloud certificate for all my subdomains and auto renewal and I need acme-dns for that. I would like to use my domain name to access local stuff so I added the record to direct main page to 192.168.137.2. I also had wildcard record pointing to the same address before I tried setting up DNS records directly on porkbun site (that I will add back later on after finding the cause of issue). I'm trying to have https to my local nginx-proxy-manager with the rest of apps in docker containers and common network without exposing any ports except 443 - all trafic will be going thru nginx and each app will have their own subdomain.

[Osiris]

@CoUsT

auth.coust.cloud.       600     IN      A       35.211.229.210

This is the "glue" send by the "porkbun.com" nameservers. But that IP address is not reachable. And it differs from the IP address you're showing below.

Are you sure the IP address 35.211.229.210 is correct?

[CoUsT]

Hey.

Yes, this is my current IP address of Google Cloud Compute Instance.

The second code block with 130.61.122.175 IP was my previous Oracle Cloud Instance and I copy pasted results from the wrong putty window. dig @0.0.0.0 auth.coust.cloud on my Google Cloud instance (the one I'm testing with currently) returns correct results:

;; QUESTION SECTION:
;auth.coust.cloud.              IN      A

;; ANSWER SECTION:
auth.coust.cloud.       3600    IN      A       35.211.229.210

I can also reach and ping the server just fine:

C:\Windows\system32>ping 35.211.229.210

Pinging 35.211.229.210 with 32 bytes of data:
Reply from 35.211.229.210: bytes=32 time=138ms TTL=54
Reply from 35.211.229.210: bytes=32 time=147ms TTL=54
Reply from 35.211.229.210: bytes=32 time=139ms TTL=54
Reply from 35.211.229.210: bytes=32 time=138ms TTL=54

Ping statistics for 35.211.229.210:
    Packets: Sent = 4, Received = 4, Lost = 0 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 138ms, Maximum = 147ms, Average = 140ms

The compute instance has mail-server tag which allows following ports:

If that IP is not reachable then there might be some sort of firewall conflict/issue? I installed firewalld (ufw doesn't work on Oracle Cloud) and allowed ports 53, 80 and 443 with following commands:

sudo firewall-cmd --zone=public --permanent --add-port=53/tcp
sudo firewall-cmd --zone=public --permanent --add-port=53/udp
sudo firewall-cmd --zone=public --permanent --add-port=80/tcp
sudo firewall-cmd --zone=public --permanent --add-port=443/tcp

After reinstalling firewall and setting up rules again I can open the website on http://35.211.229.210/ and not get timeout so I think firewall was blocking the access... I will need to investigate further what exactly is going on with the firewall but you certainly pushed me in the right direction.

[Osiris]

I can see some progress, but your DNS server isn't functioning/isn't available properly yet.

This was 2 hours ago with only timeouts: auth.coust.cloud | DNSViz

This is now: auth.coust.cloud | DNSViz

It now says "connection refused" in stead of a timeout. So probably not a firewall issue (could be tho), but more likely the DNS software isn't running.

[CoUsT]

● acme-dns.service - Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely
     Loaded: loaded (/etc/systemd/system/acme-dns.service; enabled; vendor preset: enabled)
     Active: active (running) since Sun 2021-07-04 03:33:51 UTC; 6h ago
   Main PID: 220997 (acme-dns)
      Tasks: 4 (limit: 682)
     Memory: 5.6M
     CGroup: /system.slice/acme-dns.service
             └─220997 /usr/local/bin/acme-dns

This is the output from the systemctl status acme-dns

Restarting service and running journalctl -u acme-dns outputs that the server is correctly listening on 53 and 80 ports.

Jul 04 09:47:57 free-tier systemd[1]: Stopped Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.
Jul 04 09:47:57 free-tier systemd[1]: Started Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Using config file" file=/etc/acme-dns/config.cfg
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Connected to database"
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=debug msg="Adding new record to domain" domain=auth.coust.cloud. recordtype=A
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=debug msg="Adding new record to domain" domain=auth.coust.cloud. recordtype=NS
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=debug msg="Adding new record to domain" domain=auth.coust.cloud. recordtype=SOA
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Listening HTTP" host="0.0.0.0:80"
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Listening DNS" addr="127.0.0.1:53" proto=udp
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Listening DNS" addr="127.0.0.1:53" proto=tcp
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="2021/07/04 09:47:57 [INFO][cache:0xc0000685f0] Started certificate maintenance routine"

It seems there might be some misconfiguration in the firewall after all?

[Osiris]

@CoUsT

Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Listening DNS" addr="127.0.0.1:53" proto=udp
Jul 04 09:47:57 free-tier acme-dns[222346]: time="2021-07-04T09:47:57Z" level=info msg="Listening DNS" addr="127.0.0.1:53" proto=tcp

It's only listening on localhost for the DNS part. I'm not sure if that's intended or not. Please refer to the acme-dns documentation, I'm not that familiar with the software. But only listening on localhost would hamper the remote availability of the DNS part of the software.

[CoUsT]

@Osiris

I'm not sure if that's intended or not

This is what is present in the default config and there are no mentions anywhere to edit this out. I think it simply means "start locally on port X" but when you can reach it over network it will still reply if the port is correct.

I changed it to the public IP but it results in fatal error during start up of the service:

msg="listen udp 35.211.229.210:53: bind: cannot assign requested address"

For 0.0.0.0 the result is similar but the IP/port is already in use:

msg="listen udp 0.0.0.0:53: bind: address already in use"

127.0.0.1 successfully starts and I assume it is what it should be. The problem is why port 80 is reachable and returns stuff but 53 timeouts...

I tried playing a bit with firewalls etc but can't really make the dig to resolve auth.coust.cloud or any other online dns checkers.

[Osiris]

@CoUsT

For 0.0.0.0 the result is similar but the IP/port is already in use:

msg="listen udp 0.0.0.0:53: bind: address already in use"

What's already listening on UDP port 53 if it's not acme-dns?

[CoUsT]

root@free-tier:/home/ubuntu# lsof -i :53
COMMAND   PID            USER   FD   TYPE DEVICE SIZE/OFF NODE NAME
systemd-r 617 systemd-resolve   12u  IPv4  20961      0t0  UDP localhost:domain
systemd-r 617 systemd-resolve   13u  IPv4  20962      0t0  TCP localhost:domain (LISTEN)
acme-dns  623        acme-dns    8u  IPv4  22987      0t0  UDP localhost:domain
acme-dns  623        acme-dns    9u  IPv4  22990      0t0  TCP localhost:domain (LISTEN)

It seems the default ubuntu services or something similar is for some reason listening on port 53 and possibly conflicting?

EDIT: Getting rid of local name resolver and changing acme-dns config ip from 127.0.0.1 to 0.0.0.0 and restarting fixes the connectivity issues. Now I can't resolve any names on the server itself so visiting pages is a no-no but I can make the server, well, serve as DNS. Need to dig deeper how to make both work at the same time.

[Osiris]

I'm not familiar with systemd, systemd-resolve nor Ubuntu, so I can't help you with that I'm afraid.

[CoUsT]

This is fine, you have already put me in the right direction! Thank you for the help.

For anyone stumbling upon this thread in future:

To get acme-dns working correctly on Ubuntu you have to make sure all ports are open and get rid of default and local name resolver listening on port 53 and conflicting with acme-dns. You can check if something is running on port 53 by running lsof -i :53

You have to disable systemd-resolved sudo systemctl disable systemd-resolved and modify config: under [general] change listen to "0.0.0.0:53"

For debugging issues if something still doesn't work for some reason I recommend journalctl -u acme-dns, systemctl status acme-dns and dig domain +trace +additional commands.

[JamesLE]

systemd-resolved (on Ubuntu, at least) only listens on 127.0.0.53 (a specific loopback address) by default:

# lsof -Pn -ai -c systemd-resolve
COMMAND      PID            USER   FD   TYPE   DEVICE SIZE/OFF NODE NAME
systemd-r 973914 systemd-resolve   12u  IPv4 11349502      0t0  UDP 127.0.0.53:53 
systemd-r 973914 systemd-resolve   13u  IPv4 11349503      0t0  TCP 127.0.0.53:53 (LISTEN)

That does conflict with anything trying to listen on 0.0.0.0 (all interfaces), but if you specifically set acme-dns to listen on just your host's external IP address - anything other than 0.0.0.0 or 127.0.0.53 - that should work, without needing to disable systemd-resolved. I just tested this successfully on a fresh Ubuntu 20.04.2 LTS VM.

@CoUsT, is it possible you were trying to run acme-dns as a non-root user, but without the CAP_NET_BIND_SERVICE capability that lets non-root processes bind to low port numbers; or maybe using some kind of containerization that abstracts the external IP you want to listen on, so that the acme-dns process can't find its interface?

[Osiris]

@JamesLE

but if you specifically set acme-dns to listen on just your host's external IP address - anything other than 0.0.0.0 or 127.0.0.53 - that should work,

I believe that was already tried:

@CoUsT

I changed it to the public IP but it results in fatal error during start up of the service:

msg="listen udp 35.211.229.210:53: bind: cannot assign requested address"

However, perhaps the host is behind NAT and the public IP address shouldn't be used? Instead, perhaps the internal private IP has to be used.

[CoUsT]

@JamesLE

is it possible you were trying to run acme-dns as a non- root user, but without the CAP_NET_BIND_SERVICE capability that lets non- root processes bind to low port numbers

I executed all my commands as root user. I also used the service file and instructions from the readme of acme-dns repo so that it starts automatically on boot.

@JamesLE

or maybe using some kind of containerization that abstracts the external IP you want to listen on, so that the acme-dns process can't find its interface?

All was done on a fresh installation of Ubuntu 20.04 LTS tested on both Google Cloud and Oracle Cloud instances.

I tried again just to be sure and this is the outcome:

Jul 05 13:02:37 vpn acme-dns[10837]: time="2021-07-05T13:02:37Z" level=info msg="Listening DNS" addr="130.61.122.175:53" proto=tcp4
Jul 05 13:02:37 vpn acme-dns[10837]: time="2021-07-05T13:02:37Z" level=fatal msg="listen tcp4 130.61.122.175:53: bind: cannot assign requested address"

EDIT:

@Osiris

However, perhaps the host is behind NAT and the public IP address shouldn't be used? Instead, perhaps the internal private IP has to be used.

I think this is the case. I ran ifconfig and this is the result:

ens3: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 9000
        inet 10.0.0.230  netmask 255.255.255.0  broadcast 10.0.0.255

So I assume I should put 10.0.0.230:53 in the config but that was not explained anywhere in the readme or other places making it confusing to get started with acme-dns.