Failed to authenticate some domains - nginx - netbox

[b-fraley]

@rg305

@b-fraley

root /var/acme-challenges/

@b-fraley

sudo mkdir /var/acme-challegnes

Is that a TYPO?

It was when I posted it, but I double checked and it is spelled right in my config, my bad!

@rg305

@b-fraley

"unable to connect" error.

@b-fraley

sudo certbot certonly

"certonly" won't change anything within the nginx config.
So, that problem isn't related to the new cert.
Speaking of which, please show:
certbot certificates
[to confirm all that]

Just checked and it was successful. Here is the output of sudo certbot certificates:

Found the following certs:
  Certificate Name: netbox.top-floor.net
    Serial Number: REDACTED
    Key Type: RSA
    Domains: netbox.top-floor.net
    Expiry Date: 2022-10-11 13:54:54+00:00 (VALID: 89 days)
    Certificate Path: /etc/letsencrypt/live/netbox.top-floor.net/fullchain.pem
    Private Key Path: /etc/letsencrypt/live/netbox.top-floor.net/privkey.pem

@MikeMcQ

@b-fraley

Which made it sound like I was able get my cert! But now like I said, my netbox instance gets an "unable to connect" error. And I am willing to take a stab its not LE but nginx itself.

As Rudy noted, this is unrelated. Did you stop nginx or some other key component after getting the cert? Because I do not see port 80 or port 443 open but you must have had at least port 80 open to get the cert using webroot.

I did not stop nginx, after I got the cert but I did try and reload nginx hoping the cert would be applied and it failed to restart. The error seems to indicate that port 80 is already in use. See below, but from what I can tell, both internally and externally port 80 and 443 are open on the VPS, atleast according to UFW. You can look here:

sudo ufw status verbose
Status: active
Logging: on (low)
Default: deny (incoming), allow (outgoing), deny (routed)
New profiles: skip

To                         Action      From
--                         ------      ----
2230                       ALLOW IN    Anywhere
9090/tcp                   ALLOW IN    Anywhere
80/tcp                     ALLOW IN    Anywhere
443/tcp                    ALLOW IN    Anywhere
8080/tcp                   ALLOW IN    Anywhere
8000                       ALLOW IN    Anywhere
8000 (v6)                  ALLOW IN    Anywhere (v6)

And here is an NMAP scan that says the ports are closed...

Starting Nmap 7.92 ( https://nmap.org ) at 2022-07-14 09:41 Eastern Daylight Time
Nmap scan report for netbox.top-floor.net (5.161.130.12)
Host is up (0.0057s latency).
rDNS record for 5.161.130.12: static.12.130.161.5.clients.your-server.de
Not shown: 65529 filtered tcp ports (no-response)
PORT     STATE  SERVICE
80/tcp   closed http
443/tcp  closed https
2230/tcp open   queueadm
8000/tcp closed http-alt
8080/tcp closed http-proxy
9090/tcp open   zeus-admin

Nmap done: 1 IP address (1 host up) scanned in 116.96 seconds

Anyways, here is what I did to try and restart nginx and see if the certs would work on the site:

sudo systemctl restart nginx
Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xe" for details.

Error-ed out. Then I checked it with:

USER@VPS:~$ sudo systemctl status nginx
● nginx.service - A high performance web server and a reverse proxy server
     Loaded: loaded (/lib/systemd/system/nginx.service; enabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Thu 2022-07-14 13:39:06 UTC; 6s ago
       Docs: man:nginx(8)
    Process: 18309 ExecStartPre=/usr/sbin/nginx -t -q -g daemon on; master_process on; (code=exited, status=0/SUCCESS)
    Process: 18321 ExecStart=/usr/sbin/nginx -g daemon on; master_process on; (code=exited, status=1/FAILURE)

Jul 14 13:39:04 VPS systemd[1]: Starting A high performance web server and a reverse proxy server...
Jul 14 13:39:04 VPS nginx[18321]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jul 14 13:39:04 VPS nginx[18321]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jul 14 13:39:05 VPS nginx[18321]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jul 14 13:39:05 VPS nginx[18321]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jul 14 13:39:06 VPS nginx[18321]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)
Jul 14 13:39:06 VPS nginx[18321]: nginx: [emerg] still could not bind()
Jul 14 13:39:06 VPS systemd[1]: nginx.service: Control process exited, code=exited, status=1/FAILURE
Jul 14 13:39:06 VPS systemd[1]: nginx.service: Failed with result 'exit-code'.
Jul 14 13:39:06 VPS systemd[1]: Failed to start A high performance web server and a reverse proxy server.
USER@VPS:~$

This lead me down a rabbit hole of trying to figure out why port 80 might already be in use. Because I am pretty sure its not. I found a StackOverflow guide walking me through why it might not be working. But the root cause for that was Apache web server running. Which my VPS does not have installed/configured. As you can see from the netstat output port 80 or 443 does not appear to be in use.

sudo netstat -tulpn
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:25              0.0.0.0:*               LISTEN      2067/master
tcp        0      0 127.0.0.1:8001          0.0.0.0:*               LISTEN      879/python3
tcp        0      0 127.0.0.1:6379          0.0.0.0:*               LISTEN      983/redis-server 12
tcp        0      0 192.168.122.1:53        0.0.0.0:*               LISTEN      1256/dnsmasq
tcp        0      0 127.0.0.53:53           0.0.0.0:*               LISTEN      782/systemd-resolve
tcp        0      0 0.0.0.0:2230            0.0.0.0:*               LISTEN      937/sshd: /usr/sbin
tcp        0      0 127.0.0.1:5432          0.0.0.0:*               LISTEN      998/postgres
tcp6       0      0 :::25                   :::*                    LISTEN      2067/master
tcp6       0      0 :::9090                 :::*                    LISTEN      1/init
tcp6       0      0 ::1:6379                :::*                    LISTEN      983/redis-server 12
tcp6       0      0 :::2230                 :::*                    LISTEN      937/sshd: /usr/sbin
tcp6       0      0 ::1:5432                :::*                    LISTEN      998/postgres
udp        0      0 192.168.122.1:53        0.0.0.0:*                           1256/dnsmasq
udp        0      0 127.0.0.53:53           0.0.0.0:*                           782/systemd-resolve
udp        0      0 0.0.0.0:67              0.0.0.0:*                           1256/dnsmasq
udp        0      0 5.161.130.12:68         0.0.0.0:*                           780/systemd-network

So I am kind of at a standstill. But this seems more like an issues I have with nginx and not LE. So I am not sure if this forum is still the right place for help. But I was atleast able to use webroot to get a cert!

[MikeMcQ]

That looks strange. It would be worth rebooting your server. We sometimes see that bind port 80 error when nginx has been started both with systemd and without it. They then fight over the port in peculiar ways. The nginx plug-in can actually cause that in very specific rare situations.

I don't see another nginx running either but something seems stuck.

[rg305]

@b-fraley

The error seems to indicate that port 80 is already in use.

@b-fraley

Jul 14 13:39:04 VPS nginx[18321]: nginx: [emerg] bind() to [::]:80 failed (98: Address already in use)

Yes, it does.

If you haven't rebooted, please show:
ps -ef | grep -Ei 'apache|http|nginx|java' | grep -v grep