I think this means that there is a proxy (either a copy of nginx or some other proxy software) and that your procedure for stopping organizr is not stopping the software that actually listens to port 80. The expected result from the curl command if port 80 is genuinely free is curl: (7) Failed to connect to bereskapi-ha.duckdns.org port 80: Connection refused, not a 404 error. (The 404 error is actively sent by a web server of some kind that is listening on port 80.)
1 Like
thank you
I have the same results Server: lighttpd/1.4.39
but I don’t have it on my machine!
pi@bereskapi-ha:~ ps aux | grep lighttpd
pi 8187 0.0 0.0 5992 572 pts/0 S+ 14:50 0:00 grep --color=auto lighttpd
pi@bereskapi-ha:~ lighttpd -version
-bash: lighttpd: команда не найдена
is it some sort of malware running undercover?
how can I detect and kill it?
I would check the NAT rules.
Perhaps port 80 is forwarded to another internal IP - or the firewall itself is responding on port 80.
1 Like
thank you
do you mean on mu router?
my router has a public ip 193.105.59.205
my rpi has an internal ip 192.168.2.230 on eth0
ports 80 and 443 are forwarded to 192.168.2.230 only
here is nmap results on this machine:
pi@bereskapi-ha:~ $ nmap localhost
Starting Nmap 7.40 ( https://nmap.org ) at 2020-02-12 17:15 MSK
Nmap scan report for localhost (127.0.0.1)
Host is up (0.00049s latency).
Other addresses for localhost (not scanned): ::1
Not shown: 994 closed ports
PORT STATE SERVICE
80/tcp open http
443/tcp open https
3000/tcp open ppp
8080/tcp open http-proxy
8086/tcp open d-s-n
9000/tcp open cslistener
Nmap done: 1 IP address (1 host up) scanned in 0.37 seconds
What else can I check?
1 Like
You could run sudo ss -tlp to find out what program is listening on port 80.
1 Like
thank you
here is the result with organizr container running and after stopping it:
pi@bereskapi-ha:~ sudo ss -tlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 *:2224 *:* users:(("sshd",pid=590,fd=3))
LISTEN 0 128 *:8123 *:* users:(("python3",pid=19451,fd=44))
LISTEN 0 128 :::9000 :::* users:(("docker-proxy",pid=2468,fd=4))
LISTEN 0 128 :::http :::* users:(("docker-proxy",pid=18903,fd=4))
LISTEN 0 128 :::http-alt :::* users:(("docker-proxy",pid=1376,fd=4))
LISTEN 0 128 :::2224 :::* users:(("sshd",pid=590,fd=4))
LISTEN 0 128 :::8086 :::* users:(("docker-proxy",pid=1421,fd=4))
LISTEN 0 128 :::1880 :::* users:(("docker-proxy",pid=2997,fd=4))
LISTEN 0 128 :::3000 :::* users:(("docker-proxy",pid=1248,fd=4))
LISTEN 0 128 :::https :::* users:(("docker-proxy",pid=18889,fd=4))
LISTEN 0 128 :::1883 :::* users:(("docker-proxy",pid=1390,fd=4))
LISTEN 0 128 :::8126 :::* users:(("docker-proxy",pid=1461,fd=4))
pi@bereskapi-ha:~ docker stop organizr
organizr
pi@bereskapi-ha:~ $ sudo ss -tlp
State Recv-Q Send-Q Local Address:Port Peer Address:Port
LISTEN 0 128 :2224 : users:((“sshd”,pid=590,fd=3))
LISTEN 0 128 :8123 : users:((“python3”,pid=19451,fd=44))
LISTEN 0 128 :::9000 ::: users:((“docker-proxy”,pid=2468,fd=4))
LISTEN 0 128 :::http-alt ::: users:((“docker-proxy”,pid=1376,fd=4))
LISTEN 0 128 :::2224 :::* users:((“sshd”,pid=590,fd=4))
LISTEN 0 128 :::8086 :::* users:((“docker-proxy”,pid=1421,fd=4))
LISTEN 0 128 :::1880 :::* users:((“docker-proxy”,pid=2997,fd=4))
LISTEN 0 128 :::3000 :::* users:((“docker-proxy”,pid=1248,fd=4))
LISTEN 0 128 :::1883 :::* users:((“docker-proxy”,pid=1390,fd=4))
LISTEN 0 128 :::8126 :::* users:((“docker-proxy”,pid=1461,fd=4))
sorry fo the format
but as far I can tell, NOTHING is running on port 80 after stopping organizr container
here is the list of all containers running
port 808 is used by mqtt-bridge
pi@bereskapi-ha:/opt $ docker-compose ps
Name Command State Ports
dockermon /bin/sh -c npm start Up 0.0.0.0:8126->8126/tcp
grafana /run.sh Up 0.0.0.0:3000->3000/tcp
homeassistant /bin/entry.sh python3 -m h … Up (healthy)
influxdb /entrypoint.sh influxd Up (healthy) 0.0.0.0:8086->8086/tcp
mosquitto /docker-entrypoint.sh /usr … Up 0.0.0.0:1883->1883/tcp
mqtt-bridge npm start Up 0.0.0.0:8080->8080/tcp
node-red /usr/bin/entry.sh npm star … Up (healthy) 0.0.0.0:1880->1880/tcp
organizr /init Up (healthy) 0.0.0.0:443->443/tcp, 0.0.0.0:80->80/tcp
portainer /portainer Up 0.0.0.0:9000->9000/tcp
1 Like
I think I found this lighttpd
It is running on my asus router rt-ac87u
bereska87u@RT-AC87U:/tmp/home/root# ps | grep lighttpd
18973 bereska8 1532 S grep lighttpd
21209 bereska8 12220 S /usr/sbin/lighttpd -f /tmp/.le/lighttpd.conf
bereska87u@RT-AC87U:/tmp/home/root# lighttpd -version
lighttpd/1.4.39 (ssl) - a light and fast webserver
Build-Date: May 24 2019 17:55:57
Can i kill it without messing up my router? Why is it such a problem now? Like I said I had never any problem with renewing letsencrypt certs before. I bought this router 3 years ago
I would just change the port from :80 to :XYZ
How is it forwarded to your server? Let’s Encrypt is always going to try to connect to this port from the Internet as part of the validation process.
But 80 is also used by the router (doing the forwarding) ...
gentlemen, I have great news! after updating the firmware on my router lighttpd is NOT running there anymore and I have just successfully renewed ALL certs under 30 sec!
Who knows that damn lighttpd got on my router since I only use stock firmware but I am happy the problem has been solved.
I don’t think it would be possible without your help
Thank you very much
2 Likes
I have one more question before we close this issue.
While looking for solutions I deleted btcpay.bereskapi-ha.duckdns.org directories:
rm -rf /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org
rm -rf /etc/letsencrypt/archive/btcpay.bereskapi-ha.duckdns.org
Renewing all certs letsencrypt complained about this missing subdomain. So I created a new cert for this subdomain:
./certbot-auto certonly --standalone --preferred-challenges http-01 --email your@email.address -d btcpay.bereskapi-ha.duckdns.org
And renew again with this error:
Processing /etc/letsencrypt/renewal/btcpay.bereskapi-ha.duckdns.org.conf
Traceback (most recent call last):
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/renewal.py”, line 63, in _reconstitute
renewal_candidate = storage.RenewableCert(full_path, config)
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/storage.py”, line 465, in init
self._check_symlinks()
File “/opt/eff.org/certbot/venv/local/lib/python2.7/site-packages/certbot/_internal/storage.py”, line 532, in _check_symlinks
“expected {0} to be a symlink”.format(link))
CertStorageError: expected /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org/cert.pem to be a symlink
Renewal configuration file /etc/letsencrypt/renewal/btcpay.bereskapi-ha.duckdns.org.conf is broken. Skipping.
Finally ending up with the following certs:
The following certs are not due for renewal yet:
/etc/letsencrypt/live/bereskapi-ha.duckdns.org/fullchain.pem expires on 2020-05-12 (skipped)
/etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org-0001/fullchain.pem expires on 2020-05-12 (skipped)
/etc/letsencrypt/live/ha.bereskapi-ha.duckdns.org/fullchain.pem expires on 2020-05-12 (skipped)
/etc/letsencrypt/live/nodered.bereskapi-ha.duckdns.org/fullchain.pem expires on 2020-05-12 (skipped)
/etc/letsencrypt/live/picamera.bereskapi-ha.duckdns.org/fullchain.pem expires on 2020-03-24 (skipped)
/etc/letsencrypt/live/portainer.bereskapi-ha.duckdns.org/fullchain.pem expires on 2020-05-12 (skipped)
So the question is what can I do to run certbot-auto error free now?
1 Like
You should also delete /etc/letsencrypt/renewal/bereskapi-ha.duckdns.org.conf. Normally, you should avoid renaming or deleting files in /etc/letsencrypt yourself (use certbot delete instead) because these files refer to one another.
1 Like
The proper way to delete a cert is by using:
certbot delete
Now certbot my still think it has that cert while all the files have been deleted.
start with:
certbot certificates
[let's see what that says]
1 Like
here it is:
pi@bereskapi-ha:~ $ ./certbot-auto certificates
Requesting to rerun ./certbot-auto with root privileges…
./certbot-auto has insecure permissions!
To learn how to fix them, visit Certbot-auto deployment best practices
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Renewal configuration file /etc/letsencrypt/renewal/btcpay.bereskapi-ha.duckdns.org.conf produced an unexpected error: expected /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org/cert.pem to be a symlink. Skipping.
Found the following certs:
Certificate Name: bereskapi-ha.duckdns.org
Domains: bereskapi-ha.duckdns.org
Expiry Date: 2020-05-12 21:02:06+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/bereskapi-ha.duckdns.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/bereskapi-ha.duckdns.org/privkey.pem
Certificate Name: btcpay.bereskapi-ha.duckdns.org-0001
Domains: btcpay.bereskapi-ha.duckdns.org
Expiry Date: 2020-05-12 21:05:54+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org-0001/fullchain.pem
Private Key Path: /etc/letsencrypt/live/btcpay.bereskapi-ha.duckdns.org-0001/privkey.pem
Certificate Name: ha.bereskapi-ha.duckdns.org
Domains: ha.bereskapi-ha.duckdns.org
Expiry Date: 2020-05-12 21:02:15+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/ha.bereskapi-ha.duckdns.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/ha.bereskapi-ha.duckdns.org/privkey.pem
Certificate Name: nodered.bereskapi-ha.duckdns.org
Domains: nodered.bereskapi-ha.duckdns.org
Expiry Date: 2020-05-12 21:02:24+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/nodered.bereskapi-ha.duckdns.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/nodered.bereskapi-ha.duckdns.org/privkey.pem
Certificate Name: picamera.bereskapi-ha.duckdns.org
Domains: picamera.bereskapi-ha.duckdns.org
Expiry Date: 2020-03-24 14:01:57+00:00 (VALID: 40 days)
Certificate Path: /etc/letsencrypt/live/picamera.bereskapi-ha.duckdns.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/picamera.bereskapi-ha.duckdns.org/privkey.pem
Certificate Name: portainer.bereskapi-ha.duckdns.org
Domains: portainer.bereskapi-ha.duckdns.org
Expiry Date: 2020-05-12 21:02:31+00:00 (VALID: 89 days)
Certificate Path: /etc/letsencrypt/live/portainer.bereskapi-ha.duckdns.org/fullchain.pem
Private Key Path: /etc/letsencrypt/live/portainer.bereskapi-ha.duckdns.org/privkey.pem
The following renewal configurations were invalid:
/etc/letsencrypt/renewal/btcpay.bereskapi-ha.duckdns.org.conf
1 Like
Yeah, certbot doesn't know anything about the files you deleted.
[or you had multiple certs for that same name]
Try:
certbot delete --cert-name btcpay.bereskapi-ha.duckdns.org-0001
Then retry:
That might not be a good idea because that will delete a currently working one.
Normally I would bawk at attempting to fix something by promoting:
"Two wrongs to make a right"
But since it has already been wronged (once).
As shown by:
and:
I can only think to wipe it clean and start fresh.
[trying to fix this may take much longer]
On second thought... why waste a perfectly good "opportunity"?
FEATURE REQUEST:
certbot repair
1 Like
I think that deleting the one remaining reference to the partially-deleted certificate should be all right; it will come very close to the behavior that certbot delete would have done.
Yeah! There is certbot update_symlinks (which can fix some kinds of problems due to messing around with /etc/letsencrypt contents), but I think it can always be improved.
1 Like