Failed to bind to :443 using IPv4

Help
1

I switched to letsencrypt 2 months ago and had no issues with the verification process to obtain a cert. I already have an apache server running but it does not listen on port 80 nor 443. So i used --standalone and it worked great.

I’m running Fedora 27.

Now 2 months later it is time to renew and it failed. the cron job ran as schedule but the standalone server failed to bind to port 443.
remember that my apache server doesn’t use 80/443 and the port is not in use

[root@xxxxxx ~]# netstat -an | grep 443
[root@xxxxxx ~]# netstat -an | grep 80
unix 2 [ ACC ] STREAM LISTENING 18043 /var/run/abrt/abrt.socket

the only message in the log is this:
Failed to bind to :443 using IPv4

no detailed error as to why it failed to bind. a rather unhelpful message and
adding --verbose didn’t improve it either.

selinux is enforcing and there are no audit messages about blocking access to 443.

no other errors in messages or other log files as to what the problem might be.

the command i’m using is:
certbot renew --standalone --verbose --preferred-challenges tls-sni --deploy-hook /etc/letsencrypt/deploy-hook-script --post-hook /etc/letsencrypt/post-hook-script

so how can i debug the standalone server to find out why it can’t bind to :443?

relavent messages in the log:
Single candidate plugin: * standalone
Description: Spin up a temporary webserver
Interfaces: IAuthenticator, IPlugin
Entry point: standalone = certbot.plugins.standalone:Authenticator
Initialized: <certbot.plugins.standalone.Authenticator object at 0x7f3c6fd59f28>
Prep: True
Selected authenticator <certbot.plugins.standalone.Authenticator object at 0x7f3c6fd59f28> and installer None
Plugins selected: Authenticator standalone, Installer None
.
.
Renewing an existing certificate
.
.
Performing the following challenges:
tls-sni-01 challenge for markpreston.me
Failed to bind to :443 using IPv4
Waiting for verification…

and then i get the expected connection refused since it isn’t listening on 443.

I’m not looking for a workaround because i’m not going to reconfigure my apache to listen on 80/443 just for cert renewals. It worked 2 months ago in standalone mode, it should work again. I do keep my fedora system up to date with the latest rpm/patches.

so does anyone have any ideals how to put the standalone webserver in debug mode? is there a config file for it? if not there should be.

cheers.

My domain is: markpreston.me

I ran this command: see above

It produced this output: see above

My web server is (include version):

The operating system my web server runs on is (include version):fedora 27

My hosting provider, if applicable, is: my server

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):no

3

Performing the following challenges:
http-01 challenge for markpreston.me
Failed to bind to :80 using IPv4

4

Please post your /var/log/letsencrypt/letsencrypt.log file. It should provide more information about the failure.

5

i didn’t see anything helpful in the logs which was why i was asking how to put the standalone server in debug mode but just in case i missed something, there is the log:
https://pastebin.com/RTtysuVW

6

I seem to remember someone mentioning that that “Failed to bind” message can sometimes be misleading. Can’t find it now but I see this comment in the source:

“Ubuntu, for example, will fail to bind to IPv4 if we’ve already bound to IPv6. But that’s ok, since it will accept IPv4 connections on the IPv6 socket.”

Perhaps Fedora behaves the same way? If so the problem may be something else entirely, such as a router or firewall issue.

1 Like
7

not a router or firewall issue. i’ve shutdow the firewall and makes no different. i see no indication that it was able to bind to the port it is suppose to and until it does it will fail to authenticate.

8

I think it's likely it did bind to the port; if it failed to bind to IPv6 I'd expect it to log a separate error message about that as well. At least that's what happens on Ubuntu.

9

There's a --debug-challenges option that will make certbot pause after starting up the standalone server but before asking the CA to connect to it; you could use that and manually check if it's listening on the port. (I would have mentioned this earlier but I couldn't remember what the option was called, sorry)

2 Likes
10

what’s in /etc/letsencrypt/deploy-hook-script and /etc/letsencrypt/post-hook-script ? any chance you got them reversed and a command in there is binding to 80/443, then unbinding?

11

good question. I’ve tried running the renewal without --deploy-hook and --post-hook and i get the same error. all that is in the deploy-hook is copying the certs to the location my server expects and setting the correct owner/permissions on the certs. the post-hook just sends an email.

the debug-challenge i’ve dtried and it doesn’t pause for me. it just does the same thing with a few more debug messages during the challenge phase which doesn’t help me because there isn’t anything listening on the expected port.

12

i just tried again with this command:
certbot renew --standalone --debug-challenges --verbose --preferred-challenges tls-sni

it does not pause but this time i did notice more messages and trace stack calls. I pasted the log here: https://pastebin.com/MjfjUbj2

i’m still looking though it for anything helpful as to why it fails to bind.

13

Please capture a network system call trace of certbot running and pastebin the output:

strace -e trace=network certbot renew >& strace.txt

This dumps the trace to a file so it doesn’t get mixed with certbot’s normal output. You may need to install the strace package if it is not already installed on your system.

14

thanks for that, i had thought about using strace but didn’t know the best option to use when running it.
here is the output
https://pastebin.com/57ksj6Pw

i see where it finds the correct interface/IP but then skips over it instead of using it.

getsockname(7, {sa_family=AF_INET, sin_port=htons(53103), sin_addr=inet_addr(“99.28.54.121”)}, [28->16]) = 0

it should have used this but skipped over it instead…

15

now when i do a netstat -an i see :443 is in use, looking to see what has that and stop it. this didn’t show before today.

16

clean test this time. something added port 443 to my apache config and i removed it and restarted it. nothing on 443 now

[root@markpreston letsencrypt]# netstat -an | grep 80
unix 2 [ ACC ] STREAM LISTENING 238096 /run/httpd/cgisock.9056
unix 2 [ ACC ] STREAM LISTENING 18043 /var/run/abrt/abrt.socket
unix 2 [ ] DGRAM 17804 /var/run/chrony/chronyd.sock
unix 3 [ ] STREAM CONNECTED 18045 /run/dbus/system_bus_socket
unix 3 [ ] STREAM CONNECTED 25180
unix 3 [ ] STREAM CONNECTED 19580 /run/dbus/system_bus_socket
[root@markpreston letsencrypt]# netstat -an | grep 443
[root@markpreston letsencrypt]# strace -e trace=network certbot renew >& strace.txt
[root@markpreston letsencrypt]# netstat -an | grep 443

a very different trace. so what interface is it binding to? it should be sin_addr=inet_addr(“99.28.54.121”)

any ideal why there is a EADDRINUSE (Address already in use) ???

https://pastebin.com/w1QuCMqA

17 
bind(8, {sa_family=AF_INET6, sin6_port=htons(443), inet_pton(AF_INET6, "::", &sin6_addr), sin6_flowinfo=htonl(0), sin6_scope_id=0}, 28) = 0
bind(9, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("0.0.0.0")}, 16) = -1 EADDRINUSE (Address already in use)

The IPv6 port binding is successful, but the IPv4 port binding fails because the port is already in use. This is exactly the behavior @jmorahan describes.

(The sockets are bound to :: and 0.0.0.0 which makes them listen on all available IP addresses and not any specific ones. The bindings that listed your IP specifically pertained to client connections to the ACME server.)

You seem to not mind reading straces, so maybe you won’t mind tweaking certbot’s code just a little bit to double-check?

Locate this line (highlighted in context) in /usr/lib/python3.6/site-packages/acme/standalone.py on your system:

        for ip_version in [True, False]:

and remove the True, so it just reads:

        for ip_version in [False]:

Then try running certbot again. It won’t try and use IPv6 at all now and you shouldn’t see any port binding error. Let’s see if that makes a difference.

You can restore the old behavior by readding True, or running dnf reinstall python3-acme. (It will also get written over on updates.)

1 Like
18

i changed the line from
for ip_version in [True, False]:
to
for ip_version in [False]:

and the behavior changed. it doesn’t throw an error but it still doesn’t seem to bind. to make sure there isn’t a firewall issue i even shutdown my firewall before running certbot.
log output is in: https://pastebin.com/BKfPNJnq

19

Aha, seems--debug-challenges doesn't work with certbot renew, perhaps because renew takes great care to run non-interactively. Could you try it with certbot certonly or certbot run?

Or even just try configuring apache to listen on port 80/443 and confirm that you can connect to it externally.

2 Likes
20 
bind(8, {sa_family=AF_INET, sin_port=htons(443), sin_addr=inet_addr("0.0.0.0")}, 16) = 0

It is binding. Whatever is blocking it occurs at a higher level.

Instead of a full http server you can also test with a long running Python server much like the one certbot standalone mode uses:

mkdir /var/local/webroot
cd /var/local/webroot
echo 'Hello, world!' > index.html
python3 -m http.server 80 &
server_pid=$!
certbot --webroot -w /var/local/webroot -d yourdomain.com
netstat -tln
[...anything else you can think of to debug the connection...]
kill $server_pid
21

Hi,

Thanks for the test code on launching the mini-server. That allowed me to trace the blockage which turned out to be my router after all. I completely forgot that i close all ports but one more than a month ago. It sucks badly to get old and forgetful. so i’ve opened 80/443 and the renewal worked as advertised. I was sure it couldn’t have been my router but that test proved me wrong. Hopefully this topic will be help to others that need a small test case for standalone.

thanks muchly.

2 Likes