Connection error with HTTP challenges

[fstennet]

Hey folks,

I’m trying to get a certificate installed in my HAProxy server for the domain vpl.dynu.net but I’m getting a connection error:

~$ sudo certbot certonly --standalone --preferred-challenges http --http-01-port 80 -d vpl.dynu.net -
d www.vpl.dynu.net
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator standalone, Installer None
Obtaining a new certificate
Performing the following challenges:
http-01 challenge for vpl.dynu.net
http-01 challenge for www.vpl.dynu.net
Waiting for verification...
Cleaning up challenges
Failed authorization procedure. vpl.dynu.net (http-01): urn:acme:error:connection :: The server 
could not connect to the client to verify the domain :: Fetching http://vpl.dynu.net/.well-known/acme-
challenge/Kcr_gIz-GfAvaq6ZxSYIPcIe7hGmst9VpXoGP3P2hnc: Timeout

IMPORTANT NOTES:
 - The following errors were reported by the server:

  Domain: vpl.dynu.net
  Type:   connection
   Detail: Fetching
  http://vpl.dynu.net/.well-known/acme-challenge/Kcr_gIz-GfAvaq6ZxSYIPcIe7hGmst9VpXoGP3P2hnc:
   Timeout

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A/AAAA record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

I performed a capture and can see the SYN coming in but my server doesn’t respond with the SYN-ACK.

This might be a red herring but I ran netstat a few times during the cert attempts and found that IPv6 is who listens to port 80 and not IPv4.

BEFORE running certbot:

~$ netstat -lputn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

DURING certbot:

~$ netstat -lputn
(Not all processes could be identified, non-owned process info
 will not be shown, you would have to be root to see it all.)
Active Internet connections (only servers)
Proto Recv-Q Send-Q Local Address           Foreign Address         State       PID/Program name
tcp        0      0 0.0.0.0:22              0.0.0.0:*               LISTEN      -
tcp6       0      0 :::80                   :::*                    LISTEN      -
tcp6       0      0 :::22                   :::*                    LISTEN      -

See the log file attached.
letsencrypt.txt (17.9 KB)

Thank you.

[rg305]

@fstennet

tcp6 0 0 :::80

Should also include IPv4.
And since we see "addressUsed": "186.15.1.69" in the logs, then we must ensure IPv4 is being used.
I would try connecting to
http://vpl.dynu.net/
while certbot is running to make sure it is passing through HAProxy correctly.
or
telnet 127.0.0.1 80

[fstennet]

The thing is, I’m installing the cert inside the server running HAProxy. When I do this, I have to stop the HAProxy service so that certbot can bind/listen to port 80.

[_az]

If you are running haproxy then you will want it to proxy to the Let’s Encrypt client you are using, probably with something like an ACL in your frontends:

   acl letsencrypt-acl path_beg /.well-known/acme-challenge/
   use_backend letsencrypt-backend if letsencrypt-acl

and add a backend:

backend letsencrypt-backend
   server letsencrypt 127.0.0.1:54321

and get certbot to serve the challenge via :54321, whichever way that is done.

[fstennet]

HAProxy IS the Let’s Encrypt client.

[rg305]

Then it should probably “proxy” the /.well-known/acme-challenge/ requests to itself (127.0.0.1:80).

[_az]

@fstennet

HAProxy IS the Let’s Encrypt client.

Could you describe how you are wanting to run haproxy / certbot? I'm having a hard time wrapping my head around it.

In your original post you are using certbot standalone via :80.

If you are intending to run haproxy on :80, the only way to successfully run certbot in standalone mode would be to run its listener on an alternate port and proxy to the certbot listener via an ACL in HAProxy.

If you are using some kind of extension to HAProxy, such as GitHub - janeczku/haproxy-acme-validation-plugin: 🍀 Zero-downtime ACME / Let's Encrypt certificate issuing for HAProxy , then you need to follow the instructions for invoking Certbot according to that project.

If you're using the GitHub - greenhost/certbot-haproxy: HAProxy plugin for Let's Encrypt's Certbot plugin to certbot, then you need to use the proxy ACL as I described previously.

I don't see any successful scenario where you launch certbot in http-01 listener mode on :80 where haproxy is also already listening on :80.

[fstennet]

I should have been more clear, this statement:

@fstennet

HAProxy IS the Let’s Encrypt client.

Should read:

The server that runs HAProxy IS the Let's Encrypt client.

I don't run HAProxy and Certbot at the same time, I stop the HAProxy service and then run Certbot. Pretty much following this.

[_az]

I see.

That article only does stop+certbot+start to bootstrap the initial Let's Encrypt certificate, which is where you seem to be stuck. I would argue this step is unnecessary/overcomplicating things but we'll stick to it.

For automatic renewal, the article runs Certbot on :54321 and is proxied to by an ACL in HAProxy.

Back to your problem, we see in your logs:

2017-11-21 20:12:26,033:DEBUG:acme.standalone:Failed to bind to :80 using IPv4

So, this is an environmental issue, something like:

• Something is already bound on :80 when you try to run certbot (which you've already tried to exclude, but could you please run it as root: sudo ss -tlp)
• You don't have the capability to bind to :80, which would certainly be odd as root (get more details by getting netcat/nc and running sudo nc -vvv -l -p80)

[fstennet]

HAProxy nor Certbot running:

~$ sudo ss -tlp
State       Recv-Q Send-Q                               Local Address:Port                                                Peer Address:Port
LISTEN      0      128                                              *:ssh                                                            *:*                     users:(("sshd",pid=1232,fd=3))
LISTEN      0      128                                             :::ssh                                                           :::*                     users:(("sshd",pid=1232,fd=4))

HAProxy running:

~$ sudo ss -tlp
State       Recv-Q Send-Q              Local Address:Port                               Peer Address:Port
LISTEN      0      128                             *:http                                          *:*                     users:(("haproxy",pid=2619,fd=5))
LISTEN      0      128                             *:ssh                                           *:*                     users:(("sshd",pid=1232,fd=3))
LISTEN      0      128                             *:https                                         *:*                     users:(("haproxy",pid=2619,fd=6))
LISTEN      0      128                            :::ssh                                          :::*                     users:(("sshd",pid=1232,fd=4))

Certbot running:

~$ sudo ss -tlp
State       Recv-Q Send-Q              Local Address:Port                               Peer Address:Port
LISTEN      0      128                             *:ssh                                           *:*                     users:(("sshd",pid=1232,fd=3))
LISTEN      0      5                              :::http                                         :::*                     users:(("certbot",pid=2553,fd=8))
LISTEN      0      128                            :::ssh                                          :::*                     users:(("sshd",pid=1232,fd=4))

and it looks like sudo can bind to port 80:

~$ sudo nc -vvv -l -p80
Listening on [0.0.0.0] (family 0, port 80)
^C

:~$ sudo ss -tlp
State      Recv-Q Send-Q                                     Local Address:Port                                                      Peer Address:Port
LISTEN     0      1                                                      *:http                                                                 *:*                     users:(("nc",pid=5397,fd=3))
LISTEN     0      128                                                    *:ssh                                                                  *:*                     users:(("sshd",pid=1232,fd=3))
LISTEN     0      128                                                   :::ssh                                                                 :::*                     users:(("sshd",pid=1232,fd=4))

[_az]

Thanks. I have no idea why Certbot would not bind to IPv4 80 in this instance.

@bmw is this related to your explanation here ?.

[bmw]

Yes. Similar to what I said there, not explicitly binding on IPv4 probably isn’t the problem as most Linux systems redirect IPv4 traffic to an IPv6 port by default and Certbot tries to bind using IPv4 on the systems that don’t.

If you want though, you can use --http-01-address to set the address Certbot should listen on. You can give this flag either an IPv4 or an IPv6 interface. For example, you can use --http-01-address 0 or --http-01-address 0.0.0.0 to have Certbot listen using IPv4 on all interfaces.