Problems with haproxy + letsencrypt


#1

Hi,
Apologies if this has already been asked elsewhere, but my search for answers is still unsuccessful.

In our use case, we have an Ubuntu haproxy server which hosts multiple vhosts, for example, jenkins.projectdev.com, gitlab.projectdev.com, nagios.projectdev.com. That is, one external IP address, with multiple DNS A-RECORDS for the multiple vhosts which point towards that same haproxy server. That haproxy server listens on 80 and 443, but automatically redirects all HTTP to HTTPS. All pretty simple.

Ideally, I’d like to install and renew certificates (through automation) without having to bring haproxy offline temporarily. I have configured the frontend and backends as follows:

` global
  chroot  /var/lib/haproxy
  group  haproxy
  log  127.0.0.1 local0 err
  log  127.0.0.1 local2 notice
  log  127.0.0.1 local3

  maxconn  4000
  pidfile  /var/run/haproxy.pid
  stats  socket /var/lib/haproxy/stats level admin
  user  haproxy

defaults
  log  global
  maxconn  4000
  retries  3
  timeout  http-request 10s
  timeout  queue 1m
  timeout  connect 10s
  timeout  client 1m
  timeout  server 1m
  timeout  check 10s

frontend frontend00
  bind *:443 ssl crt /etc/haproxy/frontend00.local.pem
  bind *:80
  mode http
  acl destination_jenkins-backend00 hdr_beg(host) -i jenkins
  acl destination_gitlab-backend00 hdr_beg(host) -i gitlab
  acl destination_nagios-backend00 hdr_beg(host) -i nagios
  acl destination_letsencrypt-backend00 path_beg /.well-known/acme-challenge/
  http-response replace-value Location http://(.*) https://\1
  option httplog
  redirect scheme https code 301 if !{ ssl_fc }

  use_backend jenkins-backend00 if destination_jenkins-backend00
  use_backend gitlab-backend00 if destination_gitlab-backend00
  use_backend nagios-backend00 if destination_gitlab-backend00
  use_backend letsencrypt-backend00 if destination_letsencrypt-backend00

listen server-stats-listen00
  bind *:8888
  mode http
  stats enable
  stats uri /

backend gitlab-backend00
  mode http
  option forwardfor
  option httplog
  stats enable
  server gitlab gitlab:80 check inter 12s

backend jenkins-backend00
  mode http
  option forwardfor
  option httplog
  stats enable
  server jenkins jenkins:8080 check inter 12s

backend nagios-backend00
  mode http
  option forwardfor
  option httplog
  stats enable
  server nagios nagios:8080 check inter 12s

backend letsencrypt-backend00
  mode http
  option forwardfor
  option httplog
  stats enable
  server localserverletsencrypt 127.0.0.1:6443 check inter 5s`

So, I then try to run:
sudo ./letsencrypt/letsencrypt-auto certonly -d gitlab.projectdev.com -d jenkins.projectdev.com -d nagios.projectdev.com --renew-by-default --standalone-supported-challenges tls-sni-01 --agree-tos --tls-sni-01-port=6443

Multiple problems here:

  1. haproxy does not detect that letsencrypt-auto backend service comes up in time for the request coming in from the letsencrypt-auto server - haproxy returns a 503. I can get around this by running netcat in listen mode on the same port for 10 seconds prior to running letsencrypt, tricking haproxy into detecting the backend as alive.
  2. I then had to recreate the existing haproxy self-signed certificate to include the multiple SANs (using instructions here http://stackoverflow.com/questions/21488845/how-can-i-generate-a-self-signed-certificate-with-subjectaltname-using-openssl) however letsencrypt does not accept the multiple SANs as valid.

I could probably resolve #2 by putting an HTTP->HTTPS exception in place for the ‘path_beg /.well-known/acme-challenge/’, however I’m not sure what other security implications this may have, so I’d prefer not to. The only way I can think of solving #1 is by stopping haproxy, letting lets-encrypt take its port directly, then restarting it after a successful certificate installation - however this will interrupt service…

How have other people solved these challenges?


#2

Did you try using http-01 instead of tls-sni-01?


#3

You don’t have to make the letsencrypt backend “checked”. Just remove the “check inter 5s”.

But more importantly, are you confusing http-01 and tls-sni-01? With tls-sni-01 the path /.well-known/acme-challenge/ isn’t even relevant AFAIK. That is only for the http-01 challenge.

To support tls-sni-01 you need to provide a tcp frontend that looks for the SNI and directs *.acme.invalid to the letsencrypt backend on port 6443 and all other SNI to a loopback backend that connects to the http mode frontend that terminates TLS.

For example, here are excerpts from my private playground config, i.e. nothing productive:

frontend f.tcp:443
        bind [...]:443
        mode tcp
        option tcplog

        # place hosts here that require a more compatible cipher/protocol set
        acl sni.compat req.ssl_sni -i [...]

        acl sni.le     req.ssl_sni -m end .acme.invalid
        acl sni        req.ssl_sni -m found
        acl tls        req.ssl_hello_type 1

        tcp-request inspect-delay 5s
        tcp-request content accept if tls

        default_backend b.close

        # all TLS with SNI defined is sent to the HTTPS backend
        # the HTTPS backend needs strict-sni to prevent it from
        # presenting the first loaded cert to unknown SNI requests
        # however, this saves us the duplicate definition of all
        # known SNI in this frontend
        # TLS with no SNI at all can be sent elsewhere, currently unused

        use_backend b.le:443       if sni.le
        use_backend b.https.compat if sni.compat
        use_backend b.https.secure if sni
        #use_backend ...            if tls

frontend f.https
        bind /var/run/https.secure.sock ssl accept-proxy alpn http/1.1 ciphers DHE+AESGCM:DHE+CHACHA20:ECDHE+AESGCM:ECDHE+CHACHA20:!aNULL:!SHA1:!MD5:!DSS             crt . ecdhe secp521r1 strict-sni no-tlsv10 no-tlsv11
        bind /var/run/https.compat.sock ssl accept-proxy alpn http/1.1 ciphers ECDHE+AESGCM:ECDHE+CHACHA20:ECDHE+AES:DHE+AESGCM:DHE+CHACHA20:DHE+AES:!aNULL:!MD5:!DSS crt . ecdhe secp384r1 strict-sni
        mode http
        option httplog

        # catch all SNI (from loaded certs) without a backend
        # SNI that doesn't match any cert is rejected at the TLS level
        default_backend b.deny

        # require SNI and Host: to match
        use_backend [...] if { ssl_fc_sni -i [...] } { req.hdr(Host) -i [...] }

backend b.https.secure
        mode tcp
        option tcplog
        server s.https /var/run/https.secure.sock send-proxy

backend b.https.compat
        mode tcp
        option tcplog
        server s.https /var/run/https.compat.sock send-proxy

backend b.le:443
        mode tcp
        option tcplog
        server s.le ::1:443