Create & renew certs behind HAProxy reverse proxy help

Help
1

Right now I am running Ubuntu 16.04 for my servers, and I have 2 web servers (one LAMP one LEMP) behind an HAProxy reverse proxy, which is doing SSL Passthrough. When I run certbot for auto renewal or even doing a cert-only run the service has troubles seeing my domain names and renewing my cert. A while ago I got the cert to renew on my LAMP server, however on my LEMP server I can’t get things to work. I unfortunately let my certificate expire and after that I can’t get any connection to the server to work, even for renewal. I have my reverse proxy directing all traffic to port 443 by default.

Here is the command I use:
letsencrypt certonly --webroot -w /var/www/html/ -d mydomain.com
Here is the error I get:
Failed authorization procedure. mydomain.com (http-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Could not connect to mydomain.com

These are separate VMs running these web servers. Would it be easier to just scrap all of this and do SSL termination on the reverse proxy instead? Down the line I plan to have 3 domains on a single dynamic IP (using duckdns for my dynamic DNS)

2

When you say you’re directing all traffic to port 443, does that include traffic that arrives on your HAproxy server on port 80? Or is there no listener on port 80 at all?

The validation mechanism you’re using (http-01) uses HTTP on port 80, so you will definitely need to be serving HTTP on that port. There’s no way to force the first request to be made via HTTPS on a different port. A rewrite to https:// (using HTTP Status Code 301 or 302) would be fine and the validation server would follow that redirect (which should then hit your LEMP server). The certificate used in that connection may be expired and/or self-signed, that won’t be a problem.

3

I have port 80 and 443 listening on both HAProxy and my internal servers. However, the reverse proxy does redirect using status code 301 as well as my nginx server. My Apache server (which I have gotten a cert with earlier, but doesn’t work with “letsencrypt renew”) just rewrites the URL to https://{SERVER_NAME} when it gets a port 80 request.

Since HAProxy is making the first response to my LEMP server be on port 443, would that be the cause of my certificate receiving failures?

4

Also one thing to add: I can get a successful curl on port 80 when I curl my IP address, but a curl command on port 443 gets me this error:

curl: (52) Empty reply from server.

So I’m wondering if there’s some sort of HAProxy error all of a sudden? I can connect to my LAMP server just fine

5

Some actual configs might help. Who knows.

6

I was going to get around to that, here are my HAProxy configs:

global
    log /dev/log    local0
    log /dev/log    local1 notice
    chroot /var/lib/haproxy
    stats socket /run/haproxy/admin.sock mode 660 level admin
    stats timeout 30s
    maxconn 4096
    user haproxy
    group haproxy
    daemon


defaults
    log     global
    mode    tcp
    option  tcplog
    option  dontlognull
    timeout connect 15s
    timeout client  15s
    timeout server  15s
    errorfile 400 /etc/haproxy/errors/400.http
    errorfile 403 /etc/haproxy/errors/403.http
    errorfile 408 /etc/haproxy/errors/408.http
    errorfile 500 /etc/haproxy/errors/500.http
    errorfile 502 /etc/haproxy/errors/502.http
    errorfile 503 /etc/haproxy/errors/503.http
    errorfile 504 /etc/haproxy/errors/504.http


frontend localhost80
    bind *:80
    mode http
    redirect scheme https code 301 if !{ ssl_fc }

frontend localhost443
    bind *:443
    option tcplog
    mode tcp

    acl tls req.ssl_hello_type 1

    tcp-request inspect-delay 15s
    tcp-request content accept if tls

    acl is_wordpress req.ssl_sni -i mydomain1.com   #10.0.0.165
    acl is_nextcloud req.ssl_sni -i www.mydomain2.com                #10.0.0.160
    acl is_nextcloud2 req.ssl_sni -i mydomain2.com

    use_backend nextcloud_cluster if is_nextcloud
    use_backend nextcloud_cluster if is_nextcloud2
    use_backend wordpress_cluster if is_wordpress


backend wordpress_cluster
    mode tcp

    option ssl-hello-chk

    server is_wordpress 10.0.0.165:443 check


backend nextcloud_cluster
    mode tcp

    option ssl-hello-chk

    server is_nextcloud 10.0.0.160:443 check

Here are the configs for my LEMP server:

server_tokens off;

server {
    listen 80;

    #error_page 401 403 40 /404.html;

    root /var/www/html;
    index index.php index.html index.htm;

    server_name mydomain1.com;
    return 301 https://$server_name$request_uri;
}

server {
    listen 443 ssl;

    error_page 401 403 404 /404.html;

    root /var/www/html;
    index index.php index.html index.htm;

    server_name mydomain1.com;

    #ssl on;
    #ssl_certificate /etc/letsencrypt/live/mydomain1.com/fullchain.pem;
    #ssl_certificate_key /etc/letsencrypt/live/mydomain1.com/privkey.pem;



    ssl_session_timeout 1d;
    ssl_session_cache shared:SSL:50m;
    ssl_session_tickets off;

    ssl_protocols TLSv1 TLSv1.1 TLSv1.2;
    ssl_prefer_server_ciphers on;
    ssl_dhparam /etc/ssl/certs/dhparam.pem;
    ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';

    # OCSP Stapling ---
    # fetch OCSP records from URL in ssl_certificate and cache them
    ssl_stapling on;
    ssl_stapling_verify on;

    # HSTS (ngx_http_headers_module is required) (15768000 seconds = 6 months)
    add_header Strict-Transport-Security "max-age=15768000; includeSubdomains; preload";



    #location ~ /.well-known {
    #       allow all;
    #}

    location / {
            try_files $uri $uri/ /index.php?q=$uri&$args;
    }

    location ~ \.php$ {
            include /etc/nginx/fastcgi.conf;
            try_files $uri =404;
            fastcgi_pass unix:/var/run/php/php7.0-fpm.sock;
            fastcgi_index index.php;
            include fastcgi_params;
    }

    location = /favicon.ico {
            log_not_found off;
            access_log off;
    }

    location = /robots.txt {
            allow all;
            log_not_found off;
            access_log off;
    }

    location ~ /\. {
            deny all;
    }

    location ~* /(?:uploads|files)/.*\.php$ {
            deny all;
    }
}

In the Nginx config, I have the SSL on; and location of the certs commented out, otherwise I can’t start the web server.

7

I have reason to believe it is my reverse proxy causing the problems, however I don’t know what the fix is. Currently I can curl my IP address and domain names on port 80, any attempt on port 443 results in and empty reply from the server. However, doing a curl --url https://mydomain1.com works (which has a valid cert) and https://mydomain2.com is using snakeoil for the time being (none of this is currently production), so when I curl mydomain2.com with the --url arg, I get an “Issuer certificate is invalid.” as expected

8

What do both the web server's and HAProxy's logs say when you try to renew?

These two sentences contradict each other. Does port 443 work with curl or not? First you say it doesn't, then it does?

9

I think it was because the first time I was issuing curl incorrectly. It does work.

Turns out my HAProxy logs decided to stop working and any change I make to the haproxy.cfg file or haproxy.conf file for rsyslog is continuing to result in no logs or broken configurations. What unreliable software.

10

I have done some research and decided I’m going to switch HAProxy to do SSL Termination instead of passthrough. I think it will clear up a lot of complications I am having with Let’s Encrypt and I think some other internal network connection issues I am having. I’m going to have to research HAProxy logging more too because it turns out logging in HAProxy sucks

11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.