[WIP] Certificate issuance & web service behind inacessible NAT/router/firewall

Hi. My name is Forest and I’m trying to build a software package which would make it easier for non-technical people to host things on the internet from their living room WITHOUT explicitly handing away the keys to the castle to any cloud providers. My strategy is:

  • Use AWS free tier or a similar subscription based service to get a non-email-blacklisted IP address with high uptime and point a DNS record at it.
  • Install OpenVPN on both the user’s web server and on the cloud instance, then vpn connect to the cloud instance from the web server.
  • Install HAProxy on the cloud instance and configure it in TCP mode with the user’s web server on the VPN as a backend.
  • Use Let’s Encrypt on the user’s web server to generate certificate(s).
  • Use a reverse proxy such as jwilder/nginx-proxy to terminate SSL on the user’s web-server and route requests to the appropriate service.
  • Special Bonus Points: Allow the user to run multiple backend servers and use Raft/Consul/Zookeeper to create a consensus on which one is the “leader”, then update the HAProxy configuration in the cloud to route traffic to the “leader”.

The end goal for user experience would be:

  1. install linux on web server.
  2. install name_of_sequentialread_easy_hosting_thing_here (lethasha? Lets Encrypt and Tunnel Highly Available Self Hosted Apps)
  3. Follow a guided process of signing up for an account with a Cloud service provider like Namecheap and AWS free tier.
  4. Decide on and purchase a domain name
  5. Pick from a list of services you want to run. Discourse, Email, OwnCloud, Ghost, Gogs, Jenkins, Mattermost, etc etc.
  6. Enjoy!

The security story would be (at least through my understanding) that someone with access to the cloud instance could theoretically use Let’s Encrypt to generate a new certificate and MITM your connections. But they wouldn’t be able to hide that from you, so you just need a small automated test that checks to ensure that the cloud instance is serving the right public key.

More information: https://sequentialread.com/pragmatic-path-towards-data-sovereignty-for-non-technical-users/

Ok, so now for my actual question. I have an alpha version of this setup running at home. In the past, I have generated Lets Encrypt certificates for my sub-domains by simply pointing DNS to my home IP address, forwarding ports 80 and 443 from the router I got from my ISP to my web server, then running:

# stop the existing webserver
docker-compose -f /compose/docker-compose.yml stop 
#use standalone certbot to generate certificates
./letsencrypt-auto certonly --standalone \
-d sequentialread.com \
-d git.sequentialread.com \
-d comments.sequentialread.com \
-d www.sequentialread.com \
-d www.git.sequentialread.com \
-d www.comments.sequentialread.com 

More information: https://sequentialread.com/how-to-letsencrypt-docker-jwilder-nginx-proxy/

That worked fine to get started, but I would like to be able to use certbot without having to go through the trouble of changing my DNS and forwarding ports. Especially because I recently changed ISPs to CenturyLink and apparently they have a bug where their router does not allow the user to forward port 443! So right now I have no way of generating certificates without trying to use my own router instead of CenturyLink’s.

If I try to do the same thing but bypassing the inbound TCP connection on port 443 by going through the HAProxy tcp reverse proxy in AWS, I seem to be able to renew certificates for my existing subdomains, but I can’t expand the cert to include a new subdomain. For example, I just ran the following to try to add the pwm and www.pwm subdomains:

 ./letsencrypt-auto certonly --standalone \
> -d sequentialread.com \
> -d git.sequentialread.com \
> -d comments.sequentialread.com \
> -d pwm.sequentialread.com \
> -d www.sequentialread.com \
> -d www.git.sequentialread.com \
> -d www.comments.sequentialread.com \
> -d www.pwm.sequentialread.com
Failed authorization procedure. pwm.sequentialread.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain :: Failed to connect to for TLS-SNI-01 challenge, www.pwm.sequentialread.com (tls-sni-01): urn:acme:error:connection :: The server could not connect to the client to verify the domain
:: Failed to connect to for TLS-SNI-01 challenge

 - The following errors were reported by the server:

   Domain: pwm.sequentialread.com
   Type:   connection
   Detail: Failed to connect to for TLS-SNI-01

   Domain: www.pwm.sequentialread.com
   Type:   connection
   Detail: Failed to connect to for TLS-SNI-01

   To fix these errors, please make sure that your domain name was
   entered correctly and the DNS A record(s) for that domain
   contain(s) the right IP address. Additionally, please check that
   your computer has a publicly routable IP address and that no
   firewalls are preventing the server from communicating with the
   client. If you're using the webroot plugin, you should also verify
   that you are serving files from the webroot path you provided.

Where is the IP address of the cloud instance.

I’m wondering if maybe there are some hijinks related to the SNI part of TLS? I’m not sure but I believe that the HAProxy should be blindly proxying the tcp connection. This is my HAProxy config file if that helps:

# Global settings
    chroot      /var/lib/haproxy
    pidfile     /var/run/haproxy.pid
    maxconn     4000
    user        haproxy
    group       haproxy

    # turn on stats unix socket
    stats socket /var/lib/haproxy/stats

# common defaults that all the 'listen' and 'backend' sections will
# use if not designated in their block
    mode                    tcp
    option                  dontlognull
    retries                 3
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout check           10s
    maxconn                 3000

frontend  main *:443
    default_backend recycle_computer

backend recycle_computer
    balance     roundrobin
    server      recycle_computer check

I’m stumped because there is nothing on the AWS HAProxy instance that relates to subdomain, but renewing the existing subdomains works, but adding the new ones doesnt :sweat:

my domain name: https://sequentialread.com/
certbot-auto version 0.9.3
operating system version:

uname -a
Linux sequentialread 3.16.0-4-amd64 #1 SMP Debian 3.16.7-ckt11-1+deb8u6 (2015-11-09) x86_64 GNU/Linux

Authorizations are valid for multiple days – maybe renewing only works because the authorization is still valid?
You could confirm that by using a different account – if issuing a cert for the same set of domains fails with a new account, you were using old authorizations.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.