I have a LetsEncrypt certificate for a system which I generated using Certbot --standalone when my system was open to the public internet, during development. The system has now moved into active use behind a firewall and so my options to renew the certificate are limited as follows:
I cannot open ports below 1024 for incoming TCP connections, which, as I understand it, means that --standalone is no longer an option,
I am running NGINX but it is inside a Docker container so, as I understand it (and I've tried) I can't take advantage of the --nginx plugin,
NGINX is running in proxypass mode to Jenkins, which is the active web server in the system, (the entire purpose of the system is to run Jenkins, nothing else) HOWEVER, access to Jenkins is completely locked-down [this is what NGINX is for] by SSL client authentication to known users only; LetsEncrypt won't be allowed in that way and hence I can't use --webroot.
What is the best way to get Certbot auto-renewal running in this context? Do I have to spin up another web server and open another port just for doing the renewal, or is there an easier way?
@rg305: Unfortunately not: this is deliberately an "island" system, the only way incoming connections get through to it is via forwarding down an SSH tunnel from a "gateway" machine, which is where the restriction of only ports 1024 and over comes from (because Linux, by design, will not forward ports lower than 1024).
EDIT: unless there is some other method of "proxying" of course.
Ah, sorry, I misunderstood, yes, the URL through which the system is accessed is currently through no-ip [I will be moving it to AWS but it is not there yet].
And, in case it is of use, the "gateway" machine that is at the far end of the SSH tunnel from my system is a Digital Ocean droplet (which runs the usual no-ip dynamic DNS client).
If you cannot do that, you should still use the acme-dns technique of delegating DNS authentication to a secondary dedicated namespace that only handles ACME challenges. Most commercial DNS systems do not have fine-grained access control, so you will want to ensure a compromise on your locked-down system can impact actual DNS (or domain registration).
From there, there are three options I would recommend:
For fully automated solution, you can install Certbot on the locked-down system and configure it to use your delegated DNS system.
For a semi-automated solution, you can run Certbot on an office machine and use a Certbot hook to deploy the certificates to your machine and restart services to it. I like to use a fabfile.org script to automate this, other people like to use rsync and configure the remote machine to restart services periodically or whenever a change is detected.
Another automated solution is to securely store the certificate(s) on an API or server that can be accessed by your server, and then have your server periodically poll it for changes and reconfigure itself + restart services when necessary.
There is one other option that I hope you have: If your system is allowing nginx to listen to port 80 (I can't tell if nginx/jenkins is 80/443 or running on higher random ports; I've seen both options used), you can proxy-pass the acme-challenge directory onto a higher port and run Certbot on standalone mode with -http-01-port=8080 or similar.
Oof. I feared that was the case. I've seen a few jenkins/gerrit/jira/etc systems like that. Those are extremely difficult to manage and support for clients.
Understood, thanks for all the help guys, looks like I need to investigate ACME/DNS stuff (which is all new to me). I am, at least, in the paid noip world, so hopefully that gives me what I need.
Without an API I would recommend setting the _acme-challenge sub-subdomain to a CNAME pointing to the hostname corresponding with your acme-dns instance.
If you cannot run your own acme-dns instance, the author of the acme-dns (and DNS plugin for Certbot) runs one which can be used by any one. But note that strictly speaking that would mean the operator of that acme-dns instance could also issue certificates for your domain. So you'd need to trust the operator of that acme-dns instance to not do that. Of course certificate logs are useful to check this.
Thanks: it's Friday night here and I need more brain time to parse your first sentence :-).
Frankly, I hadn't expected certificate authentication to be so tied to ports 80 and 443 (which is the fundamental problem I'm hitting); I'd assumed it would be the destination that mattered, not which port you happen to land at on that destination...?
Hmmm, yes "allowing clients to specify arbitrary ports would make the challenge less secure, and so it is not allowed by the ACME standard". I guess that, if it were possible to specify a port, then a port higher than 1024 could be used, which could be SSH-redirected. Obviously the machine in question would have to have been compromised for that to be harmful but, on the other hand, as soon as it is redirected it is no longer that actual machine...?
Anyway, it is what it is, just not what I expected.