Best way to Certbot-renew a certificate in a locked-down system

I have a LetsEncrypt certificate for a system which I generated using Certbot --standalone when my system was open to the public internet, during development. The system has now moved into active use behind a firewall and so my options to renew the certificate are limited as follows:

  • I cannot open ports below 1024 for incoming TCP connections, which, as I understand it, means that --standalone is no longer an option,
  • I am running NGINX but it is inside a Docker container so, as I understand it (and I've tried) I can't take advantage of the --nginx plugin,
  • NGINX is running in proxypass mode to Jenkins, which is the active web server in the system, (the entire purpose of the system is to run Jenkins, nothing else) HOWEVER, access to Jenkins is completely locked-down [this is what NGINX is for] by SSL client authentication to known users only; LetsEncrypt won't be allowed in that way and hence I can't use --webroot.

What is the best way to get Certbot auto-renewal running in this context? Do I have to spin up another web server and open another port just for doing the renewal, or is there an easier way?

DNS-01 validation may be an option, depending on your DNS provider, ACME client, and the restrictions between your host and your DNS API.

4 Likes

Is there a nearby device that is already using port 80 [HTTP]?
If so, it might be able to proxy those challenge requests to your system.

3 Likes

@rg305: Unfortunately not: this is deliberately an "island" system, the only way incoming connections get through to it is via forwarding down an SSH tunnel from a "gateway" machine, which is where the restriction of only ports 1024 and over comes from (because Linux, by design, will not forward ports lower than 1024).

EDIT: unless there is some other method of "proxying" of course.

@linkp: not sure how to check who our DNS provider is from here but I would guess that it is AWS.

Try ICANN Lookup

2 Likes

dig ns example.com

  • or if you have delegated a subdomain to another DNS provider:

dig ns delegated.subdomain.example.com

4 Likes

Ah, sorry, I misunderstood, yes, the URL through which the system is accessed is currently through no-ip [I will be moving it to AWS but it is not there yet].

And, in case it is of use, the "gateway" machine that is at the far end of the SSH tunnel from my system is a Digital Ocean droplet (which runs the usual no-ip dynamic DNS client).

You will probably need to use DNS-01 authentication. You have several options to implement this.

First, to manage your DNS records I strongly suggest installing acme-dns (GitHub - joohoi/acme-dns: Limited DNS server with RESTful HTTP API to handle ACME DNS challenges easily and securely.) on another server and delegating your acme_challenge records to that. That is the most secure option, but requires you to operate a server with ports 53 and 80/443 open.

If you cannot do that, you should still use the acme-dns technique of delegating DNS authentication to a secondary dedicated namespace that only handles ACME challenges. Most commercial DNS systems do not have fine-grained access control, so you will want to ensure a compromise on your locked-down system can impact actual DNS (or domain registration).

From there, there are three options I would recommend:

  • For fully automated solution, you can install Certbot on the locked-down system and configure it to use your delegated DNS system.
  • For a semi-automated solution, you can run Certbot on an office machine and use a Certbot hook to deploy the certificates to your machine and restart services to it. I like to use a fabfile.org script to automate this, other people like to use rsync and configure the remote machine to restart services periodically or whenever a change is detected.
    • Another automated solution is to securely store the certificate(s) on an API or server that can be accessed by your server, and then have your server periodically poll it for changes and reconfigure itself + restart services when necessary.

There is one other option that I hope you have: If your system is allowing nginx to listen to port 80 (I can't tell if nginx/jenkins is 80/443 or running on higher random ports; I've seen both options used), you can proxy-pass the acme-challenge directory onto a higher port and run Certbot on standalone mode with -http-01-port=8080 or similar.

6 Likes

I assume your nginx is serving the webserver on port 80? So nginx is able to circumvent your "no access to ports below 1024", right?

Why not just use the --webroot plugin? Or can't you make modifications to the nginx configuration inside Docker and the Docker file of that instance?

2 Likes

@osiris: no, [and I think the is relevant to the solution above], NGINX is on a specific port, not 80, above 1024.

Oof. I feared that was the case. I've seen a few jenkins/gerrit/jira/etc systems like that. Those are extremely difficult to manage and support for clients.

2 Likes

Ah OK, then yes, the dns-01 challenge is your only option.

However, with the free no-ip "Free Dynamic DNS" option (No-IP: Use Remote Access and DDNS to Easily Connect to Your Internet Connected Devices From Anywhere, Anytime) it seems to be impossible to add TXT records. I'm not sure it's even possible to get a sub-sub-domain for the _acme-challenge label with the free option.

Although the free service does allow CNAMEs, but I don't know about the whole _acme-challenge sub-subdomain.

1 Like

Understood, thanks for all the help guys, looks like I need to investigate ACME/DNS stuff (which is all new to me). I am, at least, in the paid noip world, so hopefully that gives me what I need.

2 Likes

Well paid noip shows TXT records.

1 Like

Seems like I can do it manually but not via an API:

2 Likes

Without an API I would recommend setting the _acme-challenge sub-subdomain to a CNAME pointing to the hostname corresponding with your acme-dns instance.

If you cannot run your own acme-dns instance, the author of the acme-dns (and DNS plugin for Certbot) runs one which can be used by any one. But note that strictly speaking that would mean the operator of that acme-dns instance could also issue certificates for your domain. So you'd need to trust the operator of that acme-dns instance to not do that. Of course certificate logs are useful to check this.

2 Likes

Thanks: it's Friday night here and I need more brain time to parse your first sentence :-).

Frankly, I hadn't expected certificate authentication to be so tied to ports 80 and 443 (which is the fundamental problem I'm hitting); I'd assumed it would be the destination that mattered, not which port you happen to land at on that destination...?

2 Likes

Here is a link to Challenge Types - Let's Encrypt and it describes the requirements.
There are other Free ACME Certificate Authorities ACME CA Comparison - Posh-ACME
and a list of DNS providers who easily integrate with Let's Encrypt DNS validation; not sure if any of this will be of much value to you.

3 Likes

Hmmm, yes "allowing clients to specify arbitrary ports would make the challenge less secure, and so it is not allowed by the ACME standard". I guess that, if it were possible to specify a port, then a port higher than 1024 could be used, which could be SSH-redirected. Obviously the machine in question would have to have been compromised for that to be harmful but, on the other hand, as soon as it is redirected it is no longer that actual machine...?

Anyway, it is what it is, just not what I expected.

1 Like