CI & auto-deploy to Kubernetes w/ kubelego

I just got a Kubernetes cluster running a couple weeks ago, with GitLab installed on it, and auto deploying the company’s website and a couple of projects in development to it using GitLab’s built in CI. It’s using kubelego to automatically get certificates for each new subdomain that’s being deployed to. That’s one for each branch, many of which are ephemeral–e.g. for a new issue, a branch is created to work on it, and a review app is deployed to a project-name-review-xxxxx.domain.com, where xxxxx is generated for each new branch…when the issue is solved and the branch merged, the deployment is removed from the cluster, but it’s already used up one of my 20 certificates for the week. Right now, we’re kind of on fire and I can see us hitting 20 certificates, where probably most of them are for those ephemeral per-issue branches and only a few of them will be for the final subdomains to be used in production.

I’m not sure what’s the best way to go about this. I’ve tried leaving out the tls-acme annotation from the manifest for those ephemeral branches, but the nginx ingress controller is set up to redirect all http traffic to https and I think I’d prefer it that way anyway. But 20 might limit us from being able to review an issue fix when we’re particularly active that week. I don’t think we’ll cross over the 500 subdomain threshold to qualify for an exemption, either. Is there a way to get a rate limit somewhere in between 20 and 500, or is it set that way due to the manual work involved? (I imagine you have to look over the request, then either change your code or at least add a database entry for every domain that asks for this…)

Hi @ftab,

Is there a way to predict the branch names ahead of time? If so, you can include many branch names on a single certificate (up to 100 names in all) and only count as 1 certificate for rate limit purposes.

If not, I don’t think Let’s Encrypt is a good solution for your use case right now and I would suggest getting a wildcard certificate from another CA until Let’s Encrypt starts to offer wildcard certificates last year. A rate limit exemption is probably not available to you because it appears that all of your certificates are used within the same organization for its own purposes.

Each certificate issuance puts load on the hardware security module (HSM) that Let’s Encrypt uses, including for the initial issuance and for signing OCSP status updates until that certificate expires (even if the certificate is no longer in use!). The HSM has a particular rate at which it can issue signatures, and this is the main reason for the certificate quantity ratelimits.

Not that I know of - part of the piece it generates is the issue or merge request number and title, followed by a 5 alphanumeric character blob - perhaps a hash or randomly generated to avoid collisions.

:grimacing: even though these throwaway subdomains I generated are no longer accessible, it still has to remember it and sign OCSP for the next 3 months? Now I feel bad…think in the process of setting this cluster up over the last few weeks I generated and subsequently lost or deleted dozens of certificates as I tore down the cluster and remade it several times

So, I think you would be better off with a wildcard certificate, which Let’s Encrypt doesn’t offer yet but will offer from next year.

I suppose since wildcards are coming next year anyway (a few months away), it doesn’t make much sense to grab a wildcard from another CA. I can work around the rate limit in the meantime, and knowing each one puts additional constant load on the hardware for the next 3 months, will probably try to figure out a way to not do it for issue branches (just staging and production).

there is also an option of using self signed certs for staging and testing and only use live certs for production

another option is to get a origin certificate from cloudflare.I know this is Let’s Encrypt forums but until wildcard certificates are available.

https://www.cloudflare.com/ssl/dedicated-certificates/.

Andrei

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.