Multiple customer k8s clusters with subdomains hitting rate limit

We are building a product where each user gets their own Kubernetes Cluster provisioned for them and we also create a subdomain to give them access to this cluster e.g. cluster1.example.com

Obviously we need to issue certificates for these clusters, right now we are using certmanager in each cluster to issue a certificate using the LetsEncrypt issuer - but quickly we will hit the rate limit of 50 new certs a week.

If we configure each cluster's certmanager to instead issue a wildcard *.example.com I assume that will breach the 5 renewal's a week rate limit. Is this correct?

If this is the case, does anyone have advice on a suggested pattern to get around this problem? We could generate a wildcard cert separately from each individual cluster but then we will have to write custom logic to ensure this cert gets renewed and pushed out to all of the clusters regularly and I would like to avoid reinventing the wheel if a simpler solution exists.

1 Like

Is anybody able to help with this?
Thanks!

It is important to know that the 50 new certificate is means really new, renewal does not count in it. So after two weeks your k8s may have 100 different certificates, after three weeks 150, and so on. However, if the speed of growth of the k8s ecosystem exceeds 50 per week, you may apply for exception. The way to apply for it is explained on the page:

5 Likes

Thanks @bruncsak! Yeah this is a good point, our new certificate rate will not exceed 50/week, so I guess we are really focused on the renewal issue. Our clusters can be ephemeral so it is possible for a cluster with the same subdomain to be recreated more than 5 times in a week.

This is probably more of a cert-manager question so I will ask those more familiar elsewhere but may as well also ask here in case you are familiar with a solution:

Do you know of anything we can use with cert-manager to pull secrets from an external store and only issue a new cert if it has expired? Then push back to that external store if we renew the cert.

I do not use certificate manager program, unfortunately I can't help. May be someone else could give an idea what software to use?

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.