K8s duplicate certs

In the company we use k8s to host, our prod, staging and many dev branches as needed, also we have a few different applications.

the naming conventions is something like:
example.com
staging.example.com
featureName.app.example.com

we've been using cert-manager and letsencrypt cluster issuer for years and no problem.

recently one of our dev branches is having this, every time we deploy a new version, issues a new cert, and since is an actively working environment it gets over the limit of 5 certs for the same domain per week, and is the only env/namespace where this happens, in any other, if the certificate remains valid, it won't be regenerated on new deployments only when expires.

any idea on why this is happening or what can I do to fix it?

Thank You in advance.

1 Like

Hello @mzvstar,

Recent change Let's Encrypt is adding two new remote perspectives for domain validation

Is there any chance of geo blocking?

2 Likes

I'm not familiar with the specific systems, but the core idea is that the certificates need to be stored in persistent storage, not part of the ephemeral container environment. But it sounds like you might already understand that. If some of your environments do work that way, and others don't, can you compare how those environments are configured differently? Can you post some of the configuration details here so that hopefully someone familiar with how to configure that ACME client can help?

7 Likes

I assume you know

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher.

Let's Encrypt uses Multi-Perspective Validation Improves Domain Validation Security - Let's Encrypt

2 Likes

Moved from Issuance Tech to Help

2 Likes

Yeah, but the problem is that when using the staging issuer, the certs are not valid in the browser so stakeholders are not able to test.

Are they testers and developers, or production environment users?

1 Like

all of them, some are prod users

1 Like

So yes the staging would be an issue for you.

1 Like

Welcome to the Let's Encrypt Community, @mzvstar! :slightly_smiling_face:

The strategy I implemented at my previous employer was to issue a certificate for the necessary domain names using a centralized, dummy ingress definition where the service application definitions don't matter. The real ingress definitions for your actual environments can then have their defaults configured to use the centralized certificate located in certmanager's namespace. This will prevent repeat issuance and the ingress manifests from "fighting over" ownership and maintenance of the certificate and consequently reporting incorrect namespace and port info to kubectl. Though it's probably obvious, this strategy completely avoids needing to issue a new certificate when you update any SDLC environment, including production, unless you decide to uninstall certmanager.

5 Likes