As part of my work I am trying to figure out a way of integrating cert-manager with some sites that span across multiple Kubernetes clusters, in particular one Kubernetes cluster A is the active site, and another Kubernetes cluster B is the passive site.
I have cert-manager installed and configured with the same Let's Encrypt account on both sides, we have a domain, let's call it "domain.for.customer.example.com". I requested a certificate for that domain 9 days ago by creating a certificate object in Kubernetes A, and the request was fulfilled successfully, we got the cert, everything was fine.
Now today a surprising thing happened (hopefully I can get more information here) when I was trying to figure out how to import the certificate objects and secrets from the active Kubernetes side to the passive Kubernetes side (B).
First I dumped the certificate object on the active site A, and I basically did kubectl apply -f certificate.yaml, remember the domain is pointing only to the active side, not the passive side, so my expectation was that the request would never be fulfilled, yet somehow cert-manager managed to get a new certificate/key (I checked both certs they are different, both are valid).
How is this even possible? Shouldn't LE contact the IP my domain is pointing to and try to fetch domain/.well-know/acme/token first before we can fetch the certificate from LE? Is the domain associated with my Let's Encrypt account somehow and as a short of optimization the validation is skipped (As in contacting http://domain/.well-known/acme/token) ?.
Thank you in advance.
PS: Not sure if this is the right section for the post, if not feel free to move it to the right one.
I'm not familiar with cert manager at all, but I'm guessing that part of what you copied over was the ACME account key. Let's Encrypt remembers valid authorizations for an account for 30 days and won't require another challenge during that time for a certificate for those same names when requested by the same account.
Ah yes that would explain it, we are in fact using the same account in both sides on purpose (I am basically creating the same ClusterIssuer and secret in cert-manager terminology on both sides).
The certificate object in cert-manager only serves to basically ask cert-manager to issue a new request to let's encrypt and track it's state (I didn't import the secret that contains the certificate/key once it has been issued).
I was not aware at all of this behavior, so it was really surprising, thanks for the clarification.
As you're scaling up your usage of Let's Encrypt, you might also want to read through the Integration Guide. It doesn't cover that specific question (though maybe it should), but covers other things you might want to be aware of as you grow your usage and start using multiple systems.
I did already, in fact I opened a post here in the past asking for best practices, hence me using the same account everywhere.
We are still on a very small phase of the integration. We are planing to use only 20 to 30 domains so I don't expect huge issues for now, but probably I will reach out again in the future when necessary.