K8s cert-manager duplicate cert ratelimited

Hello, first off thanks for all the hard work. #HugOps to you all, I’m sure today has been a trying time. I’m using cert-manager v0.9.1 within kubernetes and I am still getting a rate limiting error: “too many certificates already issued for exact set of domains” despite the increase in limits. If I look for the certificate name on https://crt.sh/ I can see certificate creation happening today multiple times with a notBefore:2020-03-03(I assume that’s what this crt.sh tool does), but it never makes it down to the cluster via cert-manager, and I get the rate limiting error. I’m out of ideas for fixing this, any suggestions?

2 Likes

I split you off of the main topic so that specific triage can happen here. Thanks for the #hugops :slight_smile:

1 Like

Thanks @Phil! I’m curious, does the crt.sh tool in fact do what I think? I am reading it as the certificate exists via letsencrypt, but the rate limit is preventing cert-manager from pulling it down. Is that accurate, or am I off with my assumptions?

@lucasreed,

That’s a good question. We run cert-manager for our certificate transparency logs, but haven’t been affected by our own rate limits there. Let’s check what the cert-manager logs say

kubectl -n ${NAMESPACE} logs ${POD}

Ok, I’ll lay out the steps I’ve done so far. I patch the Certificate object to add a .spec.renewBefore of 2159h00m00s. As expected, cert-manager starts to go through the process of validating the challenge (http01) which it does successfully. It then throws out the error below:

E0304 02:05:37.297236 1 base_controller.go:189] cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="error creating new order: acme: urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many certificates already issued for exact set of domains: REDACTED,REDACTED: see https://letsencrypt.org/docs/rate-limits/" "key"="REDACTED"

I then gave up and removed the .spec.renewBefore property of my Certificate object.

I redacted client-identifying information.

Gotcha, ok. We increased the duplicate certificate rate limit from 5/3d to 10/3d earlier today. For some reason cert-manager kept on issuing instead of applying the updated certificate. To sidestep this issue, you can add or remove of the SANs on the certificate and reissue the cert.

1 Like

Unfortunately I don’t think that will work for our case. Because of the way the validation happens over an ingress via http-01. We can’t use dns-01 validation because it’s a domain we don’t own or manage. If I add a random domain I don’t believe validation will ever pass, but I can try.

@lucasreed,

For extra eyes on this, have you tried posting in #cert-manager on kubernetes.slack.com?

Good thought, I’ve been in there following along, but thought I’d start here because of the rate limiting message. I can try the extra domain and then go chat with them if that fails.

Thanks for your responses Phil, I appreciate it!

1 Like

Regarding the extra domains, our rate limiting documentation states,

A certificate is considered a renewal (or a duplicate) of an earlier certificate if it contains the exact same set of hostnames, ignoring capitalization and ordering of hostnames. For instance, if you requested a certificate for the names [ www.example.com , example.com ], you could request four more certificates for [ www.example.com , example.com ] during the week. If you changed the set of hostnames by adding [ blog.example.com ], you would be able to request additional certificates.

So, interesting thing today. I was working with another colleague on a different cluster where the setup is more conducive of adding a fake domain. It appeared to work, as the domain was validated. Right after validation it gave the same rate-limiting errors even though it was a new certificate with the new domain added. We are unfortunately at a loss here. I imagine we’ll need to check with the cert-manager folks.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.