Letsencrypt blocking cert-manager v0.11?

This week, I’ve been updating three kubernetes clusters to cert-manager v0.11. The first two went smoothly.

On the third, LetsEncrypt is blocking me due to the ACME client being too old:

E1101 21:26:38.531057 1 setup.go:177] cert-manager/controller/clusterissuers “msg”=“failed to verify ACME account” “error”=“acme: urn:ietf:params:acme:error:rateLimited: Your ACME client is too old. Please upgrade to a newer version.” “related_resource_kind”=“Secret” “related_resource_name”=“letsencrypt-staging” “related_resource_namespace”=“cert-manager” “resource_kind”=“ClusterIssuer” “resource_name”=“letsencrypt-staging” “resource_namespace”=""

This is the current cert-manager release, so I’m thinking there’s probably something else going on. Hopefully, cert-manager v0.11 hasn’t been blocked already!

5 Likes

Yeah, something looks a bit busted with the pattern they are using.

Sending a request with User-Agent: jetstack-cert-manager/v0.9.0 succeeds but User-Agent: jetstack-cert-manager/v0.11.0 is blocked.

@lestaff

5 Likes

@rvandegrift @_az Thanks for bringing this to our attention! We’re investigating now.

2 Likes

We’ve identified and resolved the problem. Sorry about the trouble!

2 Likes

@JamesLE 20 minutes after your post saying you have “resolved the problem” I am still seeing it in LetsEncrypt staging https://acme-staging-v02.api.letsencrypt.org/directory

Is there a status/outage page where we can track this sort of thing?

3 Likes

I’m still seeing the previous error.

1 Like

Edit: gone from both environments now

1 Like

Sorry I didn’t clarify. We reverted the change immediately in production, and began iterating on it in staging. We believe this is now resolved in staging.

3 Likes

Confirmed that staging is working, I was just able to register a cluster issuer. Thanks @JamesLE!

3 Likes

Still receiving this error on my kubernetes cluster, running cert-manager 0.7.2, upgraded to 0.8? Does this mean that I have to upgrade all the way to 0.11 to get this working?

Edit: Upgraded to 0.11 problem persists.

Hi, @davidq2q,

Are you receiving the same exact error message - “Your ACME client is too old. Please upgrade to a newer version.” - or a slightly different one? Could you please paste some log output and context for us?

Thanks!

We are having same issue. cert-manager 0.5.2. Following message is displayed in our logs:

I1105 18:18:05.203780 1 controller.go:171] certificates controller: syncing item ‘name-of-tls’
I1105 18:18:05.204449 1 sync.go:274] Preparing certificate name-of-tls with issuer
I1105 18:18:05.206807 1 prepare.go:263] Cleaning up previous order for certificate name-of-tls
I1105 18:18:05.206896 1 prepare.go:279] Cleaning up old/expired challenges for Certificate name-of-tls
I1105 18:18:05.206925 1 logger.go:38] Calling CreateOrder
I1105 18:18:05.907498 1 sync.go:276] Error preparing issuer for certificate name-of-tls: acme: urn:ietf:params:acme:error:rateLimited: Your ACME client is too old. Please upgrade to a newer version.
E1105 18:18:05.907578 1 sync.go:197] [name-of-tls] Error getting certificate ‘name-of-tls’: secret “name-of-tls” not found
E1105 18:18:05.907623 1 controller.go:180] certificates controller: Re-queuing item “name-of-tls” due to error processing: acme: urn:ietf:params:acme:error:rateLimited: Your ACME client is too old. Please upgrade to a newer version.

Does this mean we have to upgrade our cert-manager? What version of cert -manager currently has working ACME communication?

Hi, @amlyn,

Let’s Encrypt is currently blocking versions lower than 0.8.0, and strongly encourages tracking the newest version. Details are available in this thread: Blocking old cert-manager versions

3 Likes

I got the same error message on my cluster on v0.5.2 and thus upgraded to v0.10.1, since v0.11.0 wouldn’t start. By the way, the newest version I can install from the catalog in Rancher is v0.5.2.
I still get this error message for one of my subdomains.

To clarify, it was working fine before. I’m currently preparing a cluster to move my services over from a single server, and am setting up one thing after the other. I have some working certificates associated with ingresses, but my most recent addition refuses to work, although i did everything exactly as before.

the newest version I can install from the catalog in Rancher is v0.5.2

That’s good to know, I’ll reach out to a friend there to see what can be done for the Rancher community.

Thank you very much!
I’m talking about the app cert-manager in the default library of Rancher. It’s very easy to use, since it is configured to work with letsencrypt out of the box (if you choose the option while deploying it) and it requires only very little effort to use. At least it used to, since v0.5.2 is not supported by letsencrypt anymore. It was a great tool until then, so I hope it can come back with a newer version.
I already tried tinkering around with it to get a newer docker image into it, but after three or four hours of trying stuff gave up, because every time I fixed one error in the CRDs, another one came up.

Have you seen this post from the Rancher forum?

Because of the changes that Jetstack is rolling out between versions, it is, unfortunately, not possible to adhere to the “one-click upgrade” objectives of the App Catalog. This is not a negative reflection of Jetstack; if anything, it’s a byproduct of their success. All the same, we’re going to temporarily remove cert-manager from the App Catalog. You’ll still be able to deploy it directly, either via Helm or through standard manifests, both of which are clearly explained in their documentation 3.

Jetstack also does a fantastic job of outlining the specific instructions for upgrading from each version 1, calling out special instructions when necessary, and you’ll find their dev team in #cert-manager on the Kubernetes Slack.

No, actually, I hadn’t. This is a pity, but then I’ll have to bite the bullet and set up my cluster with the jetstack-provided chart, I guess.

If it’s any consolation, it’s pretty easy. I’ve updated a few test clusters with it in the past week.