Blocking old cert-manager versions

[jsha]

We’ve been working with Jetstack, the authors of cert-manager, on a series of fixes to the client. Cert-manager sometimes falls into a traffic pattern where it sends excessive traffic to Let’s Encrypt’s servers, continuously. To mitigate this, we plan to start blocking all traffic from cert-manager versions less than 0.8.0 (the current semver minor release), as of November 1.

We’ll be sending out notifications to cert-manager clients that meet those criteria over the next two months.

Version 0.8.0 is much better but we still observe excessive traffic in some cases. We’re working with Jetstack to improve these cases. As new versions of cert-manager are released, we will add the non-current versions to our block list after 3 months. We strongly encourage cert-manager users to stay up-to-date with new versions.

Also, if you are a cert-manager user, there is an opportunity to help both Let’s Encrypt and Jetstack. Check the logs for your cert-manager instances. Are they making excessive requests to Let’s Encrypt (more than, say, 10 per day over multiple days)? If so, please share details at https://github.com/jetstack/cert-manager/issues/1948.

Thanks,
Jacob

[jsha]

Here’s the text of the email we plan to set to users of old cert-manager versions. I welcome feedback from anyone in the community, even people who don’t use cert-manager. Is anything missing? Is there something I could say more clearly? Thanks!

Proposed email

We’ve been working with Jetstack, the authors of cert-manager, on a series of fixes to the client. Cert-manager sometimes falls into a traffic pattern where it sends really excessive traffic to Let’s Encrypt’s servers, continuously. To mitigate this, we plan to start blocking all traffic from cert-manager versions less than 0.8.0 (the current semver minor release), as of November 1. Please upgrade all of your cert-manager instances before then.

We’re sending this email because this is the contact address of your cert-manager instance at {{ client_ip }}.

Version 0.8.0 is much better but we still observe excessive traffic in some cases. We’re working with Jetstack to improve these cases. As new versions of cert-manager are released, we will add the non-current versions to our block list after 3 months. We strongly encourage cert-manager users to stay up-to-date with new versions.

Also, there is an opportunity to help both Jetstack and Let’s Encrypt. Once you’ve upgraded, please check the logs for your cert-manager instances from time to time. Are they making excessive requests to Let’s Encrypt (more than, say, 10 per day over multiple days)? If so, please share details at https://github.com/jetstack/cert-manager/issues/1948.

Thanks,
Let’s Encrypt Team