IP Blocked Because of Ridiculously Excessive Traffic

Help
1

Hello

We have a kubernetes cluster running in Google cloud and we are using cert-manager to automatically request and renew SSL certificates.

I was upgrading our kubernetes nodes and noticed after the upgrade that we are not able to request SSL certificates from Letsencrypt anymore. I checked Google cloud logs and the cert-manager logs and I noticed that there had been lots of failing renewal requests to the Letsencrypt.
All of these renewal requests failed and returned error code 503: “could not reach ‘domainhere/.well-known/acme-challenge/bNkQzzMOJ8_ss3H9YMePHhaFpp_U2xPef7xuItZWj0U’: wrong status code ‘503’, expected ‘200’”

I investigated more and noticed that we had three certificates in our cluster which were no longer needed. We had moved deployments which were using these certificates behind CloudFlare and using CloudFlare SSL.
We did not delete the certificates from the cluster and when cert-manager tried to renew them it failed because the deployment was no longer available.

Because of the multiple failing renewal requests our IP address got blocked:
“Error initializing issuer: 403 urn:ietf:params:acme:error:rateLimited: Your IP, 35.198.101.148, has been blocked due to ridiculously excessive traffic. Once this is corrected you may request this be reviewed on our forum https://community.letsencrypt.org”

I have deleted the three old certificates so there should not be excessive traffic anymore.

Could you please unblock our IP address 35.198.101.148?

2

You will need to update cert manger first, it had bug that requested millions of certs to le server.
And @lestaff will unblock your address.

2 Likes
3

Hi @joonas, welcome to the community forum :wave:

As @orangepizza mentioned (thanks!) part of the problem is that older cert-manager versions often handled this sort of configuration problem poorly, generating much more traffic than is required without end.

Can you share which version of cert-manager you were using when the problem occurred initially, and which version you're running now?

2 Likes
4

We were running cert-manager version v0.6.2 and I have now upgraded it to version v0.7.2

5

Great, thanks. I’ll file a ticket with our SRE team to have your IP address unblocked. Someone will update this thread when the change has been made.

Thanks,

1 Like
6

@joonas,

Your IP has been unblocked. Best of luck out there.

2 Likes
7

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.