We use nginx-ingress and cert-manager in GKE, and are having rate limiting issues with DNS authorizations during our first renewal process. We have around 900 ingresses, each has around 80 total domains (40 root and 40 www via SNI) bringing us to around 72,000 domains that need DNS auth. During the initial certificate requests we worked with Let’s Encrypt to raise some of the rate limits on our account to accommodate and we were able to script the deployment of all 900 ingresses and creation of the corresponding 900 certificates.
Now we are nearing the first renewal (i am seeing 200ish hours till expires in the logs) and cert-manager appears to be trying to pre-authorize via DNS (we use google cloud dns auth) and we are again getting “429 urn:ietf:params:acme:error:rateLimited: Rate limit for ‘/acme’ reached” errors. and “429 urn:acme:error:rateLimited: Error creating new authz :: too many currently pending authorizations: see https://letsencrypt.org/docs/rate-limits/”
Looking at our logs, it looks like cert-manager is trying to prepare a 2-3 certificates a second and all if not a large portion are getting one of the two 429 errors.
My domain is:
www.technician.jobs (one of +72,000 domains managed using this stack)
My web server is (include version):
The operating system my web server runs on is (include version):
My hosting provider, if applicable, is:
Google Cloud GKE
I can login to a root shell on my machine (yes or no, or I don’t know): yes