Change in rate limits?

Hi folks,

Since the Boulder update this morning (+360b474) we’ve started getting a bunch ‘urn:acme:error:rateLimited’ errors (‘Error creating new registration :: too many registrations for this IP’). This started just after the update at about 2017-06-29 17:45 UTC.

Was there a change in the allowed rate limits, or the rate limiting algorithm? I don’t think we’ve changed our load pattern particularly.

Thanks!

@jsha or @cpu, could you comment on this?

In case it is helpful, the IPs in question are (currently) 34.227.194.39 and 52.54.178.249.

Sorry we forgot to communicate this clearly. We did in fact update the rate limits, and we should have update the rate limits page / posted on API announcements first. In short: The “registrations per IP address” limit is now 10 per 3 hours. There’s a new limit, “registrations per IP range”, that applies to IPv6 /48 ranges and is 500 per 3 hours.

Can I ask what your use case is that you need to create large numbers of registrations?

Thanks, @jsha. We run a hosting platform, so we manage many customer’s domains. For simplicity and statelessness, our cert management processes ends up creating a new registration each time it tries to generate a cert.

This was working fine until the recent update, but with this new rate limit it sounds like this is no longer a good pattern for us to use. We’re looking at refactoring our code to share a single registration now.

A couple questions to help us with this refactor:

  1. is the limit on new keys, or the agree registration call? That is, if we use the same key instead of generating a new one each time, but still call agree registration each time, will this trigger the rate limit?

  2. If we’re reusing the same account each time, we now need to pay attention to other rate limits, such as “300 Pending Authorizations on your account”. Do pending authorizations ever expire? That is: if our process occasionally crashes and leaks a registration, will this build up forever until we run out? Any other stateful limits we need to consider?

This will hit a different error, as we don't allow the same account key to be used across multiple accounts.

Yes, you will need to pay attention to the pending authz limit, though you would have to pay attention to these with separate accounts as well you would just have a little bit more headroom.

Pending authorizations expire 7 days after creation but you can also explicitly deactivate them (See "Clearing Pending Authorizations" on the rate limit documentation).

If you haven't seen it yet you might also find our large integrator guide useful. It covers some additional things to consider (especially in regards to retrying errors).

Apologies for catching you unaware on this change. I dropped the ball updating the documentation & API announcements page but will do so today!

@cpu @jsha Apparently these limits also apply to staging. During testing I’ve hit this limit pretty soon.
I don’t have the exact count but I guess it wasn’t more than cca. 20 attempts in couple of hours.
Is this as intended or should this limit be more relaxed on staging?

2017-06-30 15:43:05,762:DEBUG:requests.packages.urllib3.connectionpool:https://acme-staging.api.letsencrypt.org:443 "POST /acme/new-reg HTTP/1.1" 429 144
2017-06-30 15:43:05,763:DEBUG:acme.client:Received response:
HTTP 429
Server: nginx
Content-Type: application/problem+json
Content-Length: 144
Boulder-Request-Id: jSHbcrSspA29h-nkT4ykagHKK6bJ7qQmEJ8k2791BOM
Replay-Nonce: _IkYp-ZVg9l5l1V2jSx2_gO14J8O8E7tCG1UIZ3gvZQ
Expires: Fri, 30 Jun 2017 15:43:05 GMT
Cache-Control: max-age=0, no-cache, no-store
Pragma: no-cache
Date: Fri, 30 Jun 2017 15:43:05 GMT
Connection: close

{
  "type": "urn:acme:error:rateLimited",
  "detail": "Error creating new registration :: too many registrations for this IP",
  "status": 429
}

That's correct - they are set the same in staging presently.

I'm open to relaxing the limits slightly in staging. @jsha - what do you think?

Yep, I agree it makes sense to have a higher limit in staging. We’ll work on that.

I’ve posted a proper API announcement for this change, and the rate limits documentation has also been updated.

Thanks @n1mmy for flagging this change and reminding us to post an announcement/docs update.

Hi, we have created a page where we are trying to put all existing limits. Send me a message, if you spot something missing, we will keep updating it as quickly as possible.

https://keychest.net/content/letsencrypt_numbers_to_know

Thanks all! We reworked our code to avoid creating many new accounts and now use one account for all requests. Things seem to be working OK -- we've still got a little clean up to do as we're hitting a couple new rate limit issues, mostly we just need to tweak some retry backoffs.

This will hit a different error, as we don't allow the same account key to be used across multiple accounts.

This confused me as this seemed to be working just fine for us. Turns out our ACME client helpfully avoids re-registering if the key has already been used: https://github.com/hlandau/acme/blob/master/acmeapi/api.go#L281

Apologies for catching you unaware on this change. I dropped the ball updating the documentation & API announcements page but will do so today!

No worries! For us, a note in the changelog mentioning the rate limit change would have been ideal... that was the first thing I looked at when our alerting went off.

Thank you all again for your work on Let's Encrypt. It really is a great service to the internet. We (and our customers) very much appreciate it!

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.