Too many new orders recently

Hello,
We've got an email below from golang security team a few days ago.
So, We updated the package "golang.org/x/crypto".
Now, Our server is showing error messages with

2022/01/29 21:08:19 http: TLS handshake error from 92.118.160.9:49997: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 66.249.79.221:47900: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 157.230.54.29:43970: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 207.46.13.163:62720: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 66.249.74.118:45840: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 66.249.79.74:64796: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 66.249.65.38:53402: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 66.249.72.84:40896: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 21:08:19 http: TLS handshake error from 138.246.253.24:56650: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/

and browser shows like below

# This site can’t be reached
**site** took too long to respond.
ERR_TIMED_OUT
# This site can’t be reached
**site** took too long to respond.
ERR_SSL_PROTOCOL_ERROR

We are serving like web hosting with over 1000 domains.
I guess our service makes too many create request in short time.
Is there any way to reset create/renew limit?
Please, help to solve this issue.

Our server is golang native with package golang.org/x/crypto.

============================= email ================================

Hello gophers,

The Let’s Encrypt certificate authority is revoking all certificates issued with the TLS-ALPN-01 verification method before 00:48 UTC on 26 January 2022 due to a compliance issue. (Read more [in the Let’s Encrypt announcement](https://community.letsencrypt.org/t/2022-01-25-issue-with-tls-alpn-01-validation-method/170450).) As TLS-ALPN-01 is the preferred and default verification method used by [golang.org/x/crypto/acme/autocert](http://golang.org/x/crypto/acme/autocert), most certificates managed by autocert will be revoked beginning at 16:00 UTC on 28 January 2022. This will cause connection errors on some platforms.

We recommend updating the [golang.org/x/crypto](http://golang.org/x/crypto) module to version v0.0.0-20220126234351-aa10faf2a1f8 (or later), which will automatically renew potentially affected certificates issued before Let’s Encrypt deployed their fix.

Alternatively, delete ALL files in the autocert cache EXCEPT "acme_account+key" or "acme_account.key", and restart the application. If using [autocert.NewListener](https://pkg.go.dev/golang.org/x/crypto/acme/autocert#NewListener) on Linux, the cache is located at $XDG_CACHE_HOME/golang-autocert or $HOME/.cache/golang-autocert.

In order to get notified of similar issues in the future, we recommend setting the [Manager.Email](https://pkg.go.dev/golang.org/x/crypto/acme/autocert#Manager.Email) field.

Cheers,
Go Security team

No, the new order per 3 hour rate limit has already been increased from 300 to 1000. So you can make 1000 new orders per 3 hours, which should be plenty.

2 Likes

Eventhough, 3hours passed, The error logs still shown.
Could i check the requested order list for debugging?
or, Is there a way to revoke the requested orders?

2022/01/29 23:36:59 http: TLS handshake error from 157.55.39.62:54593: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:00 http: TLS handshake error from 114.10.16.71:51974: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:00 http: TLS handshake error from 114.119.131.50:41944: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:00 http: TLS handshake error from 114.124.240.122:52041: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:01 http: TLS handshake error from 54.191.238.170:63204: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:03 http: TLS handshake error from 92.118.160.37:40761: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:03 http: TLS handshake error from 92.118.160.37:32803: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:05 http: TLS handshake error from 54.221.27.173:34174: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:05 http: TLS handshake error from 114.4.83.135:15944: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 40.77.167.100:4225: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 87.250.224.52:60862: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 114.119.159.216:18934: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 34.77.162.10:42881: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 66.249.68.51:65494: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 114.119.149.108:62526: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 66.249.72.84:50536: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 66.249.79.89:46959: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 123.125.109.115:43488: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 180.244.167.78:23558: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 114.79.1.207:50507: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 203.175.8.20:56066: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 114.119.131.235:47802: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/
2022/01/29 23:37:11 http: TLS handshake error from 157.55.39.149:28929: 429 urn:ietf:params:acme:error:rateLimited: Error creating new order :: too many new orders recently: see https://letsencrypt.org/docs/rate-limits/

Not very strange, right? In that small piece of log it seems your ACME client is making a lot of requests in a very small time frame.

As far as I know, all the rate limits are sliding windows. So once you've hit a rate limit, the "sliding window", well, "slides" along all the new orders that came through.. So for example:

If you start at t=0 and request a certificate (and new order) every 5,4 seconds, you'd fill exactly 1000 new orders in 1,5 hours. Now, if you request a 1001th order, it will fail due to the rate limit. You'd have to wait for 1,5 hours until the rate limit is lifted.. BUT! It's not lifted entirely.. At exactly 3 hours from when this thought experiment started, only the FIRST order will be outside of the 3 hour rate limit window.. The other 999 orders are still within the window! So you can only request a single order, until you've waited 5,4 seconds for the 2nd order to fall outside of the window.. And 5,4 more seconds for the 3rd.. Et cetera.

So if you keep requesting a lot of certificates, faster than once per 10,8 seconds (which is 1000 in 3 hours), you keep seeing errors..

4 Likes

Ah, I see. Thank you for your replies.

3 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.