Brief overloads at midnight UTC (a request for help!)

Client dev
1

Let's Encrypt gets a lot of traffic, as you might imagine, but it's usually fairly spread out except for some notable peaks.

We've had some overload issues at the biggest peak throughout June at midnight (00:00) UTC, and it lasts for 15-20 seconds. We're working on various projects to scale and improve that, but those may take some time and it would be helpful to move load away from that peak.

We've identified a particular client, lego-cli, which appears to be used in some of the biggest spike. The lego-cli authors have helpfully added a random sleep to help smooth it out which will be great when people upgrade. And be sure to check out the graph linked in that issue to see the scale of the spikes.

We suspect there's some software package that's shipping lego-cli as a cronjob or similar which is running every day at midnight, utc. However, I haven't been able to find what that is. It must be popular however, for the wide variety of domains we're seeing in these load spikes. Do you know what that might be? Any leads would be helpful here.

If you're an author of a software package, or an administrator of a system, please don't schedule jobs to run exactly at midnight. It'll be less reliable for you, and cause me more pain!

16 Likes
3

Maybe its Docker container identifies as lego-cli?

It's e.g. used by dokku-letsencrypt, although that seems to have the version pinned to an older version than the mostly used in the spikes, so Dokku itself isn't the issue. But the Docker container in general might?

That container seems to be used in at least 105 pieces of code on Github.

Although I can't explain why 4.6.0 is the mostly used version in the peak when the Docker container of 4.7.0 has been published a month ago..

10 Likes
4

Oh:

8 Likes
5

We’ve identified a likely culprit: bitnami’s bncert-tool sets up a renewal cron at midnight and uses Lego-cli. That matches with many of the other clues we’ve seen. Thanks for the help!

12 Likes
6

Hm, weird, in all Bitnami guides mentioning lego, it was never lego-cli. I guess the exact binary name didn't need to be lego-cli :slight_smile:

Glad you've found the likely culprit!

8 Likes
7

The binary is called lego, but it sets its user-agent to lego-cli: lego/setup.go at 88a2bab2d9a7502bc7cd9e983fcf42499c8c2906 · go-acme/lego · GitHub

9 Likes
8

Aha, then I made an erroneous assumption.. :slight_smile:

6 Likes
10

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

11
12 Likes