I am hoping to reduce the number of rate limit questions on the forum and I believe that building some notifications in to certbot is a good way of doing this
I see 4 major pain points
A) Software automatically issuing new certificates without user being aware (renew scripts)
B) User playing with production when they should be using staging (new users)
C) Bad exit of programs means challenges are not invalidated
D) Bad awareness of how to deal with large numbers of domains (new users)
Some proposed solutions
A) There is a limit of 3 certificates in a week after which the script errors out and the user can configure emails to notify them - i have issued 3 certificates this week. Check configuration to see why
B) After a user has requested 3 certificates of the same domain a RED SCREEN appears saying - you are close to hitting the rate limits which may stop you being able to issue more certificates. If you are not aware of what rate limits are please review documentation here. If you are testing with certbot please use --staging flag.
C) I wrote some python code to check log files and invalidate pending authz. This could potentially be run as a cron job. Tool for Clearing Pending Authz - Looks at Certbot Log Files and Uses ACME Library
D) Once a user approaches 15 certificates issued in a week then a message pops up saying you can only issue 5 more certificates for the domain. You may want to think about using SAN certificates.
Renewals are not affected by rate limits, but they still count towards them. That's why it is recommended to issue new certificates before performing renewals. (E.g. if you were to renew 30 certificates for subdomains of example.com, all 30 will succeed, but after the first 20 you will be unable to issue any new certificates for example.com as you've hit the rate limit. However, if you were to issue 20 new certificates for subdomains of example.com and then perform the renewals, all operations would succeed.)
I want to do some thinking on most of the other points before responding because I'm a bit ambivalent on those suggestions at the moment for a variety of reasons, but:
C) This should, in the vast majority of cases, no longer be necessary. A change a couple weeks back causes Boulder to now re-issue the pending authz's until they are cleared or invalidated. Unfortunately I can't locate the post regarding this behavior change. Regardless, most authz issues we are seeing now are from unrelated causes.
@ahaw021 These all sound like interesting ideas but seem universally to be features that would have to be implemented client-side in an ACME client. To me the biggest challenge with that is exposing the rate limits to the client in a generalizable way. E.g. Most ACME clients are not Let's Encrypt clients and likely want to support other ACME servers as they appear. None of these rate limits would be useful to enforce against a different server that has totally different rate limits.
There are some provisions in the latest ACME draft that might make this workable but it will still be somewhat challenging. Boulder doesn't implement Section 6.5 presently but it will likely be something we revisit after launching the V2 API.
That’s still very tricky to do without an API, unless we’re going to have Certbot query CT logs or something. But I would also be wary of that because some of the rate limits could change, and we see a lot of people here ending up using quite old versions of Certbot (sometimes a full year or so out of date). It would be unfortunate if the certificates per registered domain limit doubled at some point and then for a year people were getting spurious warnings that they were about to exceed it!
I would totally agree that this would be a great feature to have in the client—I just don’t think it can be done in a reliable way unless we also get some kind of protocol-level support for querying rate limit advisories.
I don’t see it happening in a reliable way, but I could see it as a best-effort solution. If the assumption is made that the majority of users run certbot from one machine, then certbot will be aware of all certificates issued by a user, theoretically, and can display warnings based on that. It falls apart if someone uses alternate methods of issuance or has multiple servers running certbot for the same or different domains, but it is better than nothing.
That being said, I’m not sure I like how hackish it feels, and the possible false sense of security it could engender. If someone running certbot on multiple machines is relying on the warnings rather than reading up on rate limits, they could start hitting those limits through, arguably, no fault of their own. The argument there being that the warnings imply a level of safety that isn’t actually there, whereas the lack of warnings impart a responsibility on the user for their own avoidance of rate limits. In the end, I don’t think it would work out as well as intended without a more reliable method for determining issued certificates.
Yes I was thinking about this as you are right - with multiple clients you can get false positives
@schoen would it be possible to get the rate limits published as a JSON feed so they can be consumed programatically?
@DanCvrcek would this be something that keychest could potentially provide for clients. I know you are doing a whole lot of other useful checks. As keychest is also an online service we wouldn’t have issues with out of date rate limits.
I believe so. A check for the number of certificates issued per day / week is certainly doable. There are 2 aspects of it, which may complicate UI a bit:
number of certificates with exactly the same set of domains, i.e., coming from (hopefully) the same certbot copy
number of certificates per registered domain
The next update of KeyChest introduces notifications (new certificates) and a RESTful API. The first could be extended to cover LE rate limits, the latter could be used to hook KeyChest to Certbot to improve data for KeyChest notifications.
My feeling is that what you're after is to learn about config. errors and 1 or 2 days delay is not such a big issue. In that case just an extension of notifications should work reasonably nicely.
It sounds to be like another argument for an independent monitoring. I understand that most sys admins wouldn't hear about it as a cron integration solves most of the problems, but I still think people should use end-to-end monitoring, which tells you that a new certificate is indeed on your server.
(but yes, I do agree such monitoring may be an overkill if you just have a couple of servers with friendly users who ping you as soon as they go down)
