Possible new feature: paused ACME accounts

[jsha]

@petercooperjr

I don't have a sense of the scale of the problem you're trying to solve here.

Good point, I didn't elaborate on the problem statement! The main issue I'd like to solve is this: about 80% of HTTP-01 validations currently fail: currently approximate 110 rps of errors. That means we are spending a lot of resources (storage, bandwidth, CPU) on unneeded work. But also it means it's hard to evaluate when slowness in various validation systems (particularly DNS) are due to a problem on our end vs a particularly large influx of traffic from someone with failing validations.

@petercooperjr

what portion of the time one of these "zombie clients" ends up starting to have valid orders again?

I haven't done this calculation yet, but this is a good idea.

@petercooperjr

It sounds like you're kind of talking about another rate limit similar to the existing "failed validations" limit, where enough failures mean that additional requests won't even get tested. But I'm not understanding why you'd be okay in general with 5 failed validation per hour, but now want to restrict on 5 failed validations in a week.

This is a good point, that having multiple levels of rate limit would solve this in another way. For instance, one can imagine adding to the "5 failed validations per hour" a "25 failed validations per week" and "50 failed validations per month".

A couple of reasons to prefer setting a bit on the account:

• In a lot of cases, the client will really never succeed again and is completely forgotten. Spending resources on even 50 failed validations per month adds up, and doesn't benefit anyone.
• If someone does notice that their account has been paused, they can unpause it right away rather than wait for the rate limit to expire.
• Right now, calculating rate limits is somewhat expensive for us, but we do hope to improve on that.

Also keep in mind this is not quite the same as the failed validations limit: It's failed validations, combined with a long period of no successful issuance. It might also make sense to express the threshold over a long period, for instance 100 failed validations over the course of 90 days and no successful issuance in 180 days.

@petercooperjr

How many failed validations per week are typical of these zombies?

There are a couple of common cases. There are some that attempt renewal once every day. These are not a problem on their own, but when there are many of them, particularly if they are all using a stock VM image with a cron job set to a particular time of day, they can be noticeable.

Then there are some with buggy software that goes off the rails and hits us many times per second. Right now we block these when it gets bad enough, and we usually try to notify the maintainer of the software. But this is a very manual process. And some buggy clients make it hard to set an email (or discourage it by not showing how in the examples), so for some of the offenders we have no way to get in touch.

@petercooperjr

Do we have any understanding of how many clients (especially those integrated-with-hosting-solutions type where when they ask questions here they're confused by the "What client are you using" question) surface these kinds of errors to the end user (assuming that there's an end user to surface them to at all)?

I don't. Though in those cases the hosting provider in theory is in charge of handling errors. There's definitely some nuance here in whether the hosting provider creates an account per-user or on account for all their users.

@petercooperjr

You mention that for some problems you reach out via email, I'd think that adding an automated email to the contact on file would make sense at the same time as (or instead of) pausing the ACME account, to let them know that they've got some system somewhere (and tell them the IP, user agent, or maybe other headers), would work even better to alleviate problematic traffic.

This is a good idea! We should do this too.