Shared Hosting Providers (Digital Ocean, Linode) IP Blocks Hitting Rate Limit - Locking out Multiple Users

So I’ve just rebuilt my VPS and I now seem to be hitting the rate limit when trying to re-request an LE cert.

The server is Ubuntu 17.04 using the default distro client, and I’m calling it with:

/usr/bin/letsencrypt certonly --email <email>@psfshr.uk --agree-tos --standalone -d fastnet.psfshr.uk -d www.psfshr.uk -d psfshr.uk

And I’m getting back:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Starting new HTTPS connection (1): acme-v01.api.letsencrypt.org
An unexpected error occurred:
There were too many requests of a given type :: Error creating new registration :: too many registrations for this IP
Please see the logfiles in /var/log/letsencrypt for more details.

The previous install of the server had a cronjob which ran the following on a weekly basis:

/usr/bin/letsencrypt-auto renew --standalone --noninteractive --pre-hook "systemctl stop nginx; systemctl stop postfix" --post-hook "systemctl start nginx; systemctl start postfix"

So I’m not entirely sure why I’m getting the rejection, as worst case this server should have 1 cert renewed in the last month or so, and maybe 1 or 2 other certs in the last 90 days where the server has been rebuilt previously.

This server is a non-shared (i.e. private) VPS with a public fixed IP so there’s no other certs getting requested from this IP.

Any thoughts?

Hi @psfshr,

Is it possible that your server IP address is shared with a very large number of other people?

Or, is it possible that you somehow regularly delete the entire contents of /etc/letsencrypt/accounts or /etc/letsencrypt?

If not, would you be willing to post the log file that the error output mentioned?

Does your server have an IPv6 address? For IPv6, because it’s common to receive large blocks of IP space, we rate limit by the /48. It’s possible someone else, or many someones, are creating lots of registrations from neighboring IPs. However, the Accounts Per IP rate limit is very high (500 per 3 hours), so try again in a few hours? If you’re willing to share your IP I can take a look at nearby account creation activity.

I’m checking our logs to investigate whether someone from a nearby IP, but strangely I don’t see any rate limit events from either the IPv6 address or the IPv4 address I get from host psfshr.uk. Can you tell me whether you have changed IPs recently, and whether those are the addresses we should expect those requests to have come from?

2a01:7e00::/48 is a large hosting company (Linode) containing thousands of customer VPSes…

Ok, I found the new-reg request that @psfshr got rate limited on. It was on a different IPv6 address, one belonging to Digital Ocean. We’ve had at least one other subscriber on Digital Ocean report issues. Earlier today, I tracked the problem to a single misconfigured VPS that was hitting us constantly with new-reg requests and eating up all the rate limit for its neighbors. I’ve contacted the owner and asked them to fix their config, and we’ll be blocking that IP to stop the problem for now. We’re also talking about ways to mitigate the impact of similar issues in the future.

Thanks for reporting, @psfshr!

2 Likes

I’ve run in to this now too. I just spun up a brand new Digital Ocean box, pointed a domain at it, ran our ansible provisioner which installs Caddy, and Caddy is failing to start with a message in the logs saying

registration error: acme: Error 429 - urn:acme:error:rateLimited - Error creating new registration :: too many registrations for this IP

Even leaving it alone for 3 hours and then running the provisioner again produces the same error.

I just hit the same problem on a brand new server from scaleway, three hours after creating it.

After reading the above, I did this to turn off ipv6:
sysctl -w net.ipv6.conf.all.disable_ipv6=1
sysctl -w net.ipv6.conf.default.disable_ipv6=1

Then letsencrypt worked fine.

fwiw:
ipv4 51.15.65.15
ipv6 2001:bc8:4700:2300::15:609

1 Like

Apologies for the trouble. As @sandhawke and others have mentioned disabling IPv6 when you register your account initially is one short term workaround. IPv6 connectivity can be reenabled afterwards.

Please follow https://github.com/letsencrypt/boulder/issues/2738 for a better long term fix.

Thanks @cpu - disabling IPv6 did indeed solve the problem.

1 Like

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

For folks that land on this issue from a Google search the underlying problem was addressed with https://github.com/letsencrypt/boulder/pull/2782 and two separate limits. It should no longer be required to disable IPv6 to avoid this rate limit if you are on a provider with misconfigured neighbours.