Rate limit - is this effect really intended?

brunni · October 23, 2020, 11:59am

I rented a root server from german provider IONOS, which assigns each of them a subdomain of online-server.cloud. Unfortunately I cannot use that subdomain because "too many certificates already issued for: online-server.cloud". It seem the last cert for online-server.cloud was successfully generated 2019-05-07: https://crt.sh/?q=online-server.cloud

The page about rate limits only mentions weekly rate limits so I wonder what is going on here and whether it is intended

Osiris · October 23, 2020, 12:03pm

I can reproduce the error indeed and looking at another CT-log aggregator, I can confirm there haven't been any certs issued for that domain recently:

@lestaff Any idea what's going on here? I'm at a loss!

petercooperjr · October 23, 2020, 12:38pm

This probably isn't "the solution" to whatever's going on (since it doesn't seem to be immediately caused by other users using up the rate limit first), but if everybody under online-server.cloud is a different "site" run by different entities then that hosting provider should see about getting it added to the Public Suffix List. Then each subdomain would be subject to its own rate limits (in addition to each site not being able to set cookies for other sites and so forth).

_az · October 23, 2020, 4:47pm

I don't know what's wrong with the othe CT aggregators, but:

is surely not the case - plenty of certificates have been issued, even today.

_az · October 23, 2020, 4:54pm

It might have something to do with this response from Rob Stradling:

If there are "too many" certificates for a given search term, the crt.sh code has to limit the number that it will process, for performance reasons. Although I labelled this as "a temporary hack", I'm no closer to devising a solution I'm afraid. ISTM that postgres's Full Text Search just isn't geared towards sorting or paginating large result sets.

Indeed when you search for a specific domain, you see the recent certificates:

It's curious that they both suffer from the exact same issue and workaround.

ezekiel · October 23, 2020, 5:05pm

Reading the code, yes this is intended. Boulder uses "Effective TLD Plus One" for the purposes of rate-limiting. I would guess this is because there are essentially infinite combinations of sub-domains. Specifically, the "Effective TLD" here is .cloud, and the "Plus One" is online-server.

github.com

letsencrypt/boulder/blob/4c7be7041b6b7b9f6d30f223d87aa943356eb926/ra/ra.go#L1265-L1283


// domainsForRateLimiting transforms a list of FQDNs into a list of eTLD+1's
// for the purpose of rate limiting. It also de-duplicates the output
// domains. Exact public suffix matches are included.
func domainsForRateLimiting(names []string) ([]string, error) {
	var domains []string
	for _, name := range names {
		domain, err := publicsuffix.Domain(name)
		if err != nil {
			// The only possible errors are:
			// (1) publicsuffix.Domain is giving garbage values
			// (2) the public suffix is the domain itself
			// We assume 2 and include the original name in the result.
			domains = append(domains, name)
		} else {
			domains = append(domains, domain)
		}
	}
	return core.UniqueLowerNames(domains), nil
}

However, I would very much encourage you, as an IONOS customer, to send the Overrides documentation along to them https://letsencrypt.org/docs/rate-limits/#a-id-overrides-a-overrides so they can request a limit increase.

jsha · October 23, 2020, 5:26pm

@_az Yep, I came back to post this and found you'd already posted it, with an excellent citation. Thanks!

I can confirm that many certificates have been issued recently under online-server.cloud. They just don't necessary show up in that particular crt.sh search.

Regarding the Public Suffix List: I have been trying not to encourage folks to get added to the PSL lately. The PSL gets distributed to all browsers, and making it bigger is a burden to the whole web. Yes, services that hand out subdomains to anyone fit the technical definition of what should be on the list, but there are so many services that meet those criteria that if they were all added to the PSL it would be way too big. The web needs a better solution but so far no one has been able to come up with one (other than eliminating cookie setting on parent domains). The upshot is: when sites get added for the purpose of changing how Let's Encrypt apportions rate limits, that's a burden on a shared resource.

We have also dramatically increased our rate limits in recent years (50 per registered domain per week) and have streamlined our process for handling rate limit increase requests, which hopefully reduces the incentives to get added to the PSL.

petercooperjr · October 23, 2020, 5:44pm

Hmm, I was suggesting getting added to the Public Suffix List because I thought that it was the "correct" place for such things, not just for things like rate limits but other things as well that might care about if subdomains were actually supposed to be "related" or not. My apologies if that was poor advice.

jsha · October 23, 2020, 5:59pm

Yep, your understanding is correct! The complicating factor is that there are many many domains out there that "should" be on the PSL, but including them all would make the list unusuably large, so the web kind of stumbles by with the status quo where some domains add themselves to the PSL and some don't.

The main thing the PSL governs in terms of browser security is cookie setting, and that's changing a lot lately, so maybe there's light at the end of the tunnel.

Osiris · October 23, 2020, 6:25pm

Ah, nice, a third aggregator! And here I was thinking with n=2 there must be something wrong.. Not thinking even mighty Google might be erroneous.

rg305 · October 23, 2020, 8:54pm

And there is always the other options of:

using your own domain name
using a name from a domain, via DDNS provider, that is already on the PSL

No line - No waiting !

[&2* readers: Get involved; Be heard. It starts with: if you read something you like, then like it ]

system · November 22, 2020, 8:54pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.