Let's Encrypt rate limits and alternative solutions

The PSL is an initiative by Mozilla to list all the “public suffixes”. Public Suffixes are the domains people can register - which is a mixture of the ICAAN top-level domain extensions (com, edu, us), some country-level subdomains (ny.us, com.uk), and platform/service providers (*.compute.amazonaws.com [amazon cloud], *.customer-oci.com [oracle])

It’s used by a lot of software systems to determine “registrable” domains, but also by Browsers to enforce cookie policies and by the CertficateAuthorities to determine registration rules and guidelines.

In the case of LetsEncrypt, if a (sub)domain is listed in the PSL, then each domain under it is considered a registrable domain and the rate-limits will apply to that. The PSL updates periodically, and LetsEncrypt updates their Boulder instance against it periodically as well - so there is a bit of delay from PR (against PSL) to deployment (by PSL) and recognition (by LetsEncrypt).

Assuming your per-customer dashboards is on the web at customer-abc.dashboards.example.com

.com is the Public Suffix entry, so the per-domain limits apply to the registrable domain – example.com. You will only be able to provision 50 certs a week with the {customer-id}.dashboards.example.com naming system.

However if you have dashboards.example.com accepted into the PSL, then dashboards.example.com is considered the registrable domain, and the per-domain limits will apply to that instead. Consequently, you will be able to provision infinite certs a week with the {customer-id}.dashboards.example.com naming system – but only 50 certs under it (such as foo-01.dashboards.example.com through foo-50.dashboards.example.com)

2 Likes

Thanks for the explanation! I’ll make a PR for my site. Well I guess I’ll just wait to see what I get back from the rate limit increase request. I’d love to use Let’s Encrypt if possible. Fingers crossed.

you should probably use wildcards AND a private CA. so you can have your reverse proxy/load balancer use the wildcard when it communicates with the internet, but you have your own CA with no limits at all for the traffic between load balancer and user vps – I think you can even use mutual tls to authenticate on both ends.

Traffic must be encrypted to the backend node. I can’t do decryption in the load balancer and re-encrypt to the node. So the backend node’s certificate must be internet facing and be a globally trusted certificate. I wish I could use a private CA, but sadly that’s not an option

1 Like

Yes, but wait. If you provision certificates for each and every client vps, you will inevitably have access to the private key.

The only way for you not to have access to the private key is for your clients to provision the certificate themselves, and even then, you can always exfiltrate the key from ram (always) or storage (if unencrypted).

Keys are encrypted on disk and live in memory for a very short time on a server that I can’t SSH into. The keys/certs are created and provisioned by an application (never manually). So it’s the most locked down you could get given the circumstances. If I decrypt traffic at LB and re-encrypt it back to the node then I most definitely have access to the traffic whether I want to or not.

Yeah, but you could access in these circumstances as well. You control the system and its behavior, the keys are there.

The only way for this stuff to actually be zero-knowledge would be to have the key on the client, not on the server; so that you only have encrypted data and no keys, and your clients decide who can access the data by decrypting and re-encrypting with more (or fewer) public keys.

I assume it’s not a matter of trust but a matter of you wanting to be shielded from your client’s handling of sensitive (ie: medical, legal) data.

I see you as my brother in paranoia @9peppe. :slightly_smiling_face:

There are several “full-condom” protocols that I’ve worked with in research, but they rely on keeping the final decryption key in the hands of the customer. It’s a triangular protocol that basically uses the insecure hosting as storage while passing the decryption key to the customer through a separate, secure session.

Haha paranoia is healthy for everyone. At least on the internet :wink:

But yes, being as locked down as possible is my goal and having the possibility of sensitive data in plaintext anywhere will certainly not achieve that.

Given that you are treating each VPS as an isolated tenant, I feel like this would most simply be solved by getting a rate limit override to certificatesPerName and registrationsPerIP.

That way, you could simply issue the certificate as part of cloud-init/cloudboot or whatever. No complicated architecture required. Hopefully you are not doing rapid destruction/creation of machines.

Applying to the PSL may or may not be appropriate depending on what exactly your application is, but I can tell you that one of my VPS providers does have their auto-generated VPS domain (.bnr.la) on the PSL and I never have any issues generating certificates for those hosts.

3 Likes

I 100% agree with this.... but because of all the security concerns, I think the PSL list should also be explored because that will affect the cookie policies and other browser security concerns.

2 Likes

It is neither. The TLD would be, e.g., .com, and no, there's no limit on the TLD--the limit is per registered domain.

This could be a case where you should consider having your domain added to the public suffix list, or perhaps a case where an exception to the rate limits could be made. Edit: Just saw that both of these suggestions have already been made.

3 Likes

@danb35 you’re correct. I said that wrong. I’ll edit that.

@jvanasco so are you guys saying If I get *.dashboard.example.com in the PSL, then I could issue 50 unique certificates a week for customer1.dashboard.example.com?

Yes! LetsEncrypt will treat dashboard.example.com as if it were a registrable domain, so each subdomain can have 50 certificates weekly (there is a ratelimit of 5 unique sets of FQDNs on a cert per week, so you can have certs for customer1, and then 45 more that include customer1 but also at least one other domain
. Also, the major browsers will respect this in terms of cookie domain settings - which will help safeguard the data (see Learn more about the Public Suffix List)

An example of a vendor doing this is Oracle: https://github.com/publicsuffix/list/blob/master/public_suffix_list.dat#L11112-L11118

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.