Rate limiting at an educational institution


#9

Hi @jhawkins! Yep, the university model of deeply nested and highly delegated subdomains is definitely something we’ve thought about and would like to figure out the best approach to. Maybe I can flip the question around and ask: What would be your ideal way to handle the situation be? At any given threshold, it’s likely that some of the time, someone in one of the departments will burn through it. Consider, for instance, someone in the CS department experimenting with Let’s Encrypt who sets up a client to issue certificates for each of 1-1000.mydomain.uni.edu. Is there a good, scalable way of “walling off” different entities within the university so they can’t chew up each others’ rate limits? I’ve been trying to think of some way to use CAA DNS records to indicate boundaries, but haven’t come up with anything conclusive yet.


#10

What about having a much higher limit for .edu domains (since there are so few and they are more important than your average .com) and a set limit for each subdomain of a .edu domain (including subdomains of that)?


#11

@jsha Indeed! It is a difficult puzzle, and its wonderful to hear the team is working on this.

After thinking about this for a number of hours - the low hanging fruit options do appear to be either radically upping the limit for .edu domains (although ‘radical’ for one institution may be a drop in the bucket for another) as @xxjfe mentions, or whitelisting .edu addresses altogether. I will say theres some merit to the latter approach as .edu’s are neither distributed nor administered like traditional TLD’s so having a more open approach to them would have a certain logic.

Assuming a rate-limit option stays in place, I do have one question. Is there a way to run a query in the LetsEncrypt client to provide a log or manifest of certificates requested against a respective second-level domain? If it would fall to the campus IT administrators of university units to come up with policies for accountability - it would be difficult to make headway without having a toolset to know where the quota was being diverted. Even then, I don’t know how you would handle students.

Really, the quota system just doesn’t work for our situation - but that may just be the way the cookie crumbles, as they say. :wink:


#12

Not in the client, but all certificates issued are logged to certificate transparency logs which can be searched at e.g. https://crt.sh/ or https://www.google.com/transparencyreport/https/ct/. There’s also this unofficial script which scrapes crt.sh to provide a useful analysis.


#13

@jmorahan Great info, appreciate the share!


#14

Just to add we have also hit this amusing yet frustrating situation as a i started to add letsencrypt SSL certificates to a bunch of our public web servers.

> There were too many requests of a given type :: Error creating new cert :: Too many certificates already issued for: ox.ac.uk

There must be hundreds of subdomains here all under slightly different control from Departments, Colleges and Central IT and yet no one can now register any more SSL certificates for 7 days in the whole of Oxford university. It would be nice to see a different approach for educational institutions both under .edu and .ac.uk here in the UK.


#15

Hello all, just to add a “me too” here from the the University of Edinburgh (ed.ac.uk), School of Informatics (inf.ed.ac.uk), in my case.

This issue is likely to stop us from using letsencrypt in any meaningful way, as we could very well be prevented from obtaining certificates for crucial public-facing websites due to the entirely legitimate behaviour of other parts of the University, over which we have no control.

Cheers
Toby


#16

@jsha Hello once again! Just wanted to report back a latest finding - it appears, at least in our case, renewals are being counted against our rate-limit so we have begun to have certificate renewals fail. I was under the impression renewals were exempted from the 20/wk. limits?

Also, we are very eager to hear if any progress has been made on making new policy regarding university institutions.


#17

We’ve seen one or two other examples on this site of people saying their renewal tripped rate limits that shouldn’t apply for a renewal. If that’s true it would likely be a bug in Boulder (the Let’s Encrypt ACME server)

Hopefully, unlike some of those you’d be willing to share the FQDN(s) affected and the output of a failed run? With luck that ought to make it possible to either pin down the bug in Boulder (hence you’d get a fix, hopefully sooner rather than later) or figure out what you did wrong that caused your certs not to count as a renewal.


#18

Thanks for reporting! We have an open issue and a pull request to fix the behavior: https://github.com/letsencrypt/boulder/issues/1925.

I’m afraid we don’t have any news about rate limiting policy for universities.


#19

Thank you for the quick response - fantastic to hear news that a fix is in the works!!


#20

Hello all! Wanted to provide a quick update based on our experiences.

If you happen to run a cPanel server shop, the latest release (58) includes support for what they are calling AutoSSL - which enables automatic, free provisioning of DV certificates via a provider partnership they have set up with Comodo. The key size is 2048 but depending on your use case that may not be a huge issue. Just wanted to share, as they have not instituted Lets Encrypt style domain rate limits.

The cPanel AutoSSL service does allow for adding additional certificate providers, so support for Let Encrypt is rumored to be released soon. That said - the baked-in cPanel/Comodo default provider is still going to be the only real option for institutional academic use.

Hope this helps someone out there!


#21

AutoSSL currently supports Let’s Encrypt - it’s a option in cpanel / whm - see https://features.cpanel.net/topic/provide-support-for-lets-encrypt-automated-certificate-management-ssl

I use it on a number of servers, and it works really well.


#22

Great add Serverco! It should be noted that cPanel’s AutoSSL Lets Encrypt support has very recently left closed beta and is not available unless you manually run a script (see link in post above) to add the requisite patches to the server host you administer. For most users, it will not be released as a selectable option within WHM until version 60:

From cPanel: “If all goes well, we will be adding this plugin to the list of plugins provided in the WHM interface in cPanel & WHM version 60.”

In any event, the Lets Encrypt domain rate limits are still honored - so for those of us who live in the site.lab.dept.uni.edu world there remains zero point in using it over cPanel’s default Comodo provider which does not utilize such limits.


#23

@jsha Has there been any new discussion in the last 7 months about rate limiting for universities?

University of Maryland used to issue free certificates for any umd.edu subdomain from Digicert to anyone on campus, but it seems they may no longer be doing it, so the dozens of us who run our own servers on campus and are trying to use LE are running up against the rate limits very quickly.


#24

@kohenkatz, universities have fairly commonly gotten exemptions from the rate limits, but the request needs to be submitted by a responsible party at the university.

https://letsencrypt.org/docs/rate-limits/#overrides


Rate limit increase for university - clarification request
#25

@schoen This is welcome news!

We filed an exemption request yesterday for the University of Alabama but have yet to receive any response. Fingers crossed we will hear back soon, as we would love to resume testing LE certificate availability within our College’s infrastructure as an alternative to cPanel’s default Comodo SSL.


#26

4 posts were split to a new topic: Question about educational institution rate limit adjustment request


#29

A post was merged into an existing topic: Question about educational institution rate limit adjustment request


#30