Large University Integration / ACME proxying

[humbug]

I’m looking into adding Let’s Encrypt as a CA offering at our University (R1 research university). My service is part of University-wide IT and we provide free SSL certs for any University-owned domain name. Right now we get most of our web SSL certs through Comodo with an agreement through InCommon/Internet2. We’re looking at Let’s Encrypt to reduce our reliance on a single vendor.

We currently have about 4000+ active certs, mostly under the our university.edu namespace. I’m pretty sure we will need a rate limit exception but don’t expect to have a problem getting one.

What I’m trying to figure out is doing a domain vs account rate limit exception, and if I do the latter, how to funnel all requests through a single account in a distributed environment.

Our computing environment is heavily distributed–we provide a lot of central services but it’s more or less the wild west out in departments–they can set up whatever they want, however they want, if they aren’t causing problems. There are good and bad things about this approach.

With the domain rate limit approach, it’s easy to implement. I request a rate limit increase, and folks out in departments can just use ACME clients to request certs. Downsides:

-We currently have some institutional access control over who is allowed to request certs for domains. We’d lose this but I’m not sure it’s a dealbreaker
-I can envision a scenario where a person or groups of persons may accidentally or inconsiderately request enough certs to burn through even our expanded rate limit

If I chose to rate limit by account and funnel all Let’s Encrypt requests through a central portal, that’s a lot more work but I can see the following benefits:

-Maintain institutional access control on cert issuance
-Implement rate limiting at the institutional level so an accidentally bad or inconsiderate actor will just burn up their own rate limit (e.g. someone forgets to increment the counter in a loop, or isn’t using durable storage for their container keys and spins up 100 containers in one day)
-Account rate limit covers any other domains that institutional users want to use (including medical centers, etc.)

What I’m having trouble wrapping my head around is how I would implement this. I’ve been mulling over the idea of proxying ACME cert requests…tell me if this sounds crazy:

Run our own acme server–have clients on campus request certs from it. The server would implement our own access control and rate limiting, rewrite and resign the ACME request using our central account, and submit it to let’s encrypt. The proxy server would have to return the HTTP token to the client, and then let’s encrypt would do DNS/HTTP validation as normal against the original client.

The integration guide also mentions a central validation server, so I wonder if I could have our customers redirect to that server and do the domain control validation there, but still rewriting the request with our central account info as described above.

I’ll add we control DNS records for most of our domain names so that’s another option for validation–I’d like to maintain HTTP as an option since maybe 10% of customers are doing their own thing for DNS.

Am I making it way too complicated? I’m also interested in other solutions…I’m in the very early stages of evaluating this.

Thanks!

[stevenzhu]

(Disclaimer: I have no relationship to LE except a supporter, so I don't represent their opinion )

Hi

@humbug

We currently have some institutional access control over who is allowed to request certs for domains. We’d lose this but I’m not sure it’s a dealbreaker

I'm not sure by access control on request certs..
(Do you mean that you don't want all person has control on subdomains apply certificate by themself ? Or you are afriad that one person can apply domian certificate to root domain which he don't own? Abusing?)
(CAA might be a helpful tool to control abusing though)

@humbug

Run our own acme server–have clients on campus request certs from it. The server would implement our own access control and rate limiting, rewrite and resign the ACME request using our central account, and submit it to let’s encrypt. The proxy server would have to return the HTTP token to the client, and then let’s encrypt would do DNS/HTTP validation as normal against the original client.

You can run ACME server, for sure... However you don't have an public trusted CA certificate (which all certs that issued under this server will not be valid unless visitors install your root CA)
(I don't think LE will allow you to control their certificate...)

@humbug

The integration guide also mentions a central validation server, so I wonder if I could have our customers redirect to that server and do the domain control validation there, but still rewriting the request with our central account info as described above.

You don't need to point all domians there.. the customer just need to point _acme-challenge (the subdomain) to your validation domain. (And add txt record to your validation domain)

My thoughts are (disclaimer: just suggestion based on the info you provided)

Set up an web based php agent (for insurance), just like sslforfree.com (but use your University's account and have a "order code" to control issuerance. You can provide options like http or DNS validation, user just need to validate the order code first to proceed (enter domain name or other information)

Thank you

Anyway, I think a LE staff member could help you more.. (calling @schoen)

[humbug]

Access control: Currently our network group maintains a list of users who are authorized to perform administrative functions for DNS name (request changes, etc.). We also use this list to authorize the same people to request a cert. I think there’s a strong argument to be made that if someone has already set up a physical server, gotten a static IP and DNS name, and can demonstrate control of the domain name via HTTP challenge, we don’t need to do an additional authentication operation. That’s why I say this isn’t a dealbreaker. I haven’t gotten to the external account binding section of the v2 spec yet, but it sounds like it might be possible to continue to do some sort of authn with ACME if I want to.

Running our own ACME server: The idea wouldn’t be to sign the certs on this server, but to proxy requests to the LE ACME server, rewriting the account fields and resigning the request as we do it. I’m still working my way through the ACME v2 spec, but I haven’t seen anything yet that indicates this is impossible.

web based agent: We actually have an in-house solution like this already for requesting certificates from the Comodo API and our local CA. I could shoehorn LE into this pretty easily, But I really want to offer an automated solution, and ACME seems like the way to go.

[webprofusion]

If all groups are in control of their own servers and content (accessible over http) then they can just run let’s encrypt as normal using the http-01 challenge - you have no control over that (they can presumably already do it if they tried).

However they can’t request a wildcard, just the domain/subdomain they actually operate on their web server, and only if it’s public. If they are not public or they want a wildcard for their subdomain then they can only use the DNS challenge, so then they need to create a TXT record in their (sub)domain.

As you suspected, you will hit the rate limit pretty quick (20 certificates per week per primary domain) so you’d need a rate limit increase for that domain.

As well as Let’s Encrypt, you also have the option of services like https://alwaysonssl.com/

[schoen]

Rate limit increases have very often been granted to universities, so that should be possible to do.

[humbug]

No comments on the feasibility of my hare-brained ACME proxying scheme?

[schoen]

I think it might be doable. I’m trying to remember whether there’s anything in the ACME protocol that prevents this kind of proxying, and I don’t immediately recall anything.

You can’t easily stop your end users from also getting individual certificates, though, unless you can set a CAA record forbidding issuance, and control the DNS to prevent the users from adding more-specific exceptions, and then create a more-specific record authorizing issuance every time you want to issue anything.

[humbug]

Thanks for the feedback. I’ll see about giving the proxy a try. If I can find the time…

I don’t have any intention of stopping users from getting their own certificates if they want–that would be antithetical to our distributed model (or the “just enough rope to hang yourself model” as I sometimes call it). But the rate limiting and making sure everyone plays nice has to be addressed at the institutional level. The distributed model is great in a lot of ways, but it can get messy when you’re sharing a namespace with many other groups and they aren’t necessarily coordinating their use of it.

The departments are welcome to request certs for university.edu and various subdomains on their own, but if/when they hit the rate limit then we can offer a central gateway that funnels through an account with a higher rate limit. Or that’s the mental model I have, anyway.