What are the risks with a multi sub-domain tenants?

Help
#1

Here is my scenario:

I will be proving hosting to various clients. I will be provisioning their servers with a default domain:
Example

When clients create websites on their server it will auto generate sub domains:
Example

My thought is to setup a wildcard ssl for each server domain on server creation:
Example

  • *.srv-1.mydomain.com
  • *.srv-2.mydomain.com

That way as clients create websites on them no matter how many they create they will have a working SSL.

However, clients will have SSH access to their servers. So they could issue new certificates and validate the challenge on their server domain: Ex srv-1.mydomain.com

So here are my questions:

  1. What security risks or rate limit risks can you see with this?
  2. Can a client with access to srv-1 cause issues with a client on srv-2?
  3. Since my main website is mydomain.com could they also cause issue with me and my website?
  4. Is there a way to restrict the additions or certs created for specific domains? I only want to allow 1 ssl to be added to *.srv-1.mydomain.com and not allow any others to be created. (I will allow the clients to add their custom domains to the servers and all them to add ssl's to their custom domains.)
  5. I am concerned that one of my clients could go crazy and create tons of certs on their domain srv-1.mydomain.com and that would hit the rate limit for myself and other clients which would prevent me from adding new servers for new clients and providing them with an SSL. Is my concern correct?
  6. Is there a better way to do this?

Thanks

1 Like
#2

Root access?

There exists a rate limit "certificates per domain per week", currently 50 I believe. If a client on srv-1 would issue 50 certs (perhaps due to a buggy client) then you or any client won't be able to have a new cert issued, until the certs per week drops below 50 again.

What kind of issues?

Renewals are exempt from the certs per domain per week rate limit. Assuming you already have a certificate and you don't need to change its hostnames, you should be able to renew, even if a client makes your domain hit a rate limit.

You might be able to do that with a CAA record. I'm not sure if the CAA ACME extensions (about allowed ACME accounts) are in effect yet, but even with a basic CAA record, you'll be able to prevent anyone to issue any cert. Including yourself by the way.. But I assume you're the only one with DNS access?

Also: do all servers have their own IP address?

2 Likes
#3

@Osiris Thanks for your quick reply.

Root access?

Yes, think of my hosting as more of a AWS style.

What kind of issues?

I guess mainly rate limits which you answered, but could they also revoke certs on a different domain. Could they potentially revoke any certs I have on "mydomain.com"?

You might be able to do that with a CAA record

Yeah, that is what I have been reading, but information on that is pretty vague and like you said is kind of new so I am not sure I want to trust it.

do all servers have their own IP address

Yes

I am thinking I need to change their domains from "mydomain.com" to "mydomain.dev", etc that way they couldn't affect anything on "mydomain.com", but they could still effect other clients servers. I am still looking for a solution to that.

1 Like
#4

I'm not entirely sure, but as far as I know you'll need to prove ownership of all the hostnames included in a certificate, before a different account than the one the original cert has been issued with can revoke that cert. And because they won't have access to your apex domain, that shouldn't be possible.

If multiple users have a subdomain of a certain domain name, the domain should be included in the Public Suffix List to prevent the users to set cookies for domains other than their own. Without the PSL it would present a potential security issue. Let's Encrypt also uses the PSL to determine the public suffix to which the rate limits are counted. While the latter isn't any reason to add a domain to the PSL (and will lead to an inclusion request to be denied), the former security risk is.

2 Likes
#5

Well, I have been looking into CAA records and it doesn't seem to really help here.
It looks like I can restrict users from creating LE certs using a CAA record, but that would also prevent me from using it to create new certs for new servers.
Is that correct? I was thinking there was a way to request a specific acme account key that was required to be used.

Any thoughts there?

1 Like
#6

I think you want RFC 8657. Boulder, the ACME server used by Let's Encrypt currently supports those extensions, but needs to be enabled in the settings. And I'm not sure if it's enabled.

2 Likes
#7

@Osiris Thanks
There was a lot of talk a year ago that validationmethods were added to staging, but haven't made it to Production yet, but I am having difficulty finding info that it is in Production. So I might just assume that it is and try it out.

Looks like the best solution for me will be to require validationmethods=dns-01 so the clients cant validate based on http-01. I will have to test this to make sure it works.
Referencing: https://tools.ietf.org/html/rfc8657#section-4

I might include the "accounturi" as well, but I am also concerned if I loose my key or what I need to do if something gets access to it or they have my accounturi, but I know for a fact that they wont have access to my dns :slight_smile:
So I might use a combination of both, but not sure. I will definitely change the tld to something different that is only used for the wildcard sub domains and is not connected to my public domain.

PS these generic domains are just so they can get started developing, but it will be documented to them that they should not be used in production and that they should add their own custom domains. I will probably have some sort of Password protection on accessing those domains publicly so they will need to relay those credentials to their internal team when testing, etc. Still figuring it out.

2 Likes
#8

You could also look into containing them within a subdomain of your domain:
*.srv-1.dev.mydomain.com
*.srv-X.dev.mydomain.com

1 Like
#9

@rg305 That is what I am currently doing. The issue with that is they could issue certs with a http-01 challenge for various sub domains like "test.srv-1.dev.mydomain.com" which will use up the rate limits for "mydomain.com". Let's Encrypt uses the top most domain for the rate limits. So in order to prevent that I would need to change the top level domain to something else for my website. This will help me, but not my clients. I have other sub domains like "app.mydomain.com", "api.mydomain.com", and "www.mydomain.com" and I wouldn't want the renewals to fail because one of my clients started to go nuts creating certs.

1 Like
#10

As you mentioned, by changing the TLD, you merely pass the problem to a subset of users (all minus you).

The solution for all users to that problem was covered with:

This was also explained [that limit doesn't effect renewals]:

1 Like
#11

@rg305 @Osiris Thanks I totally missed the renewals are not counted as part of rate limits. That's good to know.

Also I had a look into the Public Suffix List a bit more and have a better understanding of it now.

I just purchased a .host to use for all the client servers.
I will have to look into having "mydomain.host" added to the Public Suffix List.
It seems that alone will address any issues with rate limits crossing over from one client to the other along with it being more secure because of cookie issues, etc.
I will probably still add a CAA with relevant validationmethods and accounturi when I understand that a bit more just for extra protection.

2 Likes
#12

Hi @geddedev

not completely.

You can use issue and issuewild.

issuewild allows wildcards. If no issuewild is found, issue allows all.

But a combination

srv-1.mydomain.com issue ";"
srv-1.mydomain.com issuewild "letsencrypt.org"

allows creating wildcard certificates via dns validation and blocks non-wildcard certificates.

If customers don't have DNS access, they can't create certificates via http validation.

4 Likes
#13

@JuergenAuer Thanks
That will help a lot.

Do you know if validationmethods=dns-01 is supported yet and if so what would an example look like. Also is there a preference of that over issuewild?

Thanks

2 Likes
#14

PS. Digital Ocean doesn't support the semicolon feature.

DigitalOcean DNS does not support the following CAA record features:

  1. Blocking anyone from issuing certificates by sending a semicolon ( ; ) in the value.

So I might just issue it by a fake name or something else.
Anyone have any thoughts?

1 Like
#15

Oh, thanks, good to know.

Yep, you can use any fake name you want.

Existing CAA not matching the own, published CAA value -> certificate creation isn't allowed.

3 Likes
#16

Four things:

  1. Like @Osiris noted, you need to use the Public Suffix List to at least handle the browser sandboxing concerns.

  2. I didn't know about the combination workaround that @JuergenAuer noted. You could likely implement that by using another DNS provider.

  3. In terms of your certificate design, I would just make a wildcard for *.example.com and deploy that to all the servers OR just install it on your gateway systems and decrypt SSL there.

  4. Blocking domains, the CAA record should mostly work. You can use pre/post hooks to set and unset the records. There would be a race condition with the small window of time during which you don't have a CAA record, and customers could potentially abuse that - but the odds are pretty low.

IIRC, LetsEncrypt CAA checks work like this:

  1. CAA is checked during Authorization. If that passes, there is a ~7 hour window where the response is cached and another check will not be required for issuance.
  2. If more than ~7 hours has passed since Authorization, the CAA record is checked during the finalization process. The response is not cached.

The RFC has an 8 hour limit. I think Boulder implements it with about a 7 hour limit, just to be safe. Assuming your DNS provider does not have internal caching going on, and updates the CAA immediately... it will probably take 30-60 seconds to disable the CAA record, request/validate/finalize your certificate, and reenable the CAA record. If you wait at least 8 hours between running this process, you won't risk the second window.

It is possible for clients to make a request during this 60s timeframe, but very unlikely. You can schedule the renewal attempts to happen somewhat randomly and during odd hours to help minimize this.

It's not a perfect solution, but it should still work well.

2 Likes
#17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.