Rate Limit for extending SAN

goli · March 22, 2018, 11:51pm

Hey there!

Question: Am I going to hit rate limits?

Here’s the steps I’m going to perform

I’m only dealing with a single domain.
Assuming I create 10 different SAN certificates (for 10 different subdomains).
Then I adjust the list of covered hosts up to 10 times per day.
Adjusting means: I fill the SAN up until it holds 100 hosts. As soon as the list holds 100 domains I start dropping one at a time before adding a new one to reduce the number of requests to LE.
The domains I add or remove are always going to be subdomains of the subdomains of step 2, so I’m pretty sure my 10 certificates in total have not a single overlap.

Do this a SAN adjustments count as renewals or new certificates?
Which rate limit is to be expected?

Expectation

If I get the your rate limits policy right, I’d expect to have only 10 creations on the very first minute, which doesn’t hit a rate limit. After that it’s all renewals, so no rate limit here as well. Even if I do that 10 times a day per all of those 10 certificates (doing 100 requests to LE, 10 for creating and 90 for increasing the domains list), that’s all within the allowed limits.

Please correct me if I’m wrong.

Here’s the details about my project:

We’re a company of web developers.
Each developer has his own environment *.${employee}.mycompany.wtf.
When a developer starts working on a project, he creates a virtual machine ${project}.${employee}.mycompany.wtf.
- Of course there are a couple of virtual machines per project, but they don’t need certificates, hence are not important for this question.
In regular sized projects, there are mltiple CNAMEs for different all pointing to a single web server.
- Those CNAMEs sometimes are devisions of our project, simulated CDN hosts and so on.

This sums up to being *.${project}.${employee}.mycompany.wtf.

A regular example would be:

project.goli.mycompany.wtf
cdn.project.goli.mycompany.wtf
styleguide.project.goli.mycompany.wtf
europe-division.project.goli.mycompany.wtf
american-division.project.goli.mycompany.wtf
there’s projects with 30 divisions

We intend to create a single SAN certificate per employee.
Per default, this certificate is to be issued to ${employee}.mycompany.wtf (primary domain) and holds *.${employee}.mycompany.wtf to cover all small projects.
We keep track of a list of our regular sized projects. That can be automated since every virtual machine connects to a central puppet environment. I easily can tell what’s the last 50 projects we had up and running in our company just by grepping the puppet logs.
This list gets used to add the last 98 projects as *.${project}.${employee}.mycompany.wtf to the SAN.

We’re going to set up a central certification host that keeps track of all the recent project names, knows our DNS API keys and provides our projects with valid certificates once a day.

So. Am I going to do something completely wrong? Is this going to work or will this hit every possible rate limit?

Regards,
Stephan.

_az · March 22, 2018, 11:55pm

You should almost certainly issue separate certificates and use SNI to respond to with the right one.

Consider also, the use of wildcards or a private PKI.

schoen · March 22, 2018, 11:58pm

The private PKI might also be useful to avoid revealing to the public what all of the internal project names are and who is working on each one, as would otherwise happen via Certificate Transparency.

rg305 · March 23, 2018, 12:30am

Will you be requiring the use of current certificates only?
If so, does that mean that you will need to revoke all of yesterdays' certificates or provide cert life of 1 or 2 days only?

goli · March 23, 2018, 12:54am

Wow, that was fast. Thank you!

Of course that's possible as well. But having SAN mimics our production environment while different certs with SNI does not. That's not related to LE at all, but trust me, there's reasons for that.

But as for the actual LE idea: That would definitively run into rate limits, wouldn't it?

Well, maybe my explanation was confusing. I was going to try wildcards, like *.project.employee.companyname.wtf.
But there are days I need to work on 10 different projects, and the day after I need to work on 10 completely different projects.
If I had only a single SAN cert per employee that contains 90 different wildcards (one per project), there's a huge chance I didn't need to reissue or renew it a couple of days in a row.

Of course PKI is a thing. But due to the way my team and my projects work, PKI isn't that easy to deploy.

That's a very good point! Hadn't considered that until now. Of course you're right, and that's something I need to check with our legal department. Most of the projects I'm working on are cleared in a way so that I can reveal my involvement.

Sorry for not being clear about that. I was just trying to express "the certification host runs a cronjob for renewals once a day".

I don't care about the certificates from yesterday. When my employees start working in the morning, they fire up their development environment. That's the very moment the virtual machines can update certificates. That's the moment they connect to puppet and check for updates anyway.

If that puppet run returns the certificate file I it returned yesterday, that's fine with me. If that run returns a new one because the old one expired after 90 days, that's fine with me, too. And if that puppet run returns new files because someone added a new project name to the SAN domain list meanwhile, that's fine with me as well. I even don't care if the SAN list contains projects I haven't used for months. Doesn't do any damage.

The only thing is:

The central certification host knows all employees.
The central certification host knows the last 100 projects used amongst all employees.
The central certification host does auto renewal if necessary.
If a new project appears in the list of the last 100, the central certification host starts adjusting the certificates immediately.

What I want to achieve is that every development environment can "SCP" an individual certificate file per employee at any given time and be sure it gets one that contains a proper set of wildcard domains.

... maybe I should have opened this post not at "Issuance Policy" but at "Help" because it got way out of "does this trigger rate limits" quickly

Regards,
Stephan.

_az · March 23, 2018, 1:00am

Yes, you would be limited to 20 certificates per week under your Registered Domain (mycompany.wtf), whether you used SANs or individual certificates. You are right that using SAN wildcards would probably mean less overall issuances in your case.

You could avoid this by applying for a rate limit exclusion from Let's Encrypt, but I'm not sure how your request would be viewed (seems needlessly burdensome on the CA).

Maybe if you can limit your certificates re-issuances on an "at most once every X days" schedule, your SAN idea might work, and stay within the general rate limits.

rg305 · March 23, 2018, 1:00am

Isn't that just:
*.employee.company.com
one per employee?

goli · March 23, 2018, 1:10am

Nope. it’s *.project.employee.company.com, one per combination of employee and project name. If only certs could be issued to *.*.company.com.

Real world example:

Users Stephan and Marcus
Projects Coke, Amazon
Multiple host names per project

Results in:

And now there’s not 2 developers but 10, there’s not 2 projects but 50, and there’s not only USA and EMEA but up to 30 divisions per project.
That’s up to 15’000 domain names in the range of *.*.comany.com, distributed over 500 virtual machines on 10 laptops.

Of course they don’t all run at the same time but one after another. And this is the very reason I can’t simply put LE on all of them: I’d run into rate limits after three or four days.

Regards,
Stephan.

rg305 · March 23, 2018, 1:15am

OR

??? Seems reversing project with employee ???
Have you tried shrinking the subdomain footprint?:
*.project-employee.company.com OR *.employee-project.company.com
combining them might help...

goli · March 23, 2018, 1:28am

^^ Noticed my mistake and edited my last post while you answered.

Combining host names isn’t going to work.

Our development environment is located on laptops, every developer has his own private IP range and DNS zone, all managed by a small OpenWRT/LEDE virtual machine on the developer computer, connected via VPN to the company network. There’s a central name server at our company that has NS records pointing to the OpenWRT machines on all of our laptops.

This allows my developers to be at completely different places but still access another developers current project state via HTTP. Think about one developer visiting a customer at his office, calling another developer sittng at starbucks to ask him to quickly try some changes and show remotely. With distribute IP ranges, proper routing and proper NS delegation that just works.

Regards,
Stephan.

rg305 · March 23, 2018, 1:34am

goli · March 23, 2018, 7:41am

Unortunately that’s not possible as well.

Amazon.stephan.company.com is a single virtual machine with host name amazon.
The corresponding OpenWRT instance on my laptop has a local prefix of stephan.company.com configured. Whenever the virtual machine boots up, it gets its IP address from the OpenWRT via DHCP dynamically. There is no static DHCP lease, nor is the host known to the DNS server other than by its DHCP lease.

Whenever a DHCP lease changes on the OpenWRT (create a new, expire an old one), a small script takes all host names from the file of current leases and ransforms each one into a wildcard CNAME record.

Example:

amazon.stephan.company.com boots
amazon.stephan.company.com gets an IP address via DHCP
dnsmasq triggers the “onupdate” script
*.amazon.stephan.compan.com is created as CNAME pointing at amazon.stephan.company.com
amazon.stephan.company.com shuts down
dnsmasq triggers the “onpdate” script
*.amazon.stephan.company.com CNAME is removed

If I changed *.employee.company.com to *-employee.company.com, that immediately requires me to statically add 1500 “$division-$company” values to the dnsmasq.

Trust my, I’m very sure adjusting my machine names naming schema isn’t going to work.

Regards,
Stephan.

bytecamp · March 23, 2018, 11:49am

I did not follow the whole thread, just a hint: this step looks like it can be generally be done once and does not need to be automated/executed on dhcp events.

Why do you want to configure the cname from *.foo.bar.example.com to foo.bar.example.com dynamically?

goli · March 23, 2018, 12:18pm

See exactly that post:

If someone wants to know about that in detail, please feel invited to the southern part of Germany. Step by for a coffee or two, I'll give you a tour through how the development setup is constructed.

But I won't go into further detail here publicly. It only leads even further away from my initial question and using placeholder project names and company names clearly creates more confusion as it helps.

Don't get me wrong, I'd gladly explain everything to you. But obviously questioning every single step of my development environment is only bringing even more questions up to the table.

Regards,
Stephan.

goli · March 27, 2018, 10:05pm

Sorry, I completely overlooked that response. Thank you very much for the clarification. Although that's not the answer I'd hoped for, that's the one I expected.

Without having read that specific response, I meanwhile gave it a try myself.
It's exactly like that: The 21st issue request, which is the 20th extension of the SAN, results in exceeding the request limit.

So I'l adjust the subject to "solved".

To anyone who's interested in my specific setup and might want to know some more details, just PM me. I'l gladly provide some more background on that, clarifying why things are the way they are.

Regards,
Stephan.

system · April 26, 2018, 10:05pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Rate limit for subdomain question Help	4	1262	March 2, 2017
Certificates/Domain rate limit is not clear	1	1761	November 5, 2015
SAN Certs and Ratelimiting Issuance Tech	8	3635	April 21, 2016
Let's Encrypt Rate Limits for SAN and Certificates Help	4	2050	June 30, 2017
Rate Limits when adding domains?	9	2870	December 23, 2015

Rate Limit for extending SAN

Here’s the steps I’m going to perform

Expectation

Here’s the details about my project:

Related topics