I run a small web hosting and design business and I’ve been working to integrate LE into our production workflow for new and existing customers. One of the goals being 100% required HTTPS. Everything came together surprisingly quickly using certbot and our existing Apache-based systems (with most sites running Wordpress).
In fact, I thought I was done, but during migration testing, I learned my understanding of the rate-limiting restrictions was flawed. Here is the workflow as it stands:
All of this works perfectly. The key value is always the site ID. Work can start immediately on the site, using proper HTTPS, and not effect the existing site or be dependent on new domain registration delays. It also easily facilitates additional domain aliases and changes of domain names.
But it seems to run into a few snags with LE:
After creating my 10th site, certificates began getting rejected due to the used of (1) [siteid].site.MyHosting.com initially and (2) expansion to include CustomerSite1.com with each site.
It also seems – and I’m not sure if this is still the case – that renewals use up your 20 allowed per 7 days. Since all of our certs will have random expiration dates, and each includes [siteid].site.MyHosting.com in addition to any customer domains, this would likely leave us with no ability to setup new sites on many days of the quarter.
This is a pretty simple workflow that isn’t all that uncommon (temporary urls). I’m not excited about the idea of playing games to dance around these restrictions, which will inevitably add complexity. I’d like to know:
Is my understanding correct.
Will/does LE supporting this kind of workflow?
Also, thanks to all involved on this project. It’s great to see happen.
One possible solution would be to buy a wildcard certificate from another CA for *.site.myhosting.com, and use Let’s Encrypt only for your customers’ actual domain names.
Another possible solution may be to fill out the form linked from here to request an exception from the rate limits. Not all requests are granted, but as you’re a hosting provider it may be worth a try.
Based on that link, I don’t think we have the kind of volume needed to be approved for an override, but I don’t think that’s really a solution here, as it takes very little volume to run into this issue using this very simple workflow.
As for the wildcard, yes, I realize I could go to another CA and get a wildcard, but at that point, I might as well just use another CA and not LE. It would also add complexity to our systems that wouldn’t otherwise be necessary.
It seems like LE is trying to provide a solution for exactly this kind of product and makes a point to say wildcards should not be needed, so I’m trying to understand how LE supports this. Every reference I can find on the forums about this kind of thing seems to suggest wildcards or public suffix (which the fine people over at publicsuffix.org don’t seem to appreciate).
I mostly wanted to make sure I wasn’t just missing something as this seemed like it was in line with what LE was made for. Thanks again for response.
Well, there are other ways you could get around the rate limits, but you said you didn’t want to add complexity
The simplest I can think of is to separate the approaches you take for the two kinds of domain. For the real customer domains (CustomerSite1.com etc), just request a certificate per domain. For the 54321.site.MyHosting.com domains, I’m guessing your site IDs are sequential integers? If so, you can issue a single certificate in advance to cover up to 100 of them, and that only counts as 1 against the rate limit. When you get close to the last ID, pre-issue another 100 on a second certificate. When you have so many customers that you have to renew 20 of those certificates per week, you’ll start running into the rate limit due to renewals. However, you can still circumvent this by timing things carefully - renewals count towards the rate limit but they’re not themselves subject to it, so you just need to make sure that you request the new certificate for the next batch of IDs before you run the renewals.
Your description of the rate limits is accurate. The fact that renewals use up part of your quota is definitely not intuitive and requires careful planning of renewal schedules (which can be rather impractical with many use-cases). This is something that might change at some point in the future, based on previous discussions, but it’s a tricky migration and probably not going to happen anytime soon.
The least invasive option in your particular scenario would probably be to add two vhosts for each site - one for the siteid subdomain, and one for any custom domains provided by the client. You could utilize a wildcard certificate from a different CA for the first vhost, or pre-generate a couple of multi-SAN certificates with 100 subdomains each to avoid running into rate limiting issues. The second vhost could then have a certificate that’s only valid for the custom domain, meaning it has a separate rate limiting scope.
@jmorahan, I appreciate this answer because it’s one that can be implemented using LE. It’s certainly not preferred to play games like this (like keeping track of which site ID are on which cert), but it’s functional. They’re not actually sequential site IDs, but they could certainly be made so or at least pre-determined.
It’s frustrating to repeatedly read suggestions that include a wildcard cert. How many times can you suggest wildcards until you have to question your FAQ.
I don’t actually prefer to use a wildcard, since I also think it shouldn’t be needed (for the same reasons it’s not offered), but LE does push you toward them, while simultaneously discouraging them. It doesn’t seem to be getting any traction with ACME either.
I don’t think “planning renewal schedules” is really something that can apply to most deployments that include undetermined customer fluctuations. Although I can work it out here, it feels like a crutch. I feel like that’s also on the list of things that are often suggested that make me twitch, along with wildcard certs, you can register “x” thousands per week (in threads about hostnames that can’t be predetermined), and to use a public suffix (when it’s usually not appropriate).
I look forward to LE coming up with solid solutions to these real-world production deployments.
I really appreciate the work LE is doing and I look forward to watching it evolve and participating where I can. Thanks again.
Exploring the idea of a “site id inventory” for pre-generated LE certs mentioned by both @pfg and @jmorahan…
It’s easy enough to pre-generate the 100 site id and include them in an LE certificate request, but the optimal method of authenticating all of these URLs during the certificate request seems to get a little messy.
I’m using http-01, so in this scenario, instead of provisioning the cert as-needed, I would need these 100 “inventory” site id to respond to a web request to each of the corresponding URLs. It seems they each would need their own webroot (a certbot -w with each -d) so that when they’re assigned to a customer, this webroot (which would likely be a symlink), would still work for this domain during renewal even though it will now be pointed to a customers directory (ie webroot_map in the renewal conf). This would end up touching provisioning, reporting, and billing in one way or anything to make work.
Maybe dns-01 is a better route for the “inventory” since this is an internally controlled domain? That would mean a whole new integration with the DNS provider, but possibly easier than these “fake” sites.
Opposed to the “as-needed” LE integration that took me an hour and works great until I start hitting the rate limits, I’m deep down the rabbit hole of significant modifications. This would be the complexity I had hoped to avoid.
You could point a DNS wildcard at a server that will respond to all *.site.MyHosting.com requests from a single shared webroot, and use that for validation. You would also have to proxy or redirect /.well-known/acme-challenge/ from the actual sites to this server for renewals. Depending on how complex that is to do in your setup, though, dns-01 might indeed be easier, especially if one of the existing clients or Lexicon already provides integration.
Actually, you could use a single vhost with many server name/server alias directives. That would also help keep track of them as when you use each, you would need to remove them from the vhost used to authenticate them and add them to the customer’s site vhost. It’s a little more work to create each site, but it’s doable.
The problem comes in when you have to renew them. If you’ve deleted any vhosts using the sequential or otherwise generated subdomains, you’ll have to manually reissue the certificate without that subdomain. Or just put it back in the pool of unused subdomains so you still have a vhost for it when it renews.
