I asked how to handle installing LE on my work's setup of ~150 subdomains, split 50% AWS ELB and 50% HAProxy and based on the feedback from the thread, we decided to handle SSL termination on our EC2 instances, instead of our ELB/HAProxy.
I developed a plan to split our sites into different certs based on their root domain's first letter. So, cert 1: abcd, cert 2: efgh, cert 3: ijkl, etc. (see appendix 1).
It's my understanding that when you run certbot renew it renews all certs, and not just specific ones.
Reading the Rate Limit docs
To make sure you can always renew your certificates when you need to, we have a Renewal Exemption to the Certificates per Registered Domain limit. Even if you’ve hit the limit for the week, you can still issue new certificates that count as renewals. An issuance request counts as a renewal if it contains the exact same set of hostnames as a previously issued certificate. This is the same definition used for the Duplicate Certificate limit described above. Renewals are still subject to the Duplicate Certificate limit. Also note: the order of renewals and new issuances matters. To get the maximum possible number of certificates, you must perform all new issuances before renewals during a given time window.
Note that the Renewal Exemption also means you can gradually increase the number of certificates available to your subdomains. You can issue 20 certificates in week 1, 20 more certificates in week 2, and so on, while not interfering with renewals of existing certificates.
Because I have 120 sites over 73 domains, I'd need to renew certificates the same way I requested them, correct? certbot-auto renew -n -d site1.a.com -d site2.b.com -d site3.c.com certbot-auto renew -n -d site4.d.com -d site5.e.com -d site6.f.com certbot-auto renew -n -d site7.g.com -d site8.h.com -d site9.i.com
I must have missed the original thread, but I’d be curious to read it; want to provide a link?
The short answer to your question is that certbot-auto renew by itself with no flags will work just fine for you. It will automatically request certificates with the same groupings of domain names.
You probably don’t need to take any special precautions about renewals. Your certificates are all for different domains, so they are all rate limited separately and certbot-auto renew will only renew them every 60 days anyway.
However, I think you may have misunderstood how the rate limit is applied.
You can issue as many certificates as you want for different domains. The rate limit restricts how many certificates you can have for subdomains of the same domain. So you don’t really gain anything (from a rate limiting perspective) by combining subdomains of different domains into a single certificate. (In fact, if you frequently add new subdomains to the certificate, you could even end up hitting the limits sooner!)
The link has been posted below by @jmorahan. Thanks for clarifying. LetsEncrypt has made a once impossible task really easy and I'm so so grateful. That's exactly what I wanted to be able to do, and LE just "know" so I am happy that is the case and renewing isn't going to be a nightmare.
@jmorahan, yeah I think I have misunderstood the limit completely by the sound of your reply. So, I don't need to split my sites over 7 certificates? Our set up is a bit unusual, we host a subdomain (sometimes 1, sometimes more which is pretty much a landing page), on a domain that isn't owned by us. As a result, the subdomain DNS records are split between an older HAProxy, and more recently added clients use the ELB, both point to the same 4 servers, though.
So, if I understand you correctly, I can issue a certificate for each of my domains? Is that correct?
This is the from the docs that made me split up my certs:
If you have a lot of subdomains, you may want to combine them into a single certificate, up to a limit of 100 Names per Certificate. Combined with the above limit, that means you can issue certificates containing up to 2,000 unique subdomains per week. A certificate with multiple names is often called a SAN certificate, or sometimes a UCC certificate.
It's rare we add sites - although not impossible, a new site every few months I imagine sounds feasible.
@mnordhoff - thanks for that! Someone replied to my other thread about it and it in theory is perfect - however, asking our clients to update their DNS would be a disaster I'd like to avoid. However, if we ever need to, we'll go down the ALB route 100%.
@jmorahan - wow, that's completely changed how we could do set it up. I appreciate the clarification!
So, to confirm, I wouldn't fall foul to the other rate limits if I were to request a new certificate for each unique domain + each unique subdomain name?
I'm not aware of any other rate limits you're likely to run into with that setup.
Just one thing to maybe watch out for (and this applies to your current setup too):
Do your 4 servers each serve different sites (e.g. sites A, B and C are on server 1, sites D, E and F on server 2 etc), or do they share the load of serving some of the same sites between them (eg, site A is on servers 1, 2, and 3, site B is on servers 2, 3 and 4, etc)? In the latter case, are you running certbot-auto renew on each server? If so, you might run into the "duplicate certificates" limit when they all try to renew the same certificate at once - with 4 servers you would just get away with it, but with 5 you would hit the limit and 6 would exceed it. However, if each site is allocated to a single server - or if you have a dedicated machine that runs certbot and copies the certificates to wherever they're needed - this won't be a problem.
@jmorahan, you’ve made the task so much easier - I really appreciate your input!
Yeah, all 4 servers serve the exact same sites. No, how I originally envisioned it is we’d use the AWS CLI to add/remove our other web servers instances out of the load balancer, bit awkward but I guess it’d make sure the 1 server requesting is the server getting the cert. However, I now realise that we’d also have to edit our HAProxy, and add/remove instances to that, too. I’d then SCP the certs/vhosts from the server running certbot to the other 3.
Yeah, I’ve been reading about the “dedicated machine” route that runs certbot but haven’t found too much info on it. Are you able to link to anything that explains how that’d work?
That’s the only part now that seems pretty hacky, I’d love to find a way around it.
If you’re running certbot on just one of your instances, you don’t have to remove the others - you just need a way to ensure the domain validation challenges go to the right server. For example, if you are using --webroot you can configure HAProxy / ELB to route /.well-known/acme-challenge/* URLs to that server. HAProxy can also be configured to forward the requests to the correct server for the TLS-SNI-01 challenge (used by certbot --apache), but I think the ELB can’t, so you’ll probably want to use HTTP-01 (eg --webroot) or the DNS-01 challenge (eg --dns-route53).
As for copying them to the other servers via SCP, see the documentation for the --deploy-hook option here and don’t forget to reload the webserver configuration too.
I had originally looked into the HTTP-01 challenge, but I had read that with an ELB it's impossible to route traffic based on path, and that an ALB should be used instead. And as our sites are 50% HAProxy, 50% ELB, we'd have to use TLS-SNI-01 for the ELB anyway. Unless I'm not understanding the ELB docs correctly?
So, in regards to requesting the certs, how would I best go about requesting them? Is it best to do one-by-one? I.e.
That's correct, sorry I didn't realize you were using classic ELB (since ALB is also a type of ELB).
So if you're using classic ELB you can't route either by the SNI information or by the request path, AFAIK. Other options I can think of:
Use the HTTP-01 challenge and have the other instances themselves proxy /.well-known/acme-challenge/* requests to the correct instance, or have that instance copy the challenge files to the others.
Temporarily remove instances from the ELB other than the one answering the challenge, and add them back after (as you originally planned) - in this case you could use TLS-SNI-01
Use the DNS-01 challenge, if your DNS provider is supported
Yes, that's how I would do it (with the caveat that ./certbot-auto --apache will use TLS-SNI-01, so see above regarding whether that's appropriate or not).
Remember that while ./certbot-auto --apache will obtain and install the certificate on the apache instance where it runs, it's up to you to manage how it gets propagated to the other instances. Again --deploy-hook may be helpful here, especially since any steps performed this way will be automatically repeated upon renewal.
I appreciate your input the other week! I’m finally in a great position in regards to getting LE installed across our sites. I’ve used Laravel Envoy to run these tasks:
So far so good, we’re going to be testing my script on production next week.
All finally makes sense, but now I’m starting to think about the renewal process. I understand running the renew command will update all certificates that are due to expire, and that I will need to copy the renewed certificates across our servers. However, where I’m getting stuck is whether or not certbot runs any challenges similar to when you request a certificate?
The reason I ask is because of our ELB - do I need to take the other 3 servers out of rotation so the 1st server can handle the renewal? I’ve read yes, that I would need to, but I haven’t found anything concrete.
Yes, the challenges will have to be completed again at renewal time.
You might be able to use certbot’s --pre-hook and --post-hook to manage the ELB if you don’t want to have to remove the 3 servers every time you run certbot renew from cron. The hooks only run when there’s actually something to renew.
(Or you could use one of the other methods I described in my previous comment, and not need to remove anything from the ELB )
I agree - there are many other better ways that this could be done. We’re unfortunately stuck with this method because our sites were setup many years ago, and we don’t host their DNS, so getting our clients to update their DNS to point to a new LB or add records to their DNS would probably need to happen a year or so in advance. We’re stuck in a difficult/awkward spot. I wish we could just move to an ALB and be done with it!
Actually, in regards to the HTTP-01 challenge, as I’m on an older ELB, how could I proxy/tunnel response for the /.well-known/acme-challenge/* requests? I’m not too familiar with a multiple web server setup, so I’m not 100% how something like that would be possible. I just read your comment about copying the challenge files to the others servers. How would I hook in to that? Is there a post-auth-sent / pre-auth-received (or whatever that process is called - basically after the request has been sent so the HTTP01 files exist, but before it’s “sent”, so I can send those files to the other servers) event
Well, there are (at least) two ways you could do it.
The proxy method: Use something like mod_proxy and mod_proxy_http on the servers that aren’t running certbot, to proxy those requests to the one that is. The configuration (once you’ve enabled those modules) would look something like this:
So any request matching /.well-known/acme-challenge will either go to the server running certbot, or it will go to another server which will then proxy the request to the server with certbot. Either way it should get the correct response and the challenge should succeed.
The file copying method: Yes, there are hooks: --manual-auth-hook and --manual-cleanup-hook. They are documented here. The auth hook script will be given the filename and content as environment variables, from which it can create the file and copy it to the other servers. The cleanup script can delete them afterwards. (Note that these hooks are used with the --manual plugin, rather than --webroot, though no manual intervention is actually needed when using the hooks).
I just wanted to say a huge thank you! Your advice was perfect and made such a daunting task actually pretty straight forward. We started the migration of http -> https for a handful of sites last night, and other than a few file permissions across our servers, it worked like a charm.
I have one question which I haven’t found an answer to, so I’m wondering if you have any experience with it.
When we request a certificate for multiple domains, it uses the first domain for the folder in /ect/letsencrypt/live/FIRSTDOMAIN/, I understand that /live is symlinked to the actual files in the archive directory. My question is, am I ok to rename the directories under /live?
For instance, it will use /ect/letsencrypt/live/subdomain1.site.tld, which is fine for subdomain1.site.tld, but as subdomain2.site.tld uses that same certificate, in subddomain2’s vhost .conf file, we have to link to the certificates which are in /ect/letsencrypt/live/subdomain1.site.tld.
I just want to rename that folder so it’s /ect/letsencrypt/live/site.tld. Can you forsee any issues with that? I’m unsure how the renewal process works in regards to that directory.
It is not a good idea to rename dirs inside /etc/letsencrypt/ structure. Well, you can, but you should change the name in /etc/letsencrypt/archive/, /etc/letsencrypt/live/, recreate the symbolic links in /etc/letsencrypt/live/newname/ to point the right files in ../../archive/newname/file, rename the conf file in /etc/letsencrypt/renewal/ and edit the renamed conf file to change the paths for several variables to point to the renamed dirs.
Instead of that, you can use --cert-name directive to specify the name you want: certbot [your_options_and_domains] --cert-name site.tld and certbot will create the structure for your new certificate so it won't use the first domain you are issuing a certificate for but the name specified in --cert-name parameter.
)