Question 1; Can I create a certificate that is good for a bunch of sub domains that reside on different sites/webservers? I don’t think that matters as it’s just a subject alternative name to Let’s Encrypt?
Question 2; Can I do the Let’s Encrypt challenge on ONE server, declare a list of SAN, repackage and distribute the keys to the other sites/webservers?
Question 3; Does this process I’ve outlined below have any merit, or is there an easier way you can see?
Proposed Solution: I would have one Ubuntu server using Nginx and Certbot (which seems like a nicely supported option) that handles the Let’s Encrypt challenge and declares all the subject alternative names at that time. I would then like to obtain the keys, repackage them and distribute them as a keystore to my customer sites. The customer site are using an enterprise application that I can’t change much which is Apache/Tomcat on windows.
Background: I have a lot of independent servers on customers sites and I want to get them a proper DNS name and security certificate so I can stop teaching people to put their password into unencrypted websites they can’t identify. I’m thinking of using Google DNS which allows up to 100 A records for one DNS name. If I did that the name space would look something like site1.example.comsite2.example.com etc. Each of those sites would then be a Cname or A record pointing ultimately to the customer server.
My domain is: as yet unknown
My web server is (include version): apache/tomcat on Windows AND Nginx on Ubuntu
I can login to a root shell on my machine (yes or no, or I don’t know): yes
I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no
This can work; the biggest things to worry about from my point of view are (1) that the different servers are then trusted each other with their private keys, and a compromise of any of the servers will reveal the private key for all of the servers; (2) you have to have an ongoing way to distribute new keys and certificates after a renewal.
In order to use a centralized authentication server with Let’s Encrypt, you should use either the DNS-01 challenge (if the centralized server can add DNS records automatically to the DNS zone) or the HTTP-01 challenge (if the other servers can be configured to redirect /.well-known/acme-challenge to the same location on the centralized server via an HTTP 301 redirect). The latter case has been discussed in a lot of previous threads on this forum.
For possibly better security properties in terms of reducing the sharing of private keys, you could look at the rate limits and figure out whether you can make the certificates smaller so that fewer servers have to share a single certificate.
The transport of the keys was definitely an an area that concerned me right up front but your point about one breach compromising all is new and concerning as well. Will think more on this.
As far as the challenge type I was thinking about HTTP on the linux/nginx machine, I have a purpose to throw up an informational web page outside of the enterprise apps anyway.
New question; Is it possible to make a certificate with only a couple subdomains on it (server1.example.com, server2.example.com) and have another certificate with more subdomain names (server4.example.com, server6.example.com) even though they both have the same DNS name? We could certainly split things up a bit if that were the case.
Sure, you can have arbitrary overlapping certificates for arbitrary combinations of names. The domain names covered by a certificate may or may not be subdomains of a single common domain, and multiple certificates can exist at the same time that cover the exactly the same domain name(s) or subdomains of the same domain name(s).
Let's Encrypt does have rate limits that limit the frequency with which you can issue new certificates that cover the same domain name, but this is meant to ensure a reasonable level of use of the service and does not imply a restriction on which certificates may exist or be valid at the same time.
Again thank you, it’s nice knowing that the situation is possible and with variations that will work. I didn’t even think about overlapping. Also understand rate limiting for abuse etc.
A) If you are serving multiple customers I would suggest a certificate with SANs per customer. This adheres to the policy of separation of concern and will allow you to have a web key per customer
C) The Servers you are using will be protected to some degree. Tomcat uses JKS which is a key store however Apache and Windows Means that you have to leave your certificates and keys on the disk
D) You can use -expand to add more names in the future. You can use -allow-subset to change if let’s say at renewal time you don’t need certain names
E) Certbot doesn’t support JKS out of the box but I have written an article (search JKS in this forum)
F) I would look at password protecting your web keys
G) You may want to look at building a small database to manage these. Certbot manages most of the certificates on file system artefacts
Overall you are describing an MSP / Service Provider scenario and I know plenty who have been using Let’s Encrypt Successfully
On a last note: review the range of client available. Certbot is a great option but depending on what the end goal is you may find the “no bells and whistles” clients more suitable (better to get this right first)
A) I updated the title so other searches for similar things can find it easy or so others who have experience can chime in
B) Also think about transition away from your service. I tend to advise MSPs to add a clause that the SSL certificates aren’t “owned” by them and are provided as part of the service. Termination of service does not entitle to keys and certificates etc. It seems like a small thing but better to get it out of the way early on rather than facing having to teach a customer that is leaving how to use Let’s Encrypt or worse having to “provide” them with certificates after.
ahaw021, first post:
A,D,G) I am conscious of the problems of losing control of the keys and I will do what I can. I can understand this process that you have outlined and like the enhancements it adds but quite honestly this would likely be beyond my skill level at this time. I’ll see if I can get help though. You must understand that presently the bulk of our customers are putting an IP address in the browser and signing in to a HTTP website right now, this has to be a decent improvement already.
Edit: under the get it right the first time philosophy I’ll likely be asking for help in this area, thanks for provoking the thoughts.
C) Yes that’s my understanding, keys are on disk and then item F applies.
E) Thank you, I will be looking for it soon I’d guess
F) If you are talking about a keystore password then yes the enterprise app I’m dealing with actually requires that. If you are talking about something else than that’s a problem as the enterprise app I’m dealing with on the remote sites is not something I can modify much and still receive support from the vendor. Swapping keystores is probably already putting us in a gray area, they have a documented process for exchanging csr and anything outside of that you are on your own.
Last note) I am all for easy and will certainly check and see if there is a better way. On the linux server I’m not even married to Nginx or Ubuntu, I just have some experience there. If there is an easier way just let me know!
ahaw021, second post:
A) Not a problem, thanks.
B) Defining the service offering isn’t my area but I will let them know this and several other things that have cropped up along the way. Things like we need a minimally competent person on site that can enable 443 forwarding to the server; shocker that’s not always available.
At the bottom there are a couple of automation scripts as well.
The key take aways
A JKS is not a certificate. It’s a store and as such you can have multiple certificates in one store.
You can have either SAN or Single Domain Certificates
You should import the intermediate into each JKS you are going to work with
Depending on how your tomcat is setup you can use the JKS from tomcat webconfig.server file or within Java (you need to find out which one)
I would suggest keeping the master JKS on your certbot server and then when new certs are obtained copying it to the customers servers (rather than trying to work with JKS files on each customer server and copying CSRs and certs back and forwards)
I then proceeded to add a second proxy_pass to nginx and in the process I got rid of the sites-enabled folder and rolled everything into nginx.conf for easy reading. Q1: Was that a bad idea, does certbot need to touch sites-enabled/default on a regular basis? Just a one time thing hopefully?
Right now I have the reverse proxy upgrading http to https, requesting basic_auth and signing with valid Let’s Encrypt certificate for test.example.com. Test2.example.com also works but obviously the certificate isn’t valid for that.
Q2: Where do I tell certbot to request multiple sites, including ones that will never been in this enginx config?
Off to start reading certbot docs at this point I guess.
You can do this with a -d option on the Certbot command line (I suggest one -d for each domain name).
Two things to note are:
Certbot obtains a single certificate at a time, which may be installed as a replacement for (internally we've said "a successor to") a previous certificate if you want. The certificate has a name that Certbot uses to refer to it, which by default is the name of the first domain name that was originally used for that certificate. Certbot will try to guess an appropriate certificate from the domains that you specify, or you can indicate which certificate to replace (or what to call a freshly-created one) with the --cert-name option.
For all Certbot commands that obtain a certificate, you should list every domain that you want to be covered by the new certificate you are obtaining. You never just list "domains to add" to an existing certificate. It's always a complete list. So if you want to add new names, you list the old names as well as the new ones. If you want to remove names, you get a new certificate (specifying the certificate name of the old one) with a shorter list of domains.
OK, understood. Followup question; when I last ran certbot to add new subdomains it appears to be issuing a challenge to each subdomain. This currently works as they are all on one server and I have the config allow that. What happens when future subdomains are on a whole different server on a remote site with a webserver I can’t control that isn’t going to answer that challenge?
Edit: I guess you answered that in second post of this thread. I must have skimmed that.
Edit2: just did some research on the DNS text challenge and I’m not sure I can do that with google DNS, or maybe just over my head. Not sure I can edit the web server on the enterprise app to do a 301 either. This portion of the project may just get parked until I can get wildcards next year.
In that future instance, are connections to those remote webservers going to be proxied through the server running certbot, or is DNS going to point directly to them? If the latter, even if you got DNS challenges working, how are you going to get the certificate to these remote servers?
If you’re proxying connections through this single host, you can set up a location block for .well-known/acme-challenge to not be proxied and serve the challenges from there. If not proxying, you’ll need some sort of ability to place the certificate files on the remote servers and reload their webserver configs to make them start using the certificate, so you should also be able to drop challenge files there.
Just tossing out ideas, you could also have these remote servers proxy the challenge directory back to the server running certbot that you control. The point is there are options, it just depends on your setup.