I work for a large company with a mature PKI program and we are evaluating new solutions to issue and manage SSL certificates inside our company. The services and software that the LetsEncrypt community have built are nothing short of amazing!
We are considering running our own Boulder instances to issue and manage tens of thousands of privately trusted SSL certificates.
I have seen many users talk running their own Boulder instances in these forums. Do you run Boulder for your organization? If so, how’s it working for you?
I’m always interested to hear about people outside of Let’s Encrypt using Boulder. It’s great! Please open issues as you bump into things that might be broken/confusing
We’ve been approached in the past by folks wanting to use Boulder for their own internal PKI and my advice at the time has been to skip Boulder and use cfssl & its HTTP API directly. Did you consider that option as well? I would be interested in your thoughts.
Boulder uses cfssl internally, and much of the code (& complexity!) that we build on top is focused on the needs of a CA for the web PKI with subscribers and relying parties outside of our organization. It’s certainly possible to make Boulder work in a private PKI but when you have more control of the endpoints, domains, and relying parties I suspect it’s easier to skip Boulder/ACME entirely and use cfssl.
Thanks for the high praise! I’m curious: What method does your internal PKI use to validate hostnames? As I’m sure you’re aware, Boulder uses a Domain Validation process specified in ACME in order to verify control of a hostname before issuing a certificate. Like all Domain Validation methods, it’s vulnerable to DNS spoofing and MITM attacks. This is a necessary tradeoff on the public Internet, but depending on your intranet setup you may have a more reliable way of proving domain control.
While my team manages hundreds of thousands of private certificates for other purposes, our current SSL certificate service is not great. It was recently inherited from another group and is basically:
Subscribers read some documentation and follow steps to generate a CSR
Subscribers then fill out an online request form and paste that CSR into it
My team reviews the request manually. Our current “Domain Validation” process is to check that the subscriber’s billing code matches the related billing code in the company’s DNS system.
My team then submits the request to the appropriate CA and issues a certificate.
We email the certificate back to to the subscriber along with documentation on how to install and configure it.
The subscribers ignore the documentation, frequently configure TLS incorrectly, and often come back to us for support.
We see two classes of SSL certificate subscribers in our company:
Those that run on permanent servers (conventional platforms) and need a tool to manage certs
Those that live on ephemeral servers (cloud platforms) and would prefer to call a straight API to get a cert
The first group of subscribers is currently our most common. The sheer number of manual steps and the amount of support costs involved are what make automated solutions like Boulder so attractive. Subscribers would use ACME-compatible clients and point them to our own Boulder instances. As I understand it, we need all that Boulder goodness on top of cfssl to make the validation, enrollment, configuration and renewal automation work. Is that right?
The http-01 and tls-sni-01 Domain Validation processes you implemented in Boulder are light years better than our current process. I don’t see any tools in our company that could help us do validation better. We are one company, but we are composed of hundreds of different groups of systems and people.
Today we make the second group of subscribers follow the same manual process as everyone else. It is for this group that I see a solution like cfssl fitting. We can put its API behind our API Gateway and let the latter handle authentication and authorization.
Thanks for the detailed response. I agree that automated DV through ACME (and potentially Boulder) would be a big improvement for your company! I would say: Go ahead and try it out. A few caveats:
We tend to develop Boulder primarily to meet our own needs, and are able to directly coordinate with our own Ops team on things like database migrations and config changes. If you adopt Boulder internally, you may find that we don’t do sufficient external communication on how to do those things. But we’re definitely willing to accept pull requests and process suggestions, since improvements here will also probably improve coordination with our own Ops team.
We’re likely to be very conservative about accepting feature requests and pull requests for things that we won’t use in Let’s Encrypt’s production instance. We want to keep Boulder as simple as it can be while still doing its job.
I plan to run a full stage environment of the service in our company. We will test any changes and update our prod environment only as our own business needs and policies dictate. Hopefully that will mitigate the risks of any changes made by the Boulder team or my team.
I use the LetsEncrypt service personally and I love it. I am, however, a little nervous that we might be the first to use Boulder outside LetsEncrypt. If anyone else is using Boulder, I would love to hear from you.
It’s entirely possible that you would be the first production users outside Let’s Encrypt. I haven’t heard about other production users.
I do know one CDN that runs a Boulder instance internally so they can do integration tests against it. I think it has been fairly stable for them. They update about once every 3-6 months, and they have generally required a little input on configuration at update time.
