I think we should consider making Caddy the default ACME client recommendation and if you disagree, I'd love to hear why.
No single ACME client is going to work for everyone as different users have different needs and priorities. While an open client ecosystem with many options is great as it allows for things to be built to fill the different niches, I also think having a default recommendation or two that's suggested as a starting point for new ACME users has a lot of value to help guide people through their many dozens of options.
For instance, if you look at ACME Client Implementations - Let's Encrypt, Certbot is listed as the recommended client and the many other client options are listed afterwards. Similarly, at https://certbot.eff.org/ we suggest people check with their hosting provider to see if they offer automated HTTPS and if not to we suggest they try Certbot. Despite being Certbot's lead developer, I've been considering suggesting EFF and/or ISRG try recommending Caddy as the default starting point for those looking for an ACME client. Essentially my thinking is that if people are open to using a different web server entirely, we suggest they completely switch to Caddy and if not we suggest they use it as a TLS terminating reverse proxy in front of their existing web server.
Why do I think this is better? A couple reasons:
It allows for better integration between the TLS server and the ACME management of those certificates. This is something that I think many of us have thought was important for a long time. Apache and nginx are the most popular web servers so to try to integrate with it in Certbot we attempt to reparse their server configuration with our own parsers, make modifications to it, interact it with it using shell commands, and sleep for arbitrary periods of time until the server has probably/hopefully finished doing things like reloading its configuration. While this fills a need while people are/were stuck on Apache and nginx, I think it is inherently more error prone and brittle than having the TLS server manage its certificates itself.
Caddy is written in a memory safe language. Adoption of memory safe programming languages is something that both ISRG and the broader computer security community have been encouraging more and more lately. Recommending Caddy would help this effort by encouraging people away from web server options written in C/C++.
What do people here think? What hurdles do you foresee people hitting if they try and use Caddy in this way? Is there anything we can do to at least partially eliminate these barriers?
Thank you all in advance for helping me think about this.
Overall, I say: NOT
Because it's much more than just an ACME client - not a one-size-fits-all solution IMHO.
I see taking this approach would go the way of the "NPM solution".
[which causes more problems than it resolves]
That said, I do think that anyone capable of and already willing to use a reverse proxy could benefit from using Caddy.
Or a web server at all--Caddy's a general-purpose web server, not merely a reverse proxy.
But much as I like the idea of moving away from certbot as the "official" client, and much as I like Caddy, I wouldn't agree with this change, for a number of reasons:
Making Caddy play well with an existing web server would require some pretty significant changes to system configuration--at a minimum, configuring (e.g.) Apache to listen on a different port, and then Caddy to reverse proxy to it; or just do the forklift upgrade and replace the existing webserver with Caddy.
There are many cases of users wanting to obtain certs for purposes other than a web server (mail servers come to mind); Caddy just doesn't work in this environment.
It doesn't make much sense to me to have the "official Let's Encrypt client" be sponsored by a competitor to Let's Encrypt, particularly when that competitor has a history of some pretty shady behavior.
Stack Holdings and the old apilayer (before sale) have the same owners. For those with access to the lounge, here is more information.
I personally can understand the caddy sponsorship somewhat. Need to get food on the table somehow. The developer of caddy @mholt is also active on these forums.
And I'm not sure if Caddy is that simple to configure/operate.
Let's Encrypt might want to consider that their main support base (i.e.: the volunteers of this Community) might not have that much experience with Caddy and might not want to invest the time and effort solely to provide support for the possible influx of Caddy questions.
Although I guess that's maybe also the reason for this thread
I personally probably won't be investing time & effort in Caddy.
Caddy itself is far simpler to configure and operate than Apache or Nginx, either as a webserver or as a reverse proxy*, though as you say it's less well-known than these. But you touch on another valid point, that being if Caddy's the "official Let's Encrypt client," people would reasonably expect to find support for it here--and this is no more appropriate a support forum for Caddy than it is for Apache or Nginx.
*Reverse proxy in particular is trivial:
host.domain.tld {
reverse_proxy localhost:8686
}
Repeat these blocks as necessary for any other hostname.
Coincidentally, I'm just finishing a lengthy article about this very topic!
Let me lay down a few points here:
Caddy is not just an HTTPS server, it is a server-of-servers, and cert automation features can be used by other applications as well. For example, it can be used to simply keep certificates renewed: Using Caddy to keep certificates renewed - Wiki - Caddy Community
Of course I'd be happy for LE to recommend Caddy; it's a solid recommendation. That said, it's not for everyone -- just like Certbot is great, but not for everyone. Still, I can see why LE hesitates to endorse Caddy.
As far as I know, Caddy is the only web server to enable HTTPS by default (meaning: no config needed to "turn on HTTPS" -- it is assumed) and manage certificates.
Caddy's cert automation library is CertMagic, which uses ACMEz to implement the low-level ACME requests. I would submit that its stack is perhaps the most mature ACME stack in the industry in terms of age + breadth of deployments + closeness to the original vision of ACME.
I've personally met and worked with some/most of the founders of Let's Encrypt and the ACME protocol. My impression is that their intent has always been for HTTPS to be automatic and default, built directly into the servers and applications as a native function that "just works" -- not tacked on as an afterthought. But knowing that the transition to a fully-encrypted Web would be tedious if everyone had to replace their software, they developed what became Certbot, since a drop-in CLI command on a cron that updates your server config is an easy way to transition to an encrypted Web. But it was never the long-term plan. It's time we had integrated ACME clients. (Certbot has been super crucial and I applaud the engineering behind it, btw.)
In this talk, Josh Aas demonstrates Caddy as "the ideal client experience" and it has been considered the "gold standard" implementation for about 8 years now:
We hope to see other popular server software follow Caddy’s lead.
So in a way, Let's Encrypt has (at least unofficially) recommended Caddy from the beginning. I think this could be emphasized more to encourage more applications to embed ACME features and accelerate the transition to HTTPS-default infrastructure.
I think since then a lot of the focus has understandably been on dealing with the massive growth of the CA's utilization in recent years, and enhancing PKI generally -- but I for one think we as client developers need to do more to integrate ACME into our applications natively.
CertMagic and ACMEz are ideal libraries to do this for Go programs.
The ISRG also wants to transition the Internet away from using C code at the edge -- that means nginx and Apache and OpenSSL to be replaced by memory-safe alternatives. Their memory safety initiative, Prossimo, has some very promising starts there.
Caddy is ready-to-go today though. I understand for some it can be tricky to replace existing infrastructure, but we need to do it sooner or later. And I strongly advise any organization doing so to consider memory-safe programs with ACME clients natively built-in.
Certbot isn't the right tool for everyone, yet Let's Encrypt endorses it. I could see the same thing happening with Caddy. Even if it's not a good fit for someone, it's at least a good example of what the whole point of Let's Encrypt is.
(I've communicated my opinion privately already but will share it here).
I think for greenfield projects, Caddy (or some other next-generation webserver) might be a fine choice. There is less risk going with a (comparatively to say, nginx) less proven server if you're building something new.
I am using Caddy for personal use.
If you have an established infrastructure stack, it is scary. I feel genuine fear about using anything other than nginx or haproxy as a web frontend, in my day job where I am doing infrastructure. For this reason, I'd like to see work put towards transitional technologies that will make life better for existing stacks, e.g. first party (or just much better integrated) support to existing webservers.
For example, I am a big fan of mod_md, maybe we should start pushing that as an official recommendation for Apache httpd users. It has to be packaged by the current LTS distros by now?
I think really the issue isn't "recommend Certbot" or "recommend Caddy", but having the Client Options page be a lot more comprehensive than it is now about what use cases the user is trying to solve and give better and more narrowly-tailored recommendations.
Even if it's not as complex as a "wizard" type interactive tool, if the paragraph at the top suggested a few different preferred options for different situations, that might be nice. (I mean, if securing IIS on Windows, one should really be using one of the Windows-native tools rather than certbot or Caddy, I would think.) And I think it's been said on these forums over and over again that listing a bunch of options by their implementation language (kinda, except for the Windows section?) doesn't really make any sense.
Let's Encrypt has always recommended Certbot. I get why.
But we are now 8 years in, and I do think we can be more helpful than "here's the one recommended way to get started."
It could be something simple, like this:
Do you have an existing site and server? We recommend Certbot temporarily to help you transition to HTTPS.
Are you deploying a new web server? We recommend Caddy.
Are you using other, specific infrastructure? Check their feature list or documentation for ACME / Let's Encrypt support.
Otherwise, here's a list of known clients.
How can a non-profit organization have a competitor (esp. one that is a for-profit company)?
I can say that their role with Caddy has essentially been:
Fund its full-time development (which has been crucial)
Add ZeroSSL as a second default CA for redundancy (which was a big step forward for the ecosystem)
... and I sent them some Caddy swag once
Oh, and they reported a bug that we fixed.
The Caddy project has been thriving since ZeroSSL's sponsorship, and Caddy has various other sponsors as well, with diverse goals and specialties. Not really a bad thing!
The more I think about this, the less I think the discussion should be about "which client do we recommend" and more about "how do we get more servers and applications to embed ACME support natively". Sure, use Caddy as an example -- and let's upgrade the Web!
I think Caddy is great software and often recommend it, but I have issues with it's private ownership/sponsorship. I would likely stop investing any time or resources with LetsEncrypt as a whole if it were the official recommendation.
Personally,
I do not volunteer to benefit for-profit businesses.
I do not support projects that have competing business to valuable open source initiatives. Building equity into Caddy means building equity into the commercial ZeroSSL platform and expanding their userbase. That is inherently detrimental to the long term success and stability of the LetsEncrypt initiative.
This has nothing to do with the particular ownership of Caddy, although I know several members here have had issues with that company before. I have been very vocal of the "free" and "open source" initiatives by other for-profit companies on this forum.
@jvanasco I respect your values and of course appreciate your recommending Caddy sometimes; but I'm not sure I follow the reasoning in your post.
This is confusing to me.
If you have ever watched an ad, you are donating your time and attention to benefit for-profit businesses.
If you submit a patch to an open source project used by for-profit businesses, your volunteer contribution is benefiting for-profit businesses.
If you help someone on this forum who is using Let's Encrypt in their company, you are benefiting for-profit businesses.
I think I understand this in general but what I don't understand is, what is ZeroSSL taking away from Let's Encrypt? I think we've seen LE staff multiple times on the forum mention how commercial ACME CAs are a good thing because they offer services to businesses that Let's Encrypt doesn't offer, so it expands the use of ACME and thus, spreads privacy and security on the Web.
Let's Encrypt's goal isn't to certify every site, it's to transition the Web to HTTPS.
(Kind of reminds me about how Tesla's mission isn't to sell the best and most EV cars. It's to transition the world to sustainable energy. So if someone else makes more cars and better EVs, great! That's what they're trying to do.)
Can you explain why this follows?
This makes sense if Caddy is proprietary software, but it's not. It's open source that they pay to fund. I don't see how that builds their equity. Their incentive is strategic market positioning, not to make money from Caddy. (I speak for myself btw, I'm not an employee of theirs nor do I represent them.)
Again, remember that Let's Encrypt is funded / sponsored by many industry partners who could be considered "competitors" ... if LE were a for-profit company. But they're not. It's hard to see how any honest business out there is "detrimental" to Let's Encrypt when so many of them actually fund it or further its mission by providing ACME CAs themselves.
I think people do need a bit more guidance on finding/selecting the right tech as there's a lot of noise on the existing client options page. Frankly, there's stuff on there that nobody is using.
I'm a big caddy fan, but not a frequent user currently (I have used it in production though). It would be a wise choice in something that guided users where to get started, especially if they are not currently invested in Apache/nginx.
On the other hand, nobody wants their client to be missing from an official list of recommended clients (and how do they get added if they're not on the list, what criteria is there), so therein lies the problem of showing favouritism towards specific tools.
I think it's possible that the "average" LE user barely knows what a webserver is, nonetheless have the knowledge/confidence to replace one of the extremely established ones (e.g. nginx, Apache) with Caddy, of which I respect the development and mission, but don't feel it's established/"available" enough for the typical LE user. The "stack" that any particular user has chosen heavily influences their ACME client possibilities with many niche circumstances being possible (e.g. shared hosting).
Going along with @griffin, I think the same about the "average" LE user applies to the current recommendation of Certbot. If they don't know what a web server is they probably don't know how to use the command line and administer a system, either.
Maybe it's been enough time now that the LE docs can simply recommend to check the feature/function list of their software/server/application to see if they can use ACME/LE, and to simply recommend using software/servers/applications that have ACME built-in.
I'd probably agree with that. Less than 10% of clicks to my site come from the LE client options page and I put that down to the amount of other content on that page. Realistically someone looking for the particular solution my app provides isn't going to find it on that list, especially since other devs have come along and put theirs higher in the list or listed themselves twice.
Most people just google it, and most people don't (initially) know what an ACME client is to specifically google that phrase, so instead they are looking for "free certificates for my website" etc. I'd wager it's a technical subset of site-operators that ever reach the client options page content, let alone read it.
Regarding "default" recommendations, apparently 28% of websites are running Windows Linux vs. Windows usage statistics, November 2023 - not sure about that myself, but that should reasonably affect guidance, but it generally is a somewhat grudging secondary or tertiary consideration. The dominant web server on Windows will always be IIS unless MS integrates something else.
So technically isn't Caddy owned by a commercial company instead of merely sponsored?
I'm quite sceptical about promises about the future from commercial companies. It wouldn't be the first time an open source project suddenly wouldn't be open source any longer and would require a fork by the open source community due to actions from the owning company.
To be honest, the same can be said for nginx being acquired by F5. It has had an effect on the development direction. I think commercial interests are a valid reason to be sceptical, but at the same time, it's not like this is a super recent upstart. Track record counts for something and we should be pragmatic. I'm not going to use Apache httpd because it is more purely open source - it also happens to be like 30 years old and not very good.
