Not Recommending Certbot Because of Reliance on Snap

Site Feedback
1

The Getting Started page on the website heavily steers people in the direction of Certbot:

We recommend that most people with shell access use the Certbot ACME client.

This may have been fine originally, but more recently the people running Certbot have been making it increasingly difficult to install Certbot without using the snap package manager. Lately the team has completely removed most of the distribution-specific support instructions in favor of snap.

There is a long list of reasons people want to avoid using snap, including:

  • Avoiding installing a new package manager just for a single piece of software
  • Snap requires a new daemon to be running all the time
  • Snaps using mount points thus cluttering up the system mount points
  • The auto-update feature, which cannot be turned off, which undermines both when and whether a package gets updated
  • Ideological reasons having to do with snap not using a fully open, free model.

Of these, the first four concerns seem more applicable in a production server environment due to security and stabiliity issues. And I'd imagine that certbot would mostly be used in a production server environment.

There are a whole bunch of great alternatives to certbot, but they're mostly not as well documented. The list itself is also staggeringly long which can lead to some decision fatigue. Many of us may find that certbot doesn't meet our needs specifically because of its reliance on snap, but we don't want to exhaustively research this. We would rather there be a strong consensus or at least a strong secondary recommendation on the part of Let's Encrypt.

So, I guess this is what I'd like to see. I'd like for us to question whether or not it makes sense to continue recommending certbot, given their pressuring of their userbase to use snaps, and also I'd like, whether or not Let's Encrypt continues to recommend certbot as the first option, for us to coalesce behind one or possibly more than one recommended other option(s). Also, I don't know if there is much overlap in the management of Let's Encrypt and certbot, or if Let's Encrypt has any influence over the certbot team, but if they do have this influence, I would also love to see people steering the certbot team in the direction of moving away from snap. Numerous users have already spoken out about this on the certbot forums but it seems to have fallen on deaf ears. To me, this seems to warrant either external pressure, abandonment of recommendation for certbot, or at a bare minimum, stronger recommendation of some sort of alternative.

8 Likes
2

Welcome to the Let's Encrypt Community, Alex :slightly_smiling_face:

Thanks for the great feedback. Your timing is impeccable as we've recently been having such discussions.

4 Likes
3

Thank you!

And this is great to hear! I actually stumbled across some of this discussion here too, when I was searching for information on how to install certbot without snap, and it was one of the factors that encouraged me to create the account here and bring up the topic of certbot's recommendation on the Let's Encrypt website!

4 Likes
4

So the question comes round to what alternative distribution method should they support (and I mean support, as in, are expected to provide). Currently you have the choice of:

  • Snap
  • Build your own copy (this is how all linux software used to be distributed as I'm sure you're aware).

For the certbot team the important questions are going to be how is dependency management handled, how can production builds be created and automatically published, how does updating working, and who's going to manage and test it all.

It's tempting to imagine that the developers have to climb out of money filled swimming pools just to update their release notes but I suspect resource constraints are tighter than they might first appear.

At the root of this you are probably asking for yum packages to be built for various systems, but if it was just based on popularity then they'd have to do apt etc and raw .rpm and .deb etc, for each supported processor architecture.

Outside of certbot there are other alternative tools such as bash-only scripts etc. You can also manage your certificate renewals externally then publish updated certs to target boxes (or regularly pull from secrets vaults).

While I do see your points and do understand the perspective, none of your reasons are anything other than preference based (there are no technical reasons, and you're annoyed that you have to work a certain way), so your choice of client should be as well.

6 Likes
5

@cazort could use an ACME client like mine (CertSage) that only depends upon having working PHP and does not require root access. Incidentally, CertSage and CertifyTheWeb are the only ACME clients with which I am familiar that are GUI-driven and do not require command-line. I suppose if someone were so inclined, they could use a script to submit a form POST to CertSage with all the requisite parameters to automate its usage. It's not really intended for large scale, but could be with a little creativity. :thinking:

5 Likes
6

I wonder how much of a security risk would:

  • spinning up a VM to handle all the renewals for a particular entity
  • If authenticating via DNS or HTTP, then throw in CNAMEs to point to this VM
  • Setup the ACME client to renew using the same private key original CSR (thanks @rmbolger)
  • Copy out all individual private keys to all the systems needing certs
  • Remove all private keys from the VM
  • Renew all certs on a regular basis using only the original CSR (thanks @rmbolger)
  • Schedule systems to update their own public certs after pulling down latest cert from public record

If any of that makes any sense to you, you've probably had as much to drink as I have.
And now we're both wondering...
Can it be that simple?
[only one VM with whatever ACME client you chose and some clever scripting to update public cert]
[VM may be overkill for the home user, but who doesn't have a spare RPi and an IPv6 address to use]

And how much of a pain relief it might be for those that can't/won't risk their production systems with added software(s).

Things to consider:

  • DNS auth doesn't require a web server
  • For HTTP auth: ACME clients can spin up a temporary HTTP server (don't require a web server)
  • No synchronization required with the dedicated VM (only the first private key)
  • When NOT rotating private keys, only ciphers that support PFS should be used.

[I may add more as consciousness returns]

3 Likes
7

If the identifiers in the cert haven't changed and you still have a copy of the previous CSR, then yes. Otherwise, no because you need to generate the CSR which requires the private key.

Coincidentally, this is how Posh-ACME allows for automated renewal even when users supply an externally created CSR. It just caches that original file and uses it for all subsequent renewals (or until the user supplies a new one).

4 Likes
8

Noted!

So use the first (only) private key to create the first CSR.
Then copy out the private key and destroy the local copy.
Then reuse first CSR to renew all subsequent certs ... until the cows come home!
[do they? nah, they won't - they never do on their own - dang party cows!]

3 Likes
9

Certbot is also actively distributed to Pypi to be installed with pip and is sort of supported (best effort).

6 Likes
10

This approach is supported by Certify The Web (currently that runs on Windows though! Linux version in the works).

You can centralise certificate renewals with DNS validation (and use custom CSRs if you like), then you can automatically update targets via SSH/SFTP using a deployment task, you can also add remote script tasks for things like service restarts.

For complicated stuff I like the idea of just storing to a secrets vault then including system pull from there on a schedule. This is especially useful for multi-instance stuff so you don't do near any rate limits.

Likewise you can also achieve all this using certbot on a vm + scripting.

4 Likes
11

By the same token, the use of a TLS certificate in the first place is nothing more than preference-based, right?

4 Likes
12

Please do disagree. The option is there to build the software yourself, which is the beauty of open source. You can even create and maintain your own packages (under different names). OpenSSL (for instance) has any number of distributions maintained by lots of people.

It's not an approach I favour (if it can be avoided) but it can still be done if you need more choice and are prepared to do the work. My key point is that someone needs to be prepared to do the work, and keep doing it.

4 Likes
13

Indeed, it's rather easy to criticise a hard-working team whereas the possibility exists to do it all yourself too.

6 Likes
14

I’m not sure why being “preference based” should invalidate the concerns.

Requiring users to run the snap daemon 24/7 for a piece of software that they need to run for a few minutes every ~90 days seems like a bad requirement that the LE team should not want either.

This argument always strikes me as the Open-Source Cop-Out.

Most users are not going to have the technical know-how to do this, even if they did have the time to do so.

So it comes across more as a way of shutting people up than actually considering what is being said than anything else.

IMO, obviously.

5 Likes
15

Looks like we disagree. No I'm not telling anyone to shut up, and I don't work for LE so my opinion holds no weight anyway.

Yes, I do feel that objections to running snap are based a preference. I don't mean that to be snarky - if there is a technical reason let me know. Running a daemon or not is also a preference. Likewise there are people who will gladly die on their sword before running systemd or before using nano in preference to vi. That's their preference.

4 Likes
16

I saw this and it seems preferable to snap. Has anyone actually tried this though? I searched a lot and I couldn't find a single testimony of someone having tested this method out successfully.

4 Likes
17

I think I agree with you, but I also don't necessarily think something being a "preference" means it is not important. I agree it is still possible to install certbot, and I understand the rationale (i.e. avoiding dependency hell) that people used for going to snap.

At the same time, my "preferences" have a lot behind them. "Dependency hell" results from a package having a lot of dependencies, and I think that this shows that the team maintaining the package weighed features and/or their convenience of coding, over the build being lean and simple.

The existence of scripts like acme.sh which are extraordinarily lean and have minimal dependencies, but seem pretty powerful still, suggests to me that the Certbot team could do better at keeping dependencies minimal.

Again, this is a "preference". It's based on my philosophy of coding and system administration, which is that I am, probably more averse than average to adding a new dependency, installing a new daemon, etc. And this preference results from life experience, notably, years ago when a server under my control was hacked due to an exploit in a package that I wasn't really using and didn't really need to be there. This experience led me to tighten things up and since then, no box under my control, neither as an employee, contractor, or my own personal boxes, has ever been compromised. I also have also saved a ton of money over the years, paying minimal hosting fees for hosting high-traffic sites that a lot of others would pay 10 times as much, or more, to host, and this is in large part because of these "preferences" of mine to keep things lean and minimal. And I've made a ton of money off clients who paid me to save them money. So, this general approach in life, seems to have great value to me.

So you could say I have a "preference" for my servers to be secure and for my expenses to stay minimal. But...it's also more than a preference, it's been my business model, my livelihood, my lifestyle even.

And...in the case of an individual software package like Certbot, or snap even, the security risk and the load on the server will be minimal. But here's the thing. Servers are complex. In configuring a server, I make not one, but hundreds, or perhaps thousands of choices. This package or that? Enable or disable this? How many different options in each configuration file? How many hundreds of configuration files? And a large portion of these choices have implications in terms of security and efficiency.

It's a slippery slope argument. And it's a question of precedent. And it's a question of what the broader principles are behind the choices. And in my "preferences", installation of a new package management system just to install a single package is a bad precedent to set. Same with the extra daemon. And if I did it with this package, chances are I would do it with others.

I also think that there is a "creep" here in the software development community as a whole. When the community tolerates one popular or widely-used package (and certbot is both) moving towards something like snap, it lends the whole construct of snap legitimacy. And it lends legitimacy to all the things that go into it: the bloat that led to the dependency hell that led to snap being embraced in the first place, and the "decision creep".

In my daily life, I'm negatively affected by bloat almost all day long as I use the computer. Nearly all webpages these days have heavy amounts of bloat. Many packages I use on my desktop computer, and on my server, have bloat. Bloat slows me down. It increases my electric bill because it cranks my CPU. It decreases my quality life as I sit there waiting for something to load.

So the choices we as a small, specialized community around Let's Encrypt make about a package like certbot, have consequences far and wide, for other software packages. If developers who actively push snap on their userbase see them accept it with open arms, other developers will likely follow suit. But if people give heavy pushback, other developers may be hesitant to do the same. And they might adopt other approaches to resolving "dependency hell"...like...gasp...actually making their code and projects more lean and minimal, maybe cutting out unnecessary dependencies? Maybe, just maybe, cutting out bloat? Which, as far as I'm concerned, is the holy grail here, and why I care about this whole thing in the first place.

So...the broader principle also seems important to me. I hope this explains that even if this is a "preference", it is a preference with consequences, both to the individual, and to the broader society.

8 Likes
18

I don't have any reason to think why it shouldn't work? A Python virtualenv maintained by pip seems to be a little bit more difficult to maintain "by hand" I've heared tho.. So it's not the recommended method and is only supported at best effort.. It's sort of a Hail Mary backup, but should work nonetheless.

5 Likes
19

:grin:

Optimization/minimization was one of my primary goals when developing CertSage in the first place. I grew up as a DOS-kid (with some Commodore 64 prior to that), so I come from an age of valuing optimization. I remember adding bits of assembly to my C++ code for game engines because the Borland graphics libraries were terribly bloated. I remember disabling a ton of crap in MASM because the default bloat from Microsoft added junk to my executables I didn't need.

I feel like the real argument as a developer here comes down to how much one does oneself versus using a "black box" to accomplish certain things. How far down do I want to dig? How much do I want to reinvent the wheel? Ultimately, this is the nature of any development endeavor throughout history. As software developers, I feel like many of us place the floor at the standard libraries included with the baseline distribution of our chosen language. Even then, having developed in easily more than a dozen languages, I can say that the standard libraries can often be lacking. C-POSIX anyone? STL? I can say that PHP has more than a few warts and some really terrible evolution in both language constructs and library development/maintenance. Even the expansion of languages (and platforms) can be seen as bloat. More confusion for a confusing world. That said, as our Lego bucket seems to keep getting weirder and more complex pieces added, do we stick with the bricks? The saying is that choice is better than no choice. Choice isn't free though. It has to be weighed, selected, made, and lived with.

5 Likes
20

The reason is because Certbot is a Free and Open Source Software, and creating+testing distributions to run on every platform was a significant amount of work -- work that took away resources from fixing bugs and implementing features.

Most Linux distributions did not keep Certbot up-to-date across their supported versions either - there was a point where every version of Ubuntu that was still in the active support window was running a significantly different version of Certbot, some of which were critically out-of-date.

Snapd gave the core maintainers the ability to keep Certbot up-to-date on the widest amount of systems, and conveniently takes the least amount of work.

I am not a fan of Snapd either, but anyone who has surfaced the concerns of the OP is most likely technically able to:

  • script a system to start/stop snapd periodically
  • avoid snapd and install + update Certbot from source or pip

Again, I don't like snapd myself. I understand the OP's concerns and have the same ones. I have one server that needs to run Certbot via snapd, and I hated having to install snapd just for a single package - but scheduling it has minimized the annoyance. snapd is not right for everyone, but it is right for the overwhelming majority of users, and the remaining users are typically technically able to avoid it.

(edited for bad grammar)

10 Likes