Not Recommending Certbot Because of Reliance on Snap

Linking related topics together:

https://community.letsencrypt.org/t/provide-support-for-commercial-acme-clients-such-as-acme-sh/156764?u=griffin

https://community.letsencrypt.org/t/discussion-should-clients-like-acme-sh-even-be-listed-on-the-lets-encrypt-website/156136?u=griffin

3 Likes

@griffin Not everybody can visit that category :slight_smile:

5 Likes

True. I'm not sure that some of what's said in the #lounge topics is appropriate for public consumption though.

3 Likes

For minimal dependency stuff also check out GitHub - srvrco/getssl: obtain free SSL certificates from letsencrypt ACME server Suitable for automating the process on remote servers.

4 Likes

I totally agree software bloat (and yet-another-package-manager/yet-another-daemon) is a noble thing for all developers to actively work to avoid. I think @griffin and I share more than a little of our software development lineage with resource constraints.

In the case of certbot, once you have python3 dependencies etc it kind of draws a line that it's never going to be a micro-client and instead they are opting to be able to take advantage of things like configuration providers and dns libraries in order to get breadth of functionality.

So on the original topic, is there a real lean alternative currently? Are the bash-based scripts the go-to for minimal dependencies or has anyone developed a fully featured (enough) single-executable acme client? I'm seeing GitHub - ndilieto/uacme: ACMEv2 client written in plain C with minimal dependencies - for me DNS challenges are the elephant in the room for most clients, because they're both the messiest scenario and one of the most useful validation methods.

@cazort are you looking for an alternative client that LE will actively recommend or one they will actively develop? Currently they can only recommend certbot because it's the one they take responsibility for.

5 Likes

Actually, maybe just ignore my questions, this is already all in a duplicate thread: Migrating Servers: Want to Also Stop Using Certbot - #15 by cazort

4 Likes

I think it's also worth asking why the Certbot developers think it's a bad idea for people to pick one version of Certbot and stick with it.

Now that the ACME v2 transition is out of the way, it seems to me that for people with ordinary needs there should be no reason to upgrade a working Certbot setup short of a security bug.

As I understand it, the upcoming Debian stable release will include Certbot 1.12. If there's a reason why that's not going to continue to work well with Letsencrypt's servers for the next three years, perhaps that indicates something that could be improved on Letsencrypt's side.

4 Likes

While I'm not a certbot engineer, I can guess a little bit: security issues/fixes, new features, bugfixes? Just as with almost every other piece of software?

Note that Let's Encrypt as wel as certbot are quite new relatively speaking and had some major changes in the past.. Now as everything has settled down a little bit, it probably wouldn't be such a major issue, but there are plenty of reasons to upgrade regularly left IMHO. But hey, I run Gentoo Linux and would never ever want to turn to Debian based distro's, mostly due to the lack of recent versions of software..

5 Likes

Right, I think things have settled down enough that Let's Encrypt should be aiming for "Just Works if you leave it alone", rather than "Just Works if you continually update your client software".

4 Likes

@mjw I'm pretty sure that if someone would install version 1.0.0 it would function for many years to come already.

However, I strongly believe upgrading to newer minor versions is always a good idea. One could obviously discuss about upgrading to newer major versions due to possible incompatibilities between the new and old version, but upgrading minor/patch versions should never be an issue (this is of course the task of the development team) and could improve your experience with software greatly. Therefore, it would be stupid not to upgrade IMO.

3 Likes

If Let's Encrypt expect any Certbot >= 1.0 to work, then (going back to @cazort's original suggestion), it might make sense for Let's Encrypt's documentation to recommend that people who don't like snaps should install Certbot from distro packages, rather than recommending something other than Certbot.

2 Likes

The 1.0.0 was just something I thought up, not Let's Encrypt. Also, I'm not aware of documentation recommending something else than certbot if snap is not to be used?

2 Likes

That hypothetical documentation is the subject of the "site feedback" post you are commenting on.

Security-updates and breaking-changes aside...

The LetsEncrypt system does not implement 100% of the ACME specification. There are clear divergences and implentation details where the spec allowed some interpretation. The boulder implementation is under constant development to fully support the spec - which in recent times has included account key rollover, error codes, and additional validation requests.

Consequently, aside from the various feature improvements that Certbot constantly makes to improve it's core functionality, perfect installers and support more validation methods, there are constant release cycles that deal with implementing new LetsEncrypt functionality.

Certbot 1.12 was released on Feb 2 2021; it is already 6 months old and has been outdated by 6 subsequent releases. see CHANGELOG.md Some of these changes are substantial. It would be better for Debian's users if they supported more current versions. Many of the issues addressed were not security related, but deal with important functionality or bugs.

Any version of Certbot that supports ACME v2 should be compatible with LetsEncrypt. The pain points for using the older software against ACME would be in supporting newer letsencrypt features, and utilizing the correct acme endpoints by default; there are far more changes that deal with better supporting installing certificates and authorizing via DNS. (I think initial acme v2 support and wildcards happened in release 0.22.0 on 2018-03-07).

In any event, my point is that Certbot has two primary functions:

  • A LetsEncrypt client
  • A Certificate installer

The "required" compatibility updates to the client have typically been minimal, however there are often many new client features that are needed.

It's great that you feel the client is feature-complete enough for you, but this does not necessarily hold true for all users. I don't think account-key rollover has landed in Certbot yet (I could be wrong on this, I use my own client for that).

When it comes to the installers and streamlining procurement issues, the Certbot team has been doing a lot of work to perfect that. They are catching edge-cases, supporting new situations, and often use these forums to find common issues that need to be addressed. That functionality of Certbot needs the most work and to constantly be updated. The older Certbots will probably still work, but they won't work as-well.

8 Likes

Speaking primarily in a personal capacity, I believe that @jvanasco has hit the nail on the head with both of his comments above.

Regardless of one's personal preferences for or against Snap, I think it is important to recognize two things:

  1. Regularly updating software -- everything from the OS to your webserver to your ACME client -- is critical for proper security.
  2. Using Snap provides a built-in auto-update mechanism.

These two combine to provide a powerful motivation for switching to Snap for distribution, so I understand the certbot team's direction here. Even if you think that this is an incorrect move, or that the tradeoffs were not properly weighed, or that Let's Encrypt should stop recommending certbot as a result, it is important to be empathetic to the software engineering constraints inside which the cerbot team is working.

Now speaking as a representative of Let's Encrypt: We do not intend to stop recommending certbot.

Certbot remains one of the most popular, most well documented, most actively maintained, most regularly updated, and most flexible ACME clients available. It is still a good choice for most people, most of the time. We do not believe that this change constitutes a reason to stop suggesting it.

10 Likes

I will continue to use acme.sh as long as it is working with letsencrypt. Hopefully it should be as easy as apt update in the future.

2 Likes

I use this method in all my servers since a few months ago. No problems at all (I tried using snapd but had problems).

If you know a bit about python it's very simple to setup.

6 Likes

I am not sure how package updates work with linux, but is it hard to push updates to repositories? is it a extremely long and complicated process? If it is why not launch a precompiled binary so that people can use it. Hopefully someone who knows how these apt update works explain it to me.

2 Likes

Because there's no binary to compile; it's all python scripts. But the issue isn't the compiling, it's the packaging and maintaining a repository to store the packages. Though with that said, I thought GitHub (at least) was able to automate a lot of that work.

3 Likes

Yes and Yes. Each Linux distribution has multiple versions in their active support windows, so there are dozens of repositories. Each distribution essentially forks the project and alters it to fit their codebase and style for the given version. (They don't actually use forks, but generate diffs to be applied against a specified source version).

For example, Ubuntu alone has 6 versions in their active support window. Certbot must be patched differently and run through testing for each and every version. Doing this once is a lot of work, which is why the Ubuntu maintainers only upgrade select versions of Certbot onto select distribution versions.

The Certbot team could potentially have their own repository, but that makes installation more complex (users would have to authorize a 3rd party repo), creates a lot of work for the Certbot team to maintain a repo, and puts a lot of work on them to create and test distribution packages for each and every version.

This is talking about ubuntu alone - the workload is exponentially higher when you consider all the linux/bsd/windows variants. Certbot's official instructions officially support 20+ platforms, but there are dozens of other linux platforms that support Certbot as well.

The precompiled binary would have to be tailored to each and every distribution at a specific version. The binaries only simplify installation, not building and patching for a specific operating system at a specific version. In the case of Certbot, nothing is compiled and everything is runtime - but there is still a need to patch the code to meet each operating system's build or requirements.

Snapd provides a consistent sandboxed environment across operating system platforms and versions. This means the Certbot team only has to focus on supporting a single platform/version combination for end users - snapd - not dozens.

Again, I need to stress this point, which I don't think many people understand -- neither the Certbot team nor the various Linux/BSD/OS release teams have anywhere near the resources required to support every version combination. If anyone had these resources, we'd see the most-current version of Certbot available for every single operating system's package manger for every available version. Most of the people doing this work are unpaid volunteers, though some are employed by a distribution, software project, or a company that allocates some of their hours to supporting open source efforts. Everyone on all sides of this equation has been working on supporting things with a best-effort policy, and the reality is that severely out-of-date Certbot packages are everyone's best effort.

The reality is that trying to support each system and version natively has been an epic and colossal failure. A fallback was certbot-auto, which somehow worked in many cases but was incredibly fragile and needed to be replaced. If you read the changelogs and annoucements, the Certbot team has plainly stated the adoption of snapd was directly driven by those failures.

Again, I need to stress that I am not a fan of snapd. I begrudgingly use it on one server out of many. I mostly use my own client, and on some systems I install Certbot from pip since that's within my comfort zone. For the vast majority of users across the globe though, snapd is the best solution by a wide margine -- and I am glad the Certbot and LetsEncrypt teams found this way to make their services available to the greater population.

11 Likes