Certbot snap package and Python Integration

Client dev
1

Hello,
I have been working on this framework idea for a central SSL repo. The basic idea is:

  • Certbot client = responsible for cert creation,renewal
    • installed via snap
  • Python application (systemd) = responsible for watching Certbot for changes (within the /etc/letsencrypt/)
    • When change is detected, do some updates to 3rd party systems with the cert changes

One of the big issues seems to be folder permissions to read files within /etc/letsencrypt when called via systemd. This seems to stem with an issue of snapd packages and systemd.
(Snap packages run in a confined environment that conflicts with systemd service security restrictions). [environment variables, user context, path resolution, security models)

  • I have attempted to get around these restrictions but, seems the workarounds get very messy and not a great way to implement the solution

Alternative installs:

  • I would like to stay with the snap approach, as it is the recommended way and will be best supported in the future
  • pip is a "best effort" only support, so not a good way foundation to build on
  • APT prefer to keep/stay with the recommended way. But maybe this is my only choice
  • Docker adds a level of complexity with this I am trying to avoid

The ASK:

  • any suggestions on how I might get around this issue?
  • is my only real alternative to use APT package?

Thanks

2

Do you have to use certbot? Perhaps you'd be better served by a different client, one that's more portable. Lego seems like one such option.

3 Likes
3

I forgot to mention;

  • I am not using the pre/post hooks as they are too tightly coupled to the Certbot application
    • meaning that I need my "check" application to be independent of Certbot to capture any anomalies/edge-cases where Certbot might miss

i.e. - cert is created/deleted/renewed and the hook did not run (for whatever reason), trying to track/ensure that process is not lost, is difficult. Without the ability of a "watcher" logic, I cannot really ensure the 3rd-party system is kept in-sync with changes within Certbot.

4

Sure, but the files in /etc/letsencrypt/ should be accessible by any application. Those files are outside of the snap. Otherwise Apache or nginx et c. also shouldn't be able to read those files?

Only thing is that the user needs to be root to be able to access the private key(s). As far as I know, there are no other limitations.

1 Like
5

Just trying to keep it as simple as possible. I figured if I use as many "stock" parts as possible, then maintaining this going forward makes life easier.

I could write the functionality of a "scaled-down Certbot functionality" but then the additional technical debit resides with me.
maybe my only real solution

6

One of the issues I ran into was my "watcher" app was still trying to use the (/usr/bin/certbot -> /snap/bin/certbot -> /usr/bin/snap) to acquire some cert info.
The output of this was necessary to audit what Certbot certificates it thinks it has vs actual files.

edge-case:

  • raw cert files are deleted from /etc/letsencrypt/live/
    • Certbot still thinks it has certs but no files for the "watcher" to actually parse.
      I currently do not have another idea how to account for this type of case besides using the Certbot commands.
7

Note that this output is NOT backwards compatible and could change at any time. There has been some proposals/pull requests to add JSON output, but that was also stalled for many, many years and not implemented. So you shouldn't "trust" the certbot output and expect any application to use the output to parse it somehow to break.

Isn't it suffiicient to parse the files to see if everything is "OK"? Calculate the time till expiry and see if Certbot actually renews in time should be enough, right?

That should never happen. No application other than Certbot should modify /etc/letsencrypt/. Also, Certbot will error out hard if some files are missing.

2 Likes
8

lego is a "stock" part. It is a CLI based ACME Client which runs on many platforms.

1 Like
9

  • I was piping the Certbot output in 'jc --certbot --pretty' to make the output more "transferable"

    • agree and was an acceptable risk with Certbot output changing. But I had the fallback that I would not upgrade Certbot (with breaking changes) until my application took those into account.

  • to the point of the edge-case below, these 2 components are not completely tied to each other. They can be manipulated separately.

    • if Certbot kept all that info/values within a DB, then it would be harder to have them out-of-sync (not impossible)

  • the parsing had additional steps to actually also gather the cert and key values; to be uploaded into a 3rd-party system. [As the intention of this project was to make this instance of Certbot a central certificate repo (instead of having multiple systems running individual Certbot to their own needs. Less things to manage)]

  • agree but, if something was being malicious or someone careless, they could delete that directory. I needed a way to detect that (for many reasons). In real-world environments, you never know what someone will do/try, so I try to protect the app from as many "reasonable" edge-cases.

    • Certbot will throw an error if the files are missing. I had already built a backup feature that would allow me to restore the Certbot files and recover my tracking database; just for this scenario.

Not trying to provide an argument points, just trying to explain my "chaos of thoughts" going through the workflow. Maybe my initial idea how to integrate this app was flawed, with my limited knowledge of Certbot app.
Maybe be the LEGO approach is the better way to handle this.?.?

10

Some food for thought ...

I'll assume you have a known list of certificates in this system

So, start by looking at the actual .pem files pointed at by the ../live/.. symlink. Look at the fullchain.pem and privkey.pem

Decode the fullchain.pem (or cert.pem) to ensure it has enough life remaining. This proves Certbot is running correctly and that nothing is missing.

Then compare these .pem to the .pem you copied to your 3rd party system. If different apply the new ones from Certbot

All of the above does not use Certbot at all. Just use a bash script, or python, or whatever other language you prefer. Many offer x509 decoding of .pem files.

Because of this, you could swap out Certbot for a different ACME Client to create your "pool" of certs. All that would change is where this other ACME Client stores the cert files.

1 Like
11

...and, of course, the complete removal of snap from the equation.

And really, when it comes to ACME clients, they're all third-party--it's been a great many years since ISRG had an "official" one (though their continued preference for certbot baffles me).

2 Likes
12

Sure, but, their problem with snap was a red herring. I think they were trying to use the Certbot API to validate what Certbot CLI is doing. That doesn't achieve what they wanted since they were concerned about rogue activities damaging Certbot or its "database".

Best to use an independent system to validate the Certbot certs and sync with their 3rd party system.

But, yes, if they abandon Certbot as the ACME Client they wouldn't need snap for that. They didn't provide their o/s but it may well have snap pre-installed so ...

1 Like
13

If using Certbot as the client, your best and only option is pip. The snap system autoupdates, so you have to use their "cohort" system to pin things. Certbot can change things at any time. IMHO the cohorts thing is unwieldy and a complete mess.

  • pip is a "best effort" only support, so not a good way foundation to build on

The "best effort" is for installation support. It is simply installing the source. As long as Certbot is developed in Python, pip is your best option for stuff like this.

The chances of Certbot changing the underlying data format are pretty much zero. I do not like how Certbot stores files, I advocated for it to be changed years ago when this was a still a remote possibility. Certbot is too widely deployed and utilized at this point for a change like this to realistically ever happen.

Speaking as someone who maintains their own Python ACME client and often contributes to Certbot, I would install Certbot and your monitoring app in the same Python virtualenv, and have Certbot as a dependency to your application with version pinning. You can monitor the certbot directory, but it would make more sense to use a --deploy-hook to send the new certs to your application.

You could also use Lego as mentioned above. I generally don't like mixing technologies though, and you've already targeted Python for your application.

Anyways, my client does a bit of what you want. I used it internally for a few years, open sourced a fork, then about a year ago I started a major rewrite. I am nearly done with that.

It's a Python based ACME Client and Certificate Manager, and paired with an OpenResty plugin that dynamically loads certs into nginx. The client is contained in a webapp that offers an API to systems on your LAN. Certs/Orders can be managed through the webapp or a JSON api. There are also commandline scripts that do everything. Certs are stored in a sqlite database (Postgresql used to work, it might still) but can be optionally maintained on the filesystem.

It is designed for deployments where you have many domains or many servers, and need centralized ACME management. It is specifically not for novice users or simple deployments.

The repo is here -- GitHub - aptise/peter_sslers: or how i stopped worrying and learned to love the ssl certificate

3 Likes
14

Thanks. I will take a look at your repo. It does sound like it checks a lot of the boxes I was trying to do.

15

So thanks to all the replies here, as they were constructive to provide some insight and ideas. I wanted to get an answer to my question, either way:

  • "here is what you need to make that work"
    OR
  • "you are doing it wrong"

TL;DR (for those who find this string looking for the same answers as me)

  • Probably using the Certbot app is not the best base to use to build the application on (the basic suggestion from almost everyone here)
  • Better option is to build/maintain your own ACME client or use LEGO (@danb35 had suggested)
  • Can use 'peter_sslers' (@jvanasco ) to accomplish a lot of this or fork and use as a base

Basically the answer is, "don't do what I tried, it will not work the way you expect"

1 Like