Adding end-to-end monitoring with shell scripts (no dependencies)


#1

I was trying to figure out the simplest way to add a cloud service monitoring to Let’s Encrypt agents. The ultimate goal of end-to-end monitoring is to provide a report with the state of your certificates audited from the internet, rather than locally.

The most popular agent seems to be certbot so I looked at that one first.

Basically, there are 3 main options:

  1. use certbot’s configuration subfolder “renewal-hooks”, which is in Linux usually at ‘/etc/letsencrypt/’ folder
  2. a more generic approach approach - invoking certbot with command line parameters, which specify “hooks” - scripts, which are called at particular phases of processing (e.g., –pre-hook, –post-hook, –deploy-hook, etc)
  3. wrap certbot into another script, which will use certbot internally.

The least intrusive is option 1, as it doesn’t require any changes to the server configuration. All you have to do is copy a script into a pre-defined folder. The downside is that certbot doesn’t provide any environment information (unlike option 2).

The goal was to minimize dependencies, from which the most serious is networking. I didn’t like the idea of depending on curl or other heavy libraries. The solution was to go back to basic networking - TCP sockets. We extended the KeyChest monitoring API to accept TCP requests. The server integration code is then amazingly simple:

exec 3<> "/dev/tcp/keychest.net/10023"
echo -e "command apikey" >&3
echo "$(dd bs=1000 count=1 <&3 2>/dev/null)"

The first line opens a connection, the second sends a command to the API, the last one reads the response.

Voila, an API integration in 3 lines!

More details are described in my blog:

And an initial script, which: parses certbot logs and configuration to extract required information is in KeyChest GitLab:


#2

Sounds cool! But… What does this actually do?

Sorry, but I’m not really inclined to surf to any blog, just to read why I even would want this feature. I have no idea what it could do for me. The blog wouldn’t be just for details, it holds all the info I need to actually understand the goal of this thread. And that last bit makes it kinda spammy to me personally.


#3

It submits a domain name to user account in KeyChest - which is an HTTPS audit service. As a result you get expirations and info about all your domains in one place.