Let's Encrypt Proxy for Monitoring

We would like to publish an open-source Let’s Encrypt proxy for Certbot mid next week and I look for anyone keen to help us test it.

The default behaviour is to push updates out via an API but it will also simply create JSON log files. Those should be suitable to analyze against rate-limits, certbot failures, etc.

Please let me know if you’re interested - here, messaging, or via email: dan(at)keychest-dot-net .

Brief spec/reqs:

  • it’s a Python package - hopefully should be available via pip next week.
  • should run as a service (but thinking about it, it could be just launched with certbot)
  • use: something like this for cron:
    export HTTPS_PROXY=“https://< ip address >:8080”; export REQUESTS_CA_BUNDLE=<proxy’s cert path>; certbot [your existing params]
  • logs will probably go to the /var/log/letsencrypt folder by default

Quite a few integration / usability things that can be tweaked (like the proxy sending its root cert for storing in /tmp so you don’t have to copy it by hand, …)

Update 11th December:

KeychestAmp is available for testing

The first beta version of the light proxy that successfully renewed can be installed with pip and the package is published here: https://pypi.org/project/keychestamp/

The upgrade of the KeyChest service to accept logs via the API will be done in the next few days so only local logging for now (switch --noapi).

The proxy has to be running, a quick and dirty:
pip install keychestamp
nohup keychestamp --noapi -l debug 1>/tmp/debug.log 2>&1 &

Once the proxy is up and running, it can be used remotely or on localhost. A one command run against staging environment:

(curl -x "http://127.0.0.1:8443/" -s -k http://amp.keychest.net > /tmp/ca.crt; export HTTPS_PROXY="https://127.0.0.1:8443"; export REQUESTS_CA_BUNDLE=/tmp/ca.crt; certbot renew --dry-run --force-renewal)

  • curl will pull a root cert of the proxy and copy it to the /tmp folder.
  • HTTPS_PROXY - sets the proxy for certbot, you can replace 127.0.0.1 with an appropriate IP address
  • REQUESTS_CA_BUNDLE - tells Python to search for trusted root certificates in the new file
  • and the certbot command itself - with whatever parameters you need

The logs are by default stored in /var/log/keychestamp/audit.json and the format looks like this:

{
"msg": "new-nonce", 
"id": 252, 
"proxy": "j0rwv67lpim8fgk0ruep58pu@amp.keychest.net", 
"time": "2019-12-11 16:17:40", 
"host": "a3.keychest.net",
"ip": "127.0.0.1", 
"version": "0.2.7", 
"backlog": 0, 
"log_time": "2019-12-11 16:17:40",
"server": "acme-staging-v02.api.letsencrypt.org", 
"path":  "/acme/new-nonce",
"code": 200, 
"params": {
    "epoch": 1576081060, 
    "client": "::ffff:127.0.0.1",    
    "agent": "CertbotACMEClient/0.33.1 (certbot; CentOS Linux 7 (Core)) Authenticator/standalone Installer/None (renew; flags: frn n) Py/3.6.8"
}
}

An initial documentation is https://pypi.org/project/keychestamp/ , a Swagger Rest API documentation will be updated in the next few days at https://keychest.net/api#tab_2

Dan

1 Like

doesn't this place already used by certbot?

Indeed, I thought it may be good to keep it in the same folder (to prevent possible problems with privileges), but not 100% sure this reasoning is sound.

Our recommendation from our brief discussion is to use a different directory path instead.

2 Likes

That’s great! Is this proxy specific to Certbot? That is, could other organizations using non-Certbot clients use it too? I know some places have big deployments of ACME clients but have trouble monitoring what they all are doing, and this could be helpful for them.

Sure thing - thanks for the guidance!

1 Like

Certbot is what we use for testing but there is no specific dependency. ... as far as I can see at the moment. :slight_smile:

Getting closer ..... have tidied up the integration - the proxy can be either localhost or one instance for a LAN / network so that all logs can be collected in one place.

If the proxy were @ 192.168.42.2 the following would pull the proxy's root certificate and initiates certbot / ACMEv2 client with the traffic logged by the proxy.

(curl -x "http://192.168.42.2:8443/" -s -k http://amp.keychest.net > /tmp/keychest.crt; export HTTPS_PROXY="https://192.168.42.2:8443"; export REQUESTS_CA_BUNDLE=/tmp/keychest.crt; certbot certonly --dry-run -d domain.company.com --agree-tos -m dan@company.com)

@jsha @schoen The first version for testing is available now - see my update of the initial topic.
It may not be quite smooth yet, but the idea is that the following three lines will result in a successful renewal in staging.

pip install keychestamp
nohup keychestamp --noapi -l debug 1>/tmp/debug.log 2>&1 &
(curl -x "http://127.0.0.1:8443/" -s -k http://amp.keychest.net > /tmp/ca.crt; export HTTPS_PROXY="https://127.0.0.1:8443"; export REQUESTS_CA_BUNDLE=/tmp/ca.crt; certbot renew --dry-run --force-renewal)
1 Like

Some thoughts
Maybe a proxy can used to translate acme v1 to v2 so keep holding old clients alive
And non mitm mode (but server on different domain) as most client have --server option to use different server so remove the need of installing proxy cert but a real one
(But some clients may not have that)

This would require rewriting clients to work as you intend.

We have this option (to work with —server) in our AmpX proxy, which is a deep proxy as it needs its own server account. ACME2 checks server url inside protected content so you have to re-sign if anything changes.

Translation v1 -> v2 should be relatively simple - again with the AmpX. But I’m not sure how much need is there for this.

1 Like

Some changes to the message format - for faster inserts and database queries, the server and path (which was missing) is now in the top level.

{
"msg": "new-nonce", 
"proxy": "j0rwv67lpim8fgk0ruep58pu@amp.keychest.net", 
"time": "2019-12-11 16:17:40", 
"server": "acme-staging-v02.api.letsencrypt.org", 
"path": "/acme/new-nonce",
"code": 200, 
"params": {
    "epoch": 1576081060, 
    "client": "::ffff:127.0.0.1", 
    "agent": "CertbotACMEClient/0.33.1 (certbot; CentOS Linux 7 (Core)) Authenticator/standalone Installer/None (renew; flags: frn n) Py/3.6.8"
  },
"version": "0.3.1", 
"backlog": 0, 
"log_time": "2019-12-11 16:17:40",
"host": "a3.keychest.net",
"ip": "127.0.0.1",
 "id": 252
}

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.