Post Hook with rsync failure using systemd

sebelk · December 2, 2020, 8:50pm

I ran this command: /usr/bin/certbot -q renew (really through systemd timer on Debian 10)

It produced this output:

2020-11-27 11:00:09,298:ERROR:certbot.hooks:Hook command "/usr/local/bin/copy-certs-mta.sh" returned error code 1
2020-11-27 11:00:09,299:ERROR:certbot.hooks:Error output from copy-certs-mta.sh:
root@mx1.silicon-aid.com: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
root@mx1.silicon-aid.com: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
root@mx1.silicon-aid.com: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot 0.31.0

The weird thing is that if run on manually:

certbot --dry-run certonly --webroot -w /srv/web --post-hook /usr/local/bin/copy-certs-mta.sh -d mx1.silicon-aid.com

the POST_HOOK script runs with no problem.

Even more: I've created a pair of units for testing both timer and service, that runs only my bash script and it works fine, the files are copied...

Am I missing something?

Thanks in advance!

_az · December 2, 2020, 9:00pm

Does the SSH key you're using (where the hook runs) have a passphrase on it?

Have you tried explicitly passing the SSH key to the rsync command?

sebelk · December 2, 2020, 9:05pm

Yup, It has passphrase.
I've triple-checked it, I use keychain for ssh-agent re-using.
It's weird, only as a post_hook in a systemd timer fails...

Osiris · December 2, 2020, 9:07pm

It seems your keychain doesn't like ~~cron~~systemd timers..

sebelk · December 2, 2020, 9:09pm

It's not cron it's a systemd timer. And I've said in my first post If I create a systemd timer unit with only my bash script rsync works fine.

_az · December 2, 2020, 9:18pm

You could try run env in your deploy hook and see whether the relevant variables (like SSH_AUTH_SOCK) are survivng.

As far as I know, Certbot shouldn't be wiping out those environment variables. But worth verifying.

rg305 · December 2, 2020, 10:10pm

May we see this command file?:

sebelk · December 4, 2020, 3:00pm

2020-12-04 11:35:31,983:INFO:certbot.hooks:Running post-hook command: env ; /usr/local/bin/copy-certs-mta.sh
2020-12-04 11:35:32,354:INFO:certbot.hooks:Output from env:
LANGUAGE=es_AR:es
JOURNAL_STREAM=9:180581844
COLUMNS=80
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INVOCATION_ID=b33a28737e294c73876e9dd134f1d4ce
LANG=es_AR.UTF-8
PWD=/
It looks as is does not preserve the environment, does it?

LINES=24

2020-12-04 11:35:32,354:ERROR:certbot.hooks:Hook command "env ; /usr/local/bin/copy-certs-mta.sh" returned error code 1
2020-12-04 11:35:32,354:ERROR:certbot.hooks:Error output from env:
root@mx1.silicon-aid.com: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
root@mx1.silicon-aid.com: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]
root@mx1.silicon-aid.com: Permission denied (publickey).
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: unexplained error (code 255) at io.c(235) [sender=3.1.3]

_az · December 4, 2020, 8:53pm

Why do you say that?

For example, JOURNAL_STREAM is injected by systemd as it launches Certbot, and you're seeing it in the hook execution. From that, I would say that the environment is preserved. I do not think that Certbot would strip SSH_AUTH_SOCK, there's no logic like that.

What I suspect instead, is that processes started by systemd timers are outside of the process hierarchy where SSH_AUTH_SOCK is available.

I tried the same thing on my machine (just a simple timer, not Certbot), and indeed there was no SSH_*.

I'm no expert on how this all works, but that seems intuitive to me. Your SSH agent is created by your tty or desktop environment or whatever. Your systemd timers are not descendants of those, so they wouldn't inherit anything from them.

I think you would have to figure out how to:

Start up the ssh-agent -s independently of your tty and run ssh-add etc with your passphrases. Also, importantly, preserve the SSH_AUTH_SOCK=... output and store it somewhere.
In your --deploy-hook, restore the SSH_AUTH_SOCK value before calling rsync.

rg305 · December 4, 2020, 9:08pm

Morale of the story: "Global Variables" aren't always as Global as we might expect.

sebelk · December 9, 2020, 1:15pm

#!/bin/bash
shopt -s expand_aliases
source /usr/local/etc/vars-certs
HOST=mx1
rsync   --chmod=F644  /etc/letsencrypt/live/$HOST.$DOMAIN/fullchain.pem $HOST.$DOMAIN:/etc/pki/tls/certs/$HOST.$DOMAIN
rsync   --chmod=F644  /etc/letsencrypt/live/$HOST.$DOMAIN/cert.pem $HOST.$DOMAIN:/etc/pki/tls/certs/$HOST.$DOMAIN
rsync   --chmod=F600   /etc/letsencrypt/live/$HOST.$DOMAIN/privkey.pem $HOST.$DOMAIN:/etc/pki/tls/private/$HOST.$DOMAIN

[ $? -eq 0 ] && ssh $HOST.$DOMAIN systemctl restart postfix

And /usr/local/etc/vars-certs contains:

DOMAIN=silicon-aid.com
alias rsync='rsync -uvL -e ssh'
source /root/.keychain/snappy-quiz-sh

And finally/root/.keychain/snappy-quiz-sh

SSH_AUTH_SOCK=/tmp/ssh-FyQvwyh0ZBpf/agent.30232; export SSH_AUTH_SOCK;
SSH_AGENT_PID=30233; export SSH_AGENT_PID;

sebelk · December 9, 2020, 1:17pm

I've tried with:

post_hook = bash -c "source /root/.keychain/snappy-quiz-sh" ; env; /usr/local/bin/copy-certs-mta.sh

in renewal file, without success...

EDIT 1:

Well, I've narrowed the problem, I got rid out of script, I've just have in renewal file:

env; rsync -e 'ssh -vvv -i /root/.ssh/id_ecdsa -i /root/.ssh/id_ecdsa-cert.pub'  --chmod=F644  /etc/letsencrypt/live/mx1.silicon-aid.com/fullc
hain.pem mx1.silicon-aid.com:/etc/pki/tls/certs/mx1.silicon-aid.com

And in systemd unit file I've added:

ExecStartPre=bash -c "source /root/.keychain/snappy-quiz-sh"

So I have the environment:

2020-12-09 12:13:49,376:INFO:certbot.hooks:Output from env;:
LANGUAGE=es_AR:es
SSH_AGENT_PID=30233
JOURNAL_STREAM=9:185360731
COLUMNS=80
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
INVOCATION_ID=bbb53d199e6a4e5aad7578d087092a46
LANG=es_AR.UTF-8
SSH_AUTH_SOCK=/tmp/ssh-FyQvwyh0ZBpf/agent.30232
PWD=/
LINES=24

However rsync/ssh still ignores at all the agent:

debug1: read_passphrase: can't open /dev/tty: No such device or address

It should happen something like that:

debug2: sign_and_send_pubkey: using private key "/root/.ssh/id_ecdsa" from agent for certificate

I report here because if I create a systemd unit by hand, timer with rsync and ssh works well (it uses the agent and files are copied), but not with certbot...

system · January 8, 2021, 1:17pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.