Unable to use certbot docker container with manual-auth-hook and manual-cleanup-hook

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: dev.dynamic-biosensors.com

I ran this docker-compose.yml file:

version: '3'
services:
  certbot:
    container_name: certbot
    image: certbot/certbot:latest
    command: >
             certonly
               --preferred-challenges=dns
               --manual-auth-hook /app/authenticator.sh
               --manual-cleanup-hook /app/cleanup.sh
               --domain ${DOMAIN}
               --email ${EMAIL}
               --agree-tos
               --non-interactive
               --manual
               -v
    volumes:
      - ./etc/letsencrypt:/etc/letsencrypt
      - ./certbot/data:/var/www/certbot
      - ./certbot/logs:/var/log/letsencrypt
      - ./scripts/authenticator.sh:/app/authenticator.sh
      - ./scripts/cleanup.sh:/app/cleanup.sh

with a .env file containing

DOMAIN = test000.dev.dynamic-biosensors.com
EMAIL = it@dynamic-biosensors.com

and a two files in the scripts folder:

git@dbs-git01:~/certbot/scripts$ ls -la
total 16
drwxrwxr-x 2 git git 4096 Aug 10 14:50 .
drwxrwxr-x 6 git git 4096 Aug 10 14:50 ..
-rwxrwxr-x 1 git git 1153 Aug 10 14:33 authenticator.sh
-rwxrwxr-x 1 git git  822 Aug 10 14:34 cleanup.sh

It produced this output:

git@dbs-git01:~/certbot$ sudo docker compose up
[+] Running 1/0
 ✔ Container certbot  Recreated                                                                                                                                                                                             0.0s 
Attaching to certbot
certbot  | Saving debug log to /var/log/letsencrypt/letsencrypt.log
certbot  | Plugins selected: Authenticator manual, Installer None
certbot  | Requesting a certificate for test000.dev.dynamic-biosensors.com
certbot  | Performing the following challenges:
certbot  | dns-01 challenge for test000.dev.dynamic-biosensors.com
certbot  | Running manual-auth-hook command: /app/authenticator.sh
certbot  | Hook '--manual-auth-hook' for test000.dev.dynamic-biosensors.com reported error code 127
certbot  | Hook '--manual-auth-hook' for test000.dev.dynamic-biosensors.com ran with error output:
certbot  |  /bin/sh: /app/authenticator.sh: not found
certbot  | Waiting for verification...
certbot  | Challenge failed for domain test000.dev.dynamic-biosensors.com
certbot  | dns-01 challenge for test000.dev.dynamic-biosensors.com
certbot  | 
certbot  | Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
certbot  |   Domain: test000.dev.dynamic-biosensors.com
certbot  |   Type:   dns
certbot  |   Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.test000.dev.dynamic-biosensors.com - check that a DNS record exists for this domain
certbot  | 
certbot  | Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.
certbot  | 
certbot  | 
certbot  | Cleaning up challenges
certbot  | Running manual-cleanup-hook command: /app/cleanup.sh
certbot  | Hook '--manual-cleanup-hook' for test000.dev.dynamic-biosensors.com reported error code 127
certbot  | Hook '--manual-cleanup-hook' for test000.dev.dynamic-biosensors.com ran with error output:
certbot  |  /bin/sh: /app/cleanup.sh: not found
certbot  | Some challenges have failed.
certbot  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
certbot exited with code 1

My web server is (include version):

is not reachable from the internet, which is why we are running custom scripts. We know that the scripts work because we were able to receive certificates for the server manually using those scripts.

The operating system my web server runs on is (include version):

Ubuntu 22.04.1 LTS

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don't know):

yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

certbot:latest (see above)

Hi @Crown0815, and welcome to the LE community forum :slight_smile:

It seems to me that your volume mapping might be slightly off.

Personally, I don't like using "./":

That mapping depends on where you are calling it from.
Try:
/actual/full/path/:/mapped/path/

3 Likes

Hi @rg305,

Thank you for the warm welcome :slight_smile:

I tried the solution. I changed the docker-compose.yml file to this:


version: '3'
services:
  certbot:
    container_name: certbot
    image: certbot/certbot:latest
    command: >
             certonly
               --preferred-challenges=dns
               --manual-auth-hook /app/authenticator.sh
               --manual-cleanup-hook /app/cleanup.sh
               --domain ${DOMAIN}
               --email ${EMAIL}
               --agree-tos
               --non-interactive
               --manual
               -v
               --dry-run
    volumes:
      - ./etc/letsencrypt:/etc/letsencrypt
      - ./certbot/data:/var/www/certbot
      - ./certbot/logs:/var/log/letsencrypt
      - /home/git/certbot/scripts/authenticator.sh:/app/authenticator.sh
      - /home/git/certbot/scripts/cleanup.sh:/app/cleanup.sh

(I also added --dry-run because we had already hit the 5 times per hour previously. I assume this has no effect on the result)

As a response we receive

git@dbs-git01:~/certbot$ sudo docker compose up
[+] Running 1/0
 ✔ Container certbot  Created                                                                                                                                                                                               0.0s 
Attaching to certbot
certbot  | Saving debug log to /var/log/letsencrypt/letsencrypt.log
certbot  | Plugins selected: Authenticator manual, Installer None
certbot  | Simulating a certificate request for test000.dev.dynamic-biosensors.com
certbot  | Performing the following challenges:
certbot  | dns-01 challenge for test000.dev.dynamic-biosensors.com
certbot  | Running manual-auth-hook command: /app/authenticator.sh
certbot  | Hook '--manual-auth-hook' for test000.dev.dynamic-biosensors.com reported error code 127
certbot  | Hook '--manual-auth-hook' for test000.dev.dynamic-biosensors.com ran with error output:
certbot  |  /bin/sh: /app/authenticator.sh: not found
certbot  | Waiting for verification...
certbot  | Challenge failed for domain test000.dev.dynamic-biosensors.com
certbot  | dns-01 challenge for test000.dev.dynamic-biosensors.com
certbot  | 
certbot  | Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
certbot  |   Domain: test000.dev.dynamic-biosensors.com
certbot  |   Type:   dns
certbot  |   Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.test000.dev.dynamic-biosensors.com - check that a DNS record exists for this domain
certbot  | 
certbot  | Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.
certbot  | 
certbot  | 
certbot  | Cleaning up challenges
certbot  | Running manual-cleanup-hook command: /app/cleanup.sh
certbot  | Hook '--manual-cleanup-hook' for test000.dev.dynamic-biosensors.com reported error code 127
certbot  | Hook '--manual-cleanup-hook' for test000.dev.dynamic-biosensors.com ran with error output:
certbot  |  /bin/sh: /app/cleanup.sh: not found
certbot  | Some challenges have failed.
certbot  | Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
certbot exited with code 1

The file still cannot be found.

BTW: The other bindings work fine, e.g., it is possible to read the logs that were created in the container.

Originally we tried binding the scripts folder directly to app, but this lead to the same result.

Well, that's a bit of a letdown :frowning:

I would try listing that directory [within that container].
ls -l /app/
[Does it see anything there at all?]

Maybe there are permissions on that file/folder that prevent it from being accessed:
ls -l /home/git/certbot/scripts/authenticator.sh

2 Likes

I am not sure how to do this. I tried

git@dbs-git01:~/certbot$ sudo docker compose run certbot \
> ls -l /app/

but received

usage: 
  certbot [SUBCOMMAND] [options] [-d DOMAIN] [-d DOMAIN] ...

Certbot can obtain and install HTTPS/TLS/SSL certificates.  By default,
it will attempt to use a webserver both for obtaining and installing the
certificate. 
certbot: error: unrecognized arguments: ls -l /app/

Running

ls -l /home/git/certbot/scripts/authenticator.sh

produces

git@dbs-git01:~/certbot$ ls -l /home/git/certbot/scripts/authenticator.sh
-rwxrwxr-x 1 git git 1153 Aug 10 14:33 /home/git/certbot/scripts/authenticator.sh

Which to me looks all good, right?

I managed:

/opt/certbot # ls -la /app/
total 16
drwxr-xr-x    2 root     root          4096 Aug 10 16:50 .
drwxr-xr-x    1 root     root          4096 Aug 10 16:50 ..
-rwxrwxr-x    1 1000     1000          1153 Aug 10 14:33 authenticator.sh
-rwxrwxr-x    1 1000     1000           822 Aug 10 14:34 cleanup.sh

Since I managed to get into the container I ran

certbot certonly --preferred-challenges=dns --manual-auth-hook /app/authenticator.sh --manual-cleanup-hook /app/cleanup.sh --domain test000.dev.dynamic-biosensors.com --email it@dynamic-biosensors.com --agree-t
os --non-interactive --manual -v --dry-run

which produced

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Simulating a certificate request for test000.dev.dynamic-biosensors.com
Performing the following challenges:
dns-01 challenge for test000.dev.dynamic-biosensors.com
Running manual-auth-hook command: /app/authenticator.sh
Hook '--manual-auth-hook' for test000.dev.dynamic-biosensors.com reported error code 127
Hook '--manual-auth-hook' for test000.dev.dynamic-biosensors.com ran with error output:
 /bin/sh: /app/authenticator.sh: not found
Waiting for verification...
Challenge failed for domain test000.dev.dynamic-biosensors.com
dns-01 challenge for test000.dev.dynamic-biosensors.com

Certbot failed to authenticate some domains (authenticator: manual). The Certificate Authority reported these problems:
  Domain: test000.dev.dynamic-biosensors.com
  Type:   dns
  Detail: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.test000.dev.dynamic-biosensors.com - check that a DNS record exists for this domain

Hint: The Certificate Authority failed to verify the DNS TXT records created by the --manual-auth-hook. Ensure that this hook is functioning correctly and that it waits a sufficient duration of time for DNS propagation. Refer to "certbot --help manual" and the Certbot User Guide.

Cleaning up challenges
Running manual-cleanup-hook command: /app/cleanup.sh
Hook '--manual-cleanup-hook' for test000.dev.dynamic-biosensors.com reported error code 127
Hook '--manual-cleanup-hook' for test000.dev.dynamic-biosensors.com ran with error output:
 /bin/sh: /app/cleanup.sh: not found
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

I do not get it.

To double check I ran

vi /app/authenticator.sh

and the file opened in vi as expected

What's the contents of authenticator.sh? (Without any tokens or passwords et cetera, might there be any.)

Because the error is different if Certbot itself can't find the file:

Unable to find manual-auth-hook command /tmp/foobar.sh in the PATH.
(PATH is /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/bin)
See also the --disable-hook-validation option.

Your error is very different, so I'm thinking it has more to do with the script itself than Certbot.

1 Like

They seem to exclude the root user.

3 Likes

Have you tried mounting a directory with the scripts inside, instead of the script itself?

4 Likes

The contents of authenticator.sh are

#!/bin/bash

API_KEY="**********"

# Strip only the top domain to get the zone id
DOMAIN=$(expr match "$CERTBOT_DOMAIN" '.*\.\(.*\..*\)')

# Create TXT record
ZONE_ID="*******************"
CREATE_DOMAIN="_acme-challenge.$CERTBOT_DOMAIN"
RECORD_ID=$(curl -s -X POST "https://api.hosting.ionos.com/dns/v1/zones/$ZONE_ID/records" \
     -H     "X-API-Key: $API_KEY" \
     -H     "Content-Type: application/json" \
     --data '[{"type":"TXT","name":"'$CREATE_DOMAIN'","content":"'$CERTBOT_VALIDATION'","ttl":120}]')

RECORD_ID=$(echo $RECORD_ID | cut -d ',' -f 8)
RECORD_ID=$(echo $RECORD_ID | cut -d ':' -f 2)
RECORD_ID=$(echo $RECORD_ID | cut -d '}' -f 1)

RECORD_ID=$(echo "$RECORD_ID" | tr -d '"')

# Save info for cleanup
if [ ! -d /tmp/CERTBOT_$CERTBOT_DOMAIN ];then
        mkdir -m 0700 /tmp/CERTBOT_$CERTBOT_DOMAIN
        echo $ZONE_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
        echo $RECORD_ID > /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID
fi

# Sleep to make sure the change has time to propagate over to DNS
sleep 25

For completeness also the contents of cleanup.sh

#!/bin/bash

API_KEY="**********"

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID ]; then
        ZONE_ID=$(sudo cat /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID)
        sudo rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/ZONE_ID
fi

if [ -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID ]; then
        RECORD_ID=$(sudo cat /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID)
        sudo rm -f /tmp/CERTBOT_$CERTBOT_DOMAIN/RECORD_ID
fi

# Remove the challenge TXT record from the zone
if [ -n ${ZONE_ID} ]; then
    if [ -n ${RECORD_ID} ]; then
        curl -s -X DELETE "https://api.hosting.ionos.com/dns/v1/zones/$ZONE_ID/records/$RECORD_ID" \
                -H "X-API-Key: $API_KEY" \
                -H "Content-Type: application/json"
    fi
fi

These scripts had been run successfully before, but by manually executing certbot on the host system.

Yes, same result.

I think you are onto something. I changed the command to try accessing actually non-existing files

/opt/certbot/app # certbot certonly --preferred-challenges=dns --manual-auth-hook /opt/certbot/app/authenticator2.sh --manual-cleanup-hook /opt/certbot/app/cleanup2.sh --domain test000.dev.dynamic-biosensors.com --email it@dynamic-biosensors.com --agree-tos --non-interactive --manual -v --dry-run

and get the same result as @Osiris:

...
Unable to find manual-auth-hook command /opt/certbot/app/authenticator2.sh in the PATH.
(PATH is /usr/local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin)
See also the --disable-hook-validation option.
...
1 Like

Wasn't that at supposed to be at?:

OR

1 Like

Yes, sorry, I modified the docker compose file to inject the scripts in /opt/certbot/app, because /opt/certbot/ is the working directory of the container. But it makes no difference.

Relative and absolute paths also make no difference.

It seems like @Osiris' suggestion is correct and the issue is actually in the scripts itself, or at least with their execution. So certbot can find them, but then a call from certbot to

/bin/sh: /app/cleanup.sh`

fails

Could the problem be that there is no bash installed :expressionless:

2 Likes

I confirmed that running a normal sh script and a python script works as expected. The issue seems to be the missing bash installation.

Can anybody confirm?

Try mapping /bin/bash to the container.

2 Likes

Seems to be indeed:

osiris@erazer ~ $ docker run --entrypoint /bin/bash -it --rm certbot/certbot
docker: Error response from daemon: failed to create shim task: OCI runtime create failed: runc create failed: unable to start container process: exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown.
osiris@erazer ~ $ docker run --entrypoint /bin/sh -it --rm certbot/certbot
/opt/certbot # 

The alpine image where Certbots Docker image is based upon seems to be using busybox:

/opt/certbot # ls -l /bin/sh 
lrwxrwxrwx    1 root     root            12 Mar 29 14:27 /bin/sh -> /bin/busybox
/opt/certbot # 

(which uses ash under the hood if I Google correctly)

Maybe your script is Busybox ash compatible? You'd only need to change the shebang from #!/bin/bash to e.g. #!/bin/ash.

By the way: there's also no curl installed on that alpine image.. :roll_eyes: So that's another issue. Maybe just rewrite the script to Python? :stuck_out_tongue:

OR: considre using the GitHub - helgeerbe/certbot-dns-ionos: A certbot plugin for enabling DNS authentication with IONOS plugin? You'd need to add something to your Dockerfile to install it using pip though. Not sure how difficult that is, I'm not a Docker user.

3 Likes

That would be petty :smiley:

Ash is supposed to be POSIX compatible, you can use /bin/sh, but then you have to write your scripts in actual POSIX shell, not in bash, which is the proper way: if I'm going to assume I have bash, I can assume I have perl or python.

4 Likes