Parallel run of certbot manual-auth-hook/manual-cleanup-hook for validation of multidomain cert

Help
1

Hi,

I have a multi-domain cert that I request to get created, it has 30 SAN names, 28 of them are wildcard domains. I am using certbot (v1.2.0 (docker)) with a cli.ini config below.

cerbot executes the manual auth hook, the hook creates the dns entries on Google Cloud DNS, waits for the dns entry to be replicated on all the name servers and then a additional sleep for 15s. The additional sleep was necessary as domain verification kept failing if the auth hook exited right after the the dns check on all the name servers. The error was NXDOMAIN.

This process is taking 45 mins to generate the certs! this is because for each domain the manual auth hook is being called serially.

  1. Is there a way this can the parallelized?
  2. Why is there a need for extra time (sleep 15s) after the DNS entry has already been verified on all the name servers of the registered domain? (it would fail with NXDOMAIN even after 5s of sleep, so I changed it to 15s.)
  3. Which DNS servers is Let’s Encrypt
    CAA is checking to verify the DNS challenge? (from the blog post it looks it is checking the domains authoritative name servers: https://www.eff.org/deeplinks/2018/02/technical-deep-dive-securing-automation-acme-dns-challenge-validation. I could not find this information in the docs.)
command

(ran inside a docker container):

certbot certonly --cert-name "${DOMAIN}" --domains "${DOMAIN},${SUB_DOMAINS}"
cli.ini 
rsa-key-size = 4096
agree-tos = True
email = <redacted>
server = https://acme-v02.api.letsencrypt.org/directory
test-cert = False
non-interactive = True
preferred-challenges = dns
manual = True
manual-auth-hook = /opt/certbot/scripts/manual_auth_hook.sh
manual-cleanup-hook = /opt/certbot/scripts/manual_cleanup_hook.sh
manual-public-ip-logging-ok = True
manual_auth_hook.sh (excerpt) 
timeout 120 sh << EOF
NAME_SERVERS="$(dig +short -t NS ${DOMAIN} | tr '\n' ' ')"
echo "NAME_SERVERS=\${NAME_SERVERS}" &> /proc/1/fd/1
for NS in \$NAME_SERVERS; do
  echo "NS=\${NS}" &> /proc/1/fd/1
  until dig +short -t TXT ${ACME_CHALLENGE_DOMAIN} @\${NS} | grep -q -- "${CERTBOT_VALIDATION}"; do
    echo "$(date -Is) waiting for DNS resolution of ${ACME_CHALLENGE_DOMAIN} to show ${CERTBOT_VALIDATION} on \${NS}" &> /proc/1/fd/1
    sleep 1
  done
  echo "resolved on \${NS}" &> /proc/1/fd/1
done
echo "resolved on all name servers: \${NAME_SERVERS}" &> /proc/1/fd/1
EOF

echo "$(date -Is) sleeping for 15s for DNS propogation." &> /proc/1/fd/1
sleep 15

Thanks!

2

There isn't currently a solution to this problem for manual auth hooks.

There are some issues and/or PRs in the Certbot GitHub repo discussing various approaches -- for example, some sort of "this is the last one" environment variable -- but nothing has been merged yet.

Can you use the certbot-dns-google plugin? Python plugins work with a different API and are capable of handling this sensibly (create all the records immediately, sleep once).

Google has lots of DNS servers around the world. You and Let's Encrypt might be querying different ones.

Let's Encrypt runs resolvers on their own infrastructure.

3

Hi @mnordhoff,

Thanks for the reply!

So the main reason for not using the cerbot-dns-google plugin was to avoid using service accounts. We did not want to create, manage service accounts and keys associated with it. Instead we rely on already provisioned Google Cloud user accounts that are tied to the user identity throughout the firm. The manual auth hook was the only option that we could use.

The manual-auth-hook is using gcloud to create the dns entries.

manual-auth-hook (excerpt) 
gcloud_dns="gcloud --quiet --project=${GCP_PROJECT} dns record-sets transaction"
${gcloud_dns} start --zone="${DNS_ZONE}" &> /proc/1/fd/1
${gcloud_dns} add --name="${ACME_CHALLENGE_DOMAIN}" --ttl=60 --type=TXT --zone="${DNS_ZONE}" -- "${CERTBOT_VALIDATION}" &> /proc/1/fd/1
${gcloud_dns} execute --zone="${DNS_ZONE}" &> /proc/1/fd/1

Do you have a link to the PR/issues that you mentioned? I would like to look through those and see if I can use one of them to speed this up.

Thanks!

4

Here's one I could find:

5

Thank you @mnordhoff!! That was very helpful.

I found a feature request, that if implemented will help me a lot: Feature request: Add option to know what is the last domain passed to dns hook script · Issue #5484 · certbot/certbot · GitHub.

6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.