I ran this command:
certbot + some local custom hooks to handle the dns population
... --expand --verbose --cert-name wildcard.oriondev.net -d *.oriondev.net -d oriondev.net
It produced this output:
Renewing an existing certificate for *.oriondev.net and oriondev.net
Reusing existing private key from /spirent/certmgmt/config/live/wildcard.oriondev.net/privkey.pem.
Performing the following challenges:
dns-01 challenge for oriondev.net
dns-01 challenge for oriondev.net
Running manual-auth-hook command: /spirent/certmgmt/bin/certbot-hook.py
Hook '--manual-auth-hook' for oriondev.net ran with error output:
2023-05-30 23:50:09 INFO [:main:51] attempting validation for 'oriondev.net' with '9BdGGaLB_B_jO3tVoTIPhxJXu0ecNQVGXvOM_Cfs3Cc'
2023-05-30 23:50:10 INFO [:main:146] UPSERT: _acme-challenge.oriondev.net IN TXT "9BdGGaLB_B_jO3tVoTIPhxJXu0ecNQVGXvOM_Cfs3Cc"
2023-05-30 23:50:10 INFO [:main:177] record UPSERT completed in background
Running manual-auth-hook command: /spirent/certmgmt/bin/certbot-hook.py
Hook '--manual-auth-hook' for oriondev.net ran with error output:
2023-05-30 23:50:11 INFO [:main:51] attempting validation for 'oriondev.net' with 'GlVKmIjy1Q6tXshk2mzwvgrhRzgpSHPl6O41QrU9F8E'
2023-05-30 23:50:11 INFO [:main:146] UPSERT: _acme-challenge.oriondev.net IN TXT "GlVKmIjy1Q6tXshk2mzwvgrhRzgpSHPl6O41QrU9F8E"
2023-05-30 23:50:17 INFO [:main:162] checking if change is in sync (elapsed = 0)
2023-05-30 23:50:22 INFO [:main:162] checking if change is in sync (elapsed = 5)
2023-05-30 23:50:27 INFO [:main:162] checking if change is in sync (elapsed = 10)
2023-05-30 23:50:33 INFO [:main:162] checking if change is in sync (elapsed = 16)
2023-05-30 23:50:38 INFO [:main:162] checking if change is in sync (elapsed = 21)
2023-05-30 23:50:43 INFO [:main:162] checking if change is in sync (elapsed = 26)
2023-05-30 23:50:44 INFO [:main:175] record UPSERT completed
Waiting for verification...
Challenge failed for domain oriondev.net
dns-01 challenge for oriondev.net
The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
1.29.0
I'm mainly trying to determine if this is expected behavior or not, and if so, is the expected "validation" to have BOTH TXT records populated - i.e. a TXT record lookup for _acme-challenge.oriondev.net should return both the 9BdGG.... and the GlVKmI... content records?
If it is, it's purely something I need to fix in my local hook to tolerate the concurrency. If it's not, any idea what I'm doing to trigger the behavior?
Edit: Just noticed how horribly out of date certbot is, will update that regardless.
Thank you for the confirmation that it's expected!
To answer a few of the questions above:
Yes, know about dns-route53 plugin (it's even installed on this deployment, just no longer being used), but for multiple reasons it cannot be used directly.
Reason for the hooks - multiple accounts/different creds involved, and the plugin cannot currently cope with multiple "public" domains for the same domain (it doesn't know which one to use, and doesn't have any fallback like 'update every matching one').
Both of these are too much edge-case to accomodate in native certbot code.
Two public route53 registrations are used for split-zone support - 'private' route53 registrations come with a whole pile of behavioral baggage (like you can't delegate anything with NS records) that makes it hard to integrate into an on-prem dns hierarchy.)
Running on ubuntu 20 lts latest patches with certbot in a venv and (after I submitted) fully current.
So, the end summary is - I just need to make some tweaks to my hook. It's just hard to believe it's been "working" all this time by failing once (getting ONE of the validations right), and then working on the retry (getting the other). Time to fix that in my code.