Problem with certbot manual and dns challenge

[rbrunetti]

Hello.
I’m trying to generate a wildcard let’s encrypt certificate using the DNS challenge and manual method.
The command I use is the following:

certbot certonly -n
–manual
–preferred-challenges=dns
–manual-public-ip-logging-ok
-m my-email
–manual-auth-hook /root/cerbero/certbot/generate_new_cert_pre.sh
–manual-cleanup-hook /root/cerbero/certbot/generate_new_cert_post.sh
–server https://acme-staging-v02.api.letsencrypt.org/directory
-d my-domainname -d *.my-domainname

Inside the manual-auth-hook and manual-cleanup-hook I update via API my DNS, so that the challenge can be verified.
As far as I can see, certbot performs two challenges, since I can see two TXT records being added and then deleted in my DNS (I suppose because I request and *..)

The problem is that somehow randomly the challenges (or one of the two) fail. If I repeat the request for that domain it could work, but it’s not deterministic.

Can you help me to understand what’s happening? Could it be an issue related to the TXT record TTL? I also tried to lower it to 10s, but without success.

Thanks a lot.
Kind regards
Riccardo

[Osiris]

@rbrunetti

Can you help me to understand what’s happening? Could it be an issue related to the TXT record TTL?

Probably not related to the TTL, because, as far as I know, TTL is related to caching. But the Let's Encrypt DNS client always asks the authorative DNS servers, which should deliver an uncached response.

It could however be due to propogation: the time between your script requesting the change via the API and the actual updating of the zone files in all the authorative DNS servers.

You could debug this by using --debug-challenges and while certbot is waiting for user input to continue after deploying the challenges, you can ask all the authorative DNS servers for the TXT records and check if they are there.

[JuergenAuer]

Hi @rbrunetti

@rbrunetti

Can you help me to understand what’s happening?

your domain name is required to check if your name server configuration is buggy.

[rbrunetti]

Hello Osiris.
I made another test, without success. This is the command:

certbot certonly -n --manual --preferred-challenges=dns --manual-public-ip-logging-ok -m si@host.it --debug-challenges --manual-auth-hook /root/cerbero/certbot/generate_new_cert_pre.sh --manual-cleanup-hook /root/cerbero/certbot/generate_new_cert_post.sh --server https://acme-staging-v02.api.letsencrypt.org/directory -d corsodijoomla.it -d *.corsodijoomla.it

and this is the output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for corsodijoomla.it
dns-01 challenge for corsodijoomla.it
Waiting for verification...

---

Challenges loaded. Press continue to submit to CA. Pass "-v" for more info about
challenges.

---

Cleaning up challenges
Failed authorization procedure. corsodijoomla.it (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA" found at _acme-challenge.corsodijoomla.it

IMPORTANT NOTES:

• The following errors were reported by the server:

  Domain: corsodijoomla.it
  Type: unauthorized
  Detail: Incorrect TXT record
  "PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA" found at
  _acme-challenge.corsodijoomla.it

  To fix these errors, please make sure that your domain name was
  entered correctly and the DNS A/AAAA record(s) for that domain
  contain(s) the right IP address.

While the command was running, I checked our auth DNS servers and I found:

First:

dig +short @185.84.96.5 _acme-challenge.corsodijoomla.it TXT && dig +short @185.84.97.5 _acme-challenge.corsodijoomla.it TXT
"9FCIGew5jouDS29neRrnL0jSdKo4ISnrojV_FvCC4VA"
"9FCIGew5jouDS29neRrnL0jSdKo4ISnrojV_FvCC4VA"

And after a while:

dig +short @185.84.96.5 _acme-challenge.corsodijoomla.it TXT && dig +short @185.84.97.5 _acme-challenge.corsodijoomla.it TXT
"PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA"
"PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA"

So, it seems to me that the correct TXT records where inserted into the DNS.

Thanks fot your help.
Riccardo

[JuergenAuer]

Hi @rbrunetti

@rbrunetti

And after a while:

dig +short @185.84.96.5 _acme-challenge.corsodijoomla.it TXT && dig +short @185.84.97.5 _acme-challenge.corsodijoomla.it TXT
“PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA”
“PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA”

So, it seems to me that the correct TXT records where inserted into the DNS.

that looks wrong. There are two entries with the same value.

But that command -d corsodijoomla.it -d *.corsodijoomla.it requires two entries with the same name, but different values.

Checking your domain there is no TXT entry visible - https://check-your-website.server-daten.de/?q=corsodijoomla.it#txt

[rbrunetti]

Hello.
In the dig command I reported above, I checked both our two auth DNS servers: 185.84.96.5 and 185.84.97.5.
Thus it’s correct to have two answers (not entries) with the same value. This is exactly what I wanted to check.

In this moment the entries have been deleted from our servers, since inside the manual-clean-up hook I remove the TXT records.

Regards.
Riccardo

[JuergenAuer]

@rbrunetti

Thus it’s correct to have two answers (not entries) with the same value. This is exactly what I wanted to check.

But then it's the wrong result. Then you should see 2 answers with 4 entries (complete, 2 entries per answer).

[rbrunetti]

Hello.
So, if I understand correctly, inside the manual-auth-hook I should write both the challenges?

_acme-challenge.corsodijoomla.it TXT “9FCIGew5jouDS29neRrnL0jSdKo4ISnrojV_FvCC4VA”
_acme-challenge.corsodijoomla.it TXT “PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA”

If the answer is yes: how can I retrieve both of them? At the moment I get the content of the variable: $CERTBOT_VALIDATION

Thanks
Riccardo

[JuergenAuer]

@rbrunetti

So, if I understand correctly, inside the manual-auth-hook I should write both the challenges?

_acme-challenge.corsodijoomla.it TXT “9FCIGew5jouDS29neRrnL0jSdKo4ISnrojV_FvCC4VA”
_acme-challenge.corsodijoomla.it TXT “PQPdK1a0K_p7jOKBTrQOZAgoymjIRtEztvo69XYlPCA”

Yes, that's correct and required because you want to validate two domain names - corsodijoomla.it and *.corsodijoomla.it. So the non-wildcard-part is the same, so both validations require _acme-challenge.corsodijoomla.it.

@rbrunetti

If the answer is yes: how can I retrieve both of them? At the moment I get the content of the variable: $CERTBOT_VALIDATION

No idea, never tested

[rbrunetti]

Ok, thank you very much for your help, JuergenAuer.
Does anybody can give me some hint on how to retrieve both the challenges that I need in order to put them into our DNS?

Thanks
Riccardo

[Osiris]

@rbrunetti

Does anybody can give me some hint on how to retrieve both the challenges that I need in order to put them into our DNS?

According to my simple and quick test, the output of certbot suggests it runs the scripts as many as times as necessary. So if you'd put two domains to validate on the command line, it runs the auth and cleanup scripts twice:

Test with foo.example.com and bar.example.com:

Are you OK with your IP being logged?
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
(Y)es/(N)o: y
Running manual-auth-hook command: /tmp/auth.sh
Running manual-auth-hook command: /tmp/auth.sh
Waiting for verification...
Challenge failed for domain bar.example.com
Challenge failed for domain foo.example.com
http-01 challenge for bar.example.com
http-01 challenge for foo.example.com
Cleaning up challenges
Running manual-cleanup-hook command: /tmp/cleanup.sh
Running manual-cleanup-hook command: /tmp/cleanup.sh
Some challenges have failed.

My auth script just outputted the variables to a text file:

Domain: bar.example.com
Validation string: AbCdEfG0123456789.myaccount
Remain. chal.: 1
All domains: bar.example.com,foo.example.com

Domain: foo.example.com
Validation string: XyZ0987654321.myaccount
Remain. chal.: 0
All domains: bar.example.com,foo.example.com

Using the dns-01 challenge only the value of the validation string changes (the account key part is removed), but the rest stays the same.

Luckily no rocket science here!

[schoen]

@Osiris

According to my simple and quit test the output of certbot suggests it runs the scripts as many as times as necessary.

That is exactly right. The CERTBOT_VALIDATION variable will give you only one value, but your hook script is run once per challenge. So each time, it should add the record that was indicated to it that time.

[rbrunetti]

Hello Schoen and thanks for your reply.
It seems that everything works if I do not delete the TXT record inside the manual-cleanup-hook.
In this case, the record content is simply written with the first challenge and then replaced with the second challenge.
The only side-effect is that the second TXT will remain inside the zone and I need to delete it at the very end.
I can’t really understand why this happens, but this is what I’m seeing “experimentally”.
Kind regards
Riccardo

[Osiris]

@rbrunetti

It seems that everything works if I do not delete the TXT record inside the manual-cleanup-hook.

That's strange, because if you look at the certbot output, the cleanup hook is only ran after both the challanges have been performed by the validation server. Are you sure your auth-script isn't just overwriting any previous existing TXT record? The idea is that an auth-script is adding the TXT record only, not overwriting it.

[rbrunetti]

Hello Osiris.
Now I modified the scripts in order to add both the challenges and then delete them inside the cleanup hook once for all.
It seems this solves the issue.
Thanks everybody for your help.
Riccardo