Multi-level Wildcard Renewal is failing

jeffsani · February 14, 2020, 9:38pm

I ran this command:

Options used in the renewal process

[renewalparams]
account = 2b8838f9899d92d51a3f2ba91aa3dd0c
manual_public_ip_logging_ok = True
renew_hook = /etc/letsencrypt/renewal-hooks/deploy/deploy.sh
manual_auth_hook = /etc/letsencrypt/renewal-hooks/pre/authenticate.sh
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = manual
pref_challs = dns-01,

It produced this output:
{
“identifier”: {
“type”: “dns”,
“value”: “mountaintrips.com”
},
“status”: “invalid”,
“expires”: “2020-02-21T20:59:43Z”,
“challenges”: [
{
“type”: “dns-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Incorrect TXT record “h_O_dPHpTQIkmGgooCMDPsQ2Z16W5auV09GCRohIB4Q” found at _acme-challenge.mountaintrips.com”,
“status”: 403
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997906/glTonA”,
“token”: “78sHNSOTnPuUkXd_KmqICLlF9YRg9ICjtQLFZIB4Vo8”
}
],
“wildcard”: true
}

My web server is (include version):
manual

The operating system my web server runs on is (include version):
CentOS Linux release 7.7.1908 (Core)

My hosting provider, if applicable, is:
godaddy - dns, web - self-hosted

I can login to a root shell on my machine (yes or no, or I don’t know):

yes

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):

certbot 1.0.0

The original certificate requests authenticated successfully. When I went to run the renewals however, the validation failed. Here is the config:

renew_before_expiry = 30 days

version = 0.39.0
archive_dir = /etc/letsencrypt/archive/mountaintrips.com
cert = /etc/letsencrypt/live/mountaintrips.com/cert.pem
privkey = /etc/letsencrypt/live/mountaintrips.com/privkey.pem
chain = /etc/letsencrypt/live/mountaintrips.com/chain.pem
fullchain = /etc/letsencrypt/live/mountaintrips.com/fullchain.pem

Options used in the renewal process

[renewalparams]
account = 2b8838f9899d92d51a3f2ba91aa3dd0c
manual_public_ip_logging_ok = True
renew_hook = /etc/letsencrypt/renewal-hooks/deploy/deploy.sh
manual_auth_hook = /etc/letsencrypt/renewal-hooks/pre/authenticate.sh
server = https://acme-v02.api.letsencrypt.org/directory
authenticator = manual
pref_challs = dns-01,

when I did the original request, I used “-d mountaintrips.com -d *.mountaintrips.com” so that both the root and any subdomain would be in the certificate. I notice that certbot does 2 separate validations ( for each host specified,) but only one cert ultimately get’s created. The authenticate script I use is pretty basic. It does a callout to the godaddy API to update the TXT record:

DNS_REC_NAME="_acme-challenge"
DNS_REC_DATA="CERTBOT_VALIDATION" DNS_REC_TTL="600" ############################################################ curl -s -X PUT "{GODADDY_URL}/v1/domains/{CERTBOT_DOMAIN}/records/TXT/{DNS_REC_NAME}" -H “Accept: application/json” -H “Content-Type: application/json” -H “Authorization: sso-key {GODADDY_API_KEY}:{GODADDY_API_SECRET}” -d “[{ “data”: “{DNS_REC_DATA}\", \"name\": \"{DNS_REC_NAME}”, “ttl”: ${DNS_REC_TTL} }]”
############################################################

Sleep for a bit to make sure the change has time to propagate

sleep 90

Anybody else running into this issue?

_az · February 14, 2020, 10:31pm

I think what’s happening is that the PUT API replaces all the TXT entries under _acme-challenge, so you’re wiping out your other challenge by accident.

So if the order of operations in Certbot is:

Call auth hook for first _acme-challenge challenge
Call auth hook for second _acme-challenge challenge
Validate first challenge
Validate second challenge
Call cleanup hook for first _acme-challenge challenge
Call cleanup hook for second _acme-challenge challenge

The API call in (2) essentially undoes the API call in (1), and then either (3) or (4) fails.

GoDaddy provides a PATCH version of that API call, which adds a record, rather than replacing all of the records matching the _acme-challenge label: https://developer.godaddy.com/doc/endpoint/domains#/v1/recordAdd

I think that should do what you want.

Also, check out acme.sh’s implementation of its GoDaddy DNS hook. It uses the PUT version, but it also includes all of the existing _acme-challenge TXT records in the PUT call, so earlier ones don’t get deleted.

Of course … you could just use acme.sh which supports GoDaddy natively without messing about with hooks: https://github.com/acmesh-official/acme.sh/wiki/dnsapi#4-use-godaddycom-domain-api-to-automatically-issue-cert

jeffsani · February 14, 2020, 11:55pm

Thanks for the reply. Hmm. But how did it work for the initial certificate request which was successful? Problem is that the location of the _acme-challenge TXT record for *.domain.com and domain.com are in the same place. If I removed the -d domain.com then it would work but anyone browsing to the site without a host would get an SSL error. Is the order of operations different for the first cert vs renewal? I can’t use acme as I have other custom operations in a deploy.sh script that moves the certs to a load balancer.

_az · February 15, 2020, 12:22am

I would guess that you fluked it by performing one of the authorizations on one run, and the other on another run.

Because authorizations on a single Let’s Encrypt account get remembered/reused for some period of time between different orders, tricky issues like this one can get easily masked.

That’s why since Certbot 0.40-ish, --dry-run deactivates all previous valid authorizations to prevent these kinds of false positive successes.

Yes. And that’s the difference between PUT vs PATCH.

PUT sets the value of _acme-challenge to the values you provide in the PUT call.

PATCH adds the value you provide to the existing set of _acme-challenge values. So you would end up with both values under _acme-challenge.

From ACME perspective, there’s no such thing as renewal. It’s all the same.

I don’t see why that would be a problem. acme.sh also allows you to have custom deploy hooks (as part of the acme.sh --install step).

Of course, it’s your choice to keep using Certbot if you want - I believe the PUT/PATCH thing solves this for you.

_az · February 15, 2020, 12:29am

To add some context, when Let’s Encrypt added wildcard support in the first place, heaps of custom hooks and clients had this precise issue, that they were incapable of creating multiple _acme-challenge TXT records against a single DNS label. So there was a lot of rewriting and bug fixing undertaken.

Unfortunately you hit the same pitfall .

jeffsani · February 15, 2020, 12:29am

Ok, so are you saying I need to have multiple _acme-challenge records? I still do not understand how that would work. The LE DNS call would get a random response from GoDaddy in that scenario right? In my zone * is not a subdomain. Also, if you look at the logs, Certbot issues the same request for *.mountaintrips.com as mountaintrips.com. Both authorizations are “mountaintrips.com” - see logs below.

content-length: 487
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
location: https://acme-v02.api.letsencrypt.org/acme/order/72222906/2332907212
boulder-requester: 72222906
date: Fri, 14 Feb 2020 20:59:43 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 000119Ol7O15AVbr9zd2GkrZZSEJCOmBP3NUhpGSQVNC9u4

{
“status”: “pending”,
“expires”: “2020-02-21T20:59:43.880751746Z”,
“identifiers”: [
{
“type”: “dns”,
“value”: “*.mountaintrips.com”
},
{
“type”: “dns”,
“value”: “mountaintrips.com”
}
],
“authorizations”: [
“https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997906”,
“https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997907”
],
“finalize”: “https://acme-v02.api.letsencrypt.org/acme/finalize/72222906/2332907212”
}
2020-02-14 12:59:43,929:DEBUG:acme.client:Storing nonce: 000119Ol7O15AVbr9zd2GkrZZSEJCOmBP3NUhpGSQVNC9u4
2020-02-14 12:59:43,930:DEBUG:acme.client:JWS payload:

2020-02-14 12:59:43,947:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997906:
{
“protected”: “eyJub25jZSI6ICIwMDAxMTlPbDdPMTVBVmJyOXpkMkdrclpaU0VKQ09tQlAzTlVocEdTUVZOQzl1NCIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjgxODk5NzkwNiIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC83MjIyMjkwNiIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “”,
“signature”: “UODSTg0yIAmetwT_vSGuTda-0CrEfOeqJXeAstUbuyMhOYRsuUI3Y108iqM538UMFowmE1PWTarqbEll2iODysMsdEvbJqC6Hzct7gAvPafEs3VaqCDoX57vfXOZwDEpPrduwXPUmMrC3m3hrhc17y5YPcl3T2h7fiqIYo8WX3MlrOjr94hPCkD-_39dkar4FgrFv7c4aBqBeyjxIFFqFHe12blLafixo8-XDxvK0THqWFDlG6384qi7jWdF0nTma6vQasH71REqwuDsrc56SCgqu0GSDFguX0Rda5Vtbdl7uXm25SLicPrTVWT6k94Wh66H7dmdc9BagLoHfQ2M3w”
}
2020-02-14 12:59:44,002:DEBUG:urllib3.connectionpool:“POST /acme/authz-v3/2818997906 HTTP/1.1” 200 389
2020-02-14 12:59:44,003:DEBUG:acme.client:Received response:
HTTP 200
content-length: 389
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
boulder-requester: 72222906
date: Fri, 14 Feb 2020 20:59:43 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0001IK4vVSJhXGnlA6_aNtx-bl7IByWPrA0lAFgbzoo1E5E

{
“identifier”: {
“type”: “dns”,
“value”: “mountaintrips.com”
},
“status”: “pending”,
“expires”: “2020-02-21T20:59:43Z”,
“challenges”: [
{
“type”: “dns-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997906/glTonA”,
“token”: “78sHNSOTnPuUkXd_KmqICLlF9YRg9ICjtQLFZIB4Vo8”
}
],
“wildcard”: true
}
2020-02-14 12:59:44,004:DEBUG:acme.client:Storing nonce: 0001IK4vVSJhXGnlA6_aNtx-bl7IByWPrA0lAFgbzoo1E5E
2020-02-14 12:59:44,005:DEBUG:acme.client:JWS payload:

2020-02-14 12:59:44,014:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997907:
{
“protected”: “eyJub25jZSI6ICIwMDAxSUs0dlZTSmhYR25sQTZfYU50eC1ibDdJQnlXUHJBMGxBRmdiem9vMUU1RSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjgxODk5NzkwNyIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC83MjIyMjkwNiIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “”,
“signature”: “EtR5c-sLi7fOxy8qFUngCXFH30c_KDfar-fpvIxpRxtqqtNNbMjOUPtBHs9ae74Ptowl4bRjwp08IvZcoYw63NJtSUzM5PE3Oi4fJjiAVS8svzpTEWHZq7Y2FFZkZ64jKpRWGk6eR_ZAzGGDVkTP9NYPCQ3-07LyG0IdXtBpMPaMdZVbpQvn32NBZs56ZKTLTMKknDHJ17za5RbgKqKpwCo-x-QYHJokWjOk7dpW5wErV8AYNmbEKJKSb5H5leKYB5ELdJnf03IXwY6QdPliFpQleT1VO4t1WvlFp5v3T4gP12EOx26AzP_r_qQTRWL3pHrIAABcSrpPmjk02uPlLg”
}
2020-02-14 12:59:44,061:DEBUG:urllib3.connectionpool:“POST /acme/authz-v3/2818997907 HTTP/1.1” 200 795
2020-02-14 12:59:44,062:DEBUG:acme.client:Received response:
HTTP 200
content-length: 795
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
boulder-requester: 72222906
date: Fri, 14 Feb 2020 20:59:44 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 00015urcOSzm1pBl1OXtOyGf3nw70_9W8tMr1ORx4jbTuxM

{
“identifier”: {
“type”: “dns”,
“value”: “mountaintrips.com”
},
“status”: “pending”,
“expires”: “2020-02-21T20:59:43Z”,
“challenges”: [
{
“type”: “http-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/lEsnfA”,
“token”: “Z7g1Q6R4NjJxyOssw_z020YISQBbbTWC4LnQj8i8Eo4”
},
{
“type”: “dns-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/mA2FfQ”,
“token”: “Z7g1Q6R4NjJxyOssw_z020YISQBbbTWC4LnQj8i8Eo4”
},
{
“type”: “tls-alpn-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/-A47jA”,
“token”: “Z7g1Q6R4NjJxyOssw_z020YISQBbbTWC4LnQj8i8Eo4”
}
]
}
2020-02-14 12:59:44,063:DEBUG:acme.client:Storing nonce: 00015urcOSzm1pBl1OXtOyGf3nw70_9W8tMr1ORx4jbTuxM
2020-02-14 12:59:44,064:INFO:certbot._internal.auth_handler:Performing the following challenges:
2020-02-14 12:59:44,064:INFO:certbot._internal.auth_handler:dns-01 challenge for mountaintrips.com
2020-02-14 12:59:44,064:INFO:certbot._internal.auth_handler:dns-01 challenge for mountaintrips.com
2020-02-14 12:59:44,068:INFO:certbot._internal.hooks:Running manual-auth-hook command: /etc/letsencrypt/renewal-hooks/pre/authenticate.sh
2020-02-14 13:01:14,611:INFO:certbot._internal.hooks:Running manual-auth-hook command: /etc/letsencrypt/renewal-hooks/pre/authenticate.sh
2020-02-14 13:02:45,325:INFO:certbot._internal.auth_handler:Waiting for verification…
2020-02-14 13:02:45,326:DEBUG:acme.client:JWS payload:
{
“type”: “dns-01”,
“resource”: “challenge”
}
2020-02-14 13:02:45,333:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997906/glTonA:
{
“protected”: “eyJub25jZSI6ICIwMDAxNXVyY09Tem0xcEJsMU9YdE95R2Yzbnc3MF85Vzh0TXIxT1J4NGpiVHV4TSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvMjgxODk5NzkwNi9nbFRvbkEiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNzIyMjI5MDYiLCAiYWxnIjogIlJTMjU2In0”,
“payload”: “ewogICJ0eXBlIjogImRucy0wMSIsIAogICJyZXNvdXJjZSI6ICJjaGFsbGVuZ2UiCn0”,
“signature”: “n07VTkWbVxDNQ14A0oIO5GZMDOzFZwSgU2hQ6W_33o3uHq3V0EIUa842MBuwvobpNwlXk4NdPs_vQogc4oGJr6rz3iQA8_lL9eLxJ07oyhylgoE49mdX7J4lbdEbF_g_JizYrcKUGxnBcP_X2tO5TFpRiuzlt1eOn3LKDo_wQ2Ea6HhZX9aJjifBFrPCSr4546xlksGgFSUtWJwdZvWR96-QRjtbSIJnRy243rL8XbJIU3uCq5Z4686psBqLFQih7GcrLSMDWvAfcE6Cob_JLqcIRgibjLFqdpjnQNTvdlG0ULjUFAk4Nrl5cFujpaJuxxxqKBwjy_6KdpaLxtQIhA”
}
2020-02-14 13:02:45,386:DEBUG:urllib3.connectionpool:“POST /acme/chall-v3/2818997906/glTonA HTTP/1.1” 200 184
2020-02-14 13:02:45,387:DEBUG:acme.client:Received response:
HTTP 200
content-length: 184
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”, https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997906;rel=“up”
location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997906/glTonA
boulder-requester: 72222906
date: Fri, 14 Feb 2020 21:02:45 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002J1ITVU7Rrinr_3NBHt17WQAVGF0vEnkitxzvX1_WR2Q

{
“type”: “dns-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997906/glTonA”,
“token”: “78sHNSOTnPuUkXd_KmqICLlF9YRg9ICjtQLFZIB4Vo8”
}
2020-02-14 13:02:45,388:DEBUG:acme.client:Storing nonce: 0002J1ITVU7Rrinr_3NBHt17WQAVGF0vEnkitxzvX1_WR2Q
2020-02-14 13:02:45,389:DEBUG:acme.client:JWS payload:
{
“type”: “dns-01”,
“resource”: “challenge”
}
2020-02-14 13:02:45,395:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/mA2FfQ:
{
“protected”: “eyJub25jZSI6ICIwMDAySjFJVFZVN1JyaW5yXzNOQkh0MTdXUUFWR0YwdkVua2l0eHp2WDFfV1IyUSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvY2hhbGwtdjMvMjgxODk5NzkwNy9tQTJGZlEiLCAia2lkIjogImh0dHBzOi8vYWNtZS12MDIuYXBpLmxldHNlbmNyeXB0Lm9yZy9hY21lL2FjY3QvNzIyMjI5MDYiLCAiYWxnIjogIlJTMjU2In0”,
“payload”: “ewogICJ0eXBlIjogImRucy0wMSIsIAogICJyZXNvdXJjZSI6ICJjaGFsbGVuZ2UiCn0”,
“signature”: “ggVd7T8gfVK2SrTcucEb7b6jFXQmLR19BlwaRZfX7rhZbnExFDumrtLBFKs_irZujCbJ2LOBB9TI7-RF_wyePriOrVnqNjwVVYR_eFxIjhz6AjAjtU2vNRAbHQyXWBOw3Wn6uIQ8eA4oDAW1P1FA1fAgLbAMhpW8Ck6rupVljRbPccvw34gdJD8K8nWP03nACZVaLzf9ls1PWdgGuQckgGvmJwxOTZzk5aWThv-Gl-k2exJ5ph2K1IhR7RhzMuY4RWh2UdNciIDPhkzWW5dGf3RYCda31u80hhra_MtsXWU9uRyHSc_2tp5PXDNICt-5wgeCtNff1BY3pxqh5D-CLw”
}
2020-02-14 13:02:45,445:DEBUG:urllib3.connectionpool:“POST /acme/chall-v3/2818997907/mA2FfQ HTTP/1.1” 200 184
2020-02-14 13:02:45,446:DEBUG:acme.client:Received response:
HTTP 200
content-length: 184
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”, https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997907;rel=“up”
location: https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/mA2FfQ
boulder-requester: 72222906
date: Fri, 14 Feb 2020 21:02:45 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002hqCUciUymD1mjtO9hgqwpmi4xRN4vTNkpFcrsN9Co3g

{
“type”: “dns-01”,
“status”: “pending”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/mA2FfQ”,
“token”: “Z7g1Q6R4NjJxyOssw_z020YISQBbbTWC4LnQj8i8Eo4”
}
2020-02-14 13:02:45,447:DEBUG:acme.client:Storing nonce: 0002hqCUciUymD1mjtO9hgqwpmi4xRN4vTNkpFcrsN9Co3g
2020-02-14 13:02:46,449:DEBUG:acme.client:JWS payload:

2020-02-14 13:02:46,454:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997906:
{
“protected”: “eyJub25jZSI6ICIwMDAyaHFDVWNpVXltRDFtanRPOWhncXdwbWk0eFJONHZUTmtwRmNyc045Q28zZyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjgxODk5NzkwNiIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC83MjIyMjkwNiIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “”,
“signature”: “WTQ-SiRLzFcNCPq5waI_SP9Krc4oYrgDI6jWTGkTdCsgkV0KQXuqj8daIakIvOpCK0QK0HR6AbRjomDqqn93dIGnPyLO_TY28xwO9MRfZlNJ6IbwQSd2y8LK_gcThJqtpZm59_4gqCnYZSB1CIy_FV-bEbMfpmr14J34TUOBV3ZQ2jBd6dq3guQ0ALQPHCwhf0cJrOy14RCcDh0ycYEZC_CJa8XOiyfNwm-v-LFXhji3H6vKlEq2yLGqfMAp5XqEo0kPUbMS6AVae9gvWQLhyPeEA4-kXJEHskI0o4LecCKZKu-PPh3yajM_xN4_TduaBcU5mOzqcZIToOrb7Z2tHA”
}
2020-02-14 13:02:46,498:DEBUG:urllib3.connectionpool:“POST /acme/authz-v3/2818997906 HTTP/1.1” 200 629
2020-02-14 13:02:46,499:DEBUG:acme.client:Received response:
HTTP 200
content-length: 629
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
boulder-requester: 72222906
date: Fri, 14 Feb 2020 21:02:46 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0002pTLCpq-EJ9YOr4f-JC2mzN-YelAHgdNi-ZQ_ifxBtRw

{
“identifier”: {
“type”: “dns”,
“value”: “mountaintrips.com”
},
“status”: “invalid”,
“expires”: “2020-02-21T20:59:43Z”,
“challenges”: [
{
“type”: “dns-01”,
“status”: “invalid”,
“error”: {
“type”: “urn:ietf:params:acme:error:unauthorized”,
“detail”: “Incorrect TXT record “h_O_dPHpTQIkmGgooCMDPsQ2Z16W5auV09GCRohIB4Q” found at _acme-challenge.mountaintrips.com”,
“status”: 403
},
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997906/glTonA”,
“token”: “78sHNSOTnPuUkXd_KmqICLlF9YRg9ICjtQLFZIB4Vo8”
}
],
“wildcard”: true
}
2020-02-14 13:02:46,499:DEBUG:acme.client:Storing nonce: 0002pTLCpq-EJ9YOr4f-JC2mzN-YelAHgdNi-ZQ_ifxBtRw
2020-02-14 13:02:46,501:DEBUG:acme.client:JWS payload:

2020-02-14 13:02:46,506:DEBUG:acme.client:Sending POST request to https://acme-v02.api.letsencrypt.org/acme/authz-v3/2818997907:
{
“protected”: “eyJub25jZSI6ICIwMDAycFRMQ3BxLUVKOVlPcjRmLUpDMm16Ti1ZZWxBSGdkTmktWlFfaWZ4QnRSdyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYXV0aHotdjMvMjgxODk5NzkwNyIsICJraWQiOiAiaHR0cHM6Ly9hY21lLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC83MjIyMjkwNiIsICJhbGciOiAiUlMyNTYifQ”,
“payload”: “”,
“signature”: “q51-eWyKsp3iMvGd_DW63YHKpYzAMQnm4-TCAYA0qugsPosPbMqBv5fdYZo5y9PgMgirOPmUSu1o_0Ocg_n9aQTYmp51nV7OYU-txX6x2rL5r_P4mai15OOkJgJb-P8zvyF1inr8Huld533l07zFH8O9A5NI9lyz4oWoYMjGrXmwE2wbQ-3vY2usvjN7ZPvQbl6U7ZPOmqV5tameMl4E4tYNVjVUuvUSnrfqTlf5wpR825L6V25KxBFUF0tG96dKiBNz7wOx3UCsF1Wzfp2wOdmd8N8o9VR-ThOeTCt57AFLU0GTnLsWruDgIu6fvVEAGogTEhqyOdiZcCEHB9X5ZA”
}
2020-02-14 13:02:46,550:DEBUG:urllib3.connectionpool:“POST /acme/authz-v3/2818997907 HTTP/1.1” 200 464
2020-02-14 13:02:46,551:DEBUG:acme.client:Received response:
HTTP 200
content-length: 464
cache-control: public, max-age=0, no-cache
strict-transport-security: max-age=604800
server: nginx
connection: keep-alive
link: https://acme-v02.api.letsencrypt.org/directory;rel=“index”
boulder-requester: 72222906
date: Fri, 14 Feb 2020 21:02:46 GMT
x-frame-options: DENY
content-type: application/json
replay-nonce: 0001CFma0RwhK-JcgEDwGqbG_upuX4lEquPX9OuLmdX9pDk

{
“identifier”: {
“type”: “dns”,
“value”: “mountaintrips.com”
},
“status”: “valid”,
“expires”: “2020-03-15T21:02:45Z”,
“challenges”: [
{
“type”: “dns-01”,
“status”: “valid”,
“url”: “https://acme-v02.api.letsencrypt.org/acme/chall-v3/2818997907/mA2FfQ”,
“token”: “Z7g1Q6R4NjJxyOssw_z020YISQBbbTWC4LnQj8i8Eo4”,
“validationRecord”: [
{
“hostname”: “mountaintrips.com”
}
]
}
]
}
2020-02-14 13:02:46,551:DEBUG:acme.client:Storing nonce: 0001CFma0RwhK-JcgEDwGqbG_upuX4lEquPX9OuLmdX9pDk
2020-02-14 13:02:46,552:WARNING:certbot._internal.auth_handler:Challenge failed for domain mountaintrips.com
2020-02-14 13:02:46,552:INFO:certbot._internal.auth_handler:dns-01 challenge for mountaintrips.com
2020-02-14 13:02:46,553:DEBUG:certbot._internal.reporter:Reporting to user: The following errors were reported by the server:

Domain: mountaintrips.com
Type: unauthorized
Detail: Incorrect TXT record “h_O_dPHpTQIkmGgooCMDPsQ2Z16W5auV09GCRohIB4Q” found at _acme-challenge.mountaintrips.com

_az · February 15, 2020, 12:31am

I think what needs to be understood in this context is that DNS is capable of serving multiple TXT records under a single label.

For example, one of my test domains:

$ dig +noall +answer _acme-challenge.foo.monkas.xyz TXT
_acme-challenge.foo.monkas.xyz. 594 IN  TXT     "23DmSWvjBnO9sDgvJQ1rVsuGNMOUC2OdaRP8IlhykbY"
_acme-challenge.foo.monkas.xyz. 594 IN  TXT     "rum78MgEmL4lkkem8N3Dz9eaXjYLPikcA_bywhLKhzk"

GoDaddy allows you to create this record set by either specifying them all at once in the PUT call, or using the PATCH call to add one to the existing record set.

rmbolger · February 15, 2020, 12:34am

If it helps with understanding, it’s exactly the same as having multiple A records for a given label when using DNS round robin.

jeffsani · February 15, 2020, 12:34am

Yes I know that. But when you have multiple records (A, TXT, etc…), when you query them, you will get a random response. So unless LE is requesting all TXT records and parsing through those, the authentication will potentially fail.

_az · February 15, 2020, 1:08am

Yes. Let’s Encrypt asks the authoritative DNS servers for all the TXT records under the label, and then checks if any one of them matches the one it’s looking for.

Here is the literal text from RFC8555:

To validate a DNS challenge, the server performs the following steps:

Compute the SHA-256 digest [FIPS180-4] of the stored key
authorization

Query for TXT records for the validation domain name

Verify that the contents of one of the TXT records match the
digest value

jeffsani · February 15, 2020, 12:35am

Yes - this is my problem…

jeffsani · February 15, 2020, 12:36am

Aha! Ok, let me try this. Thank you.

jeffsani · February 15, 2020, 12:45am

Perfect. I changed the PUT to PATCH but it throws an error:

{
“code”: “NOT_FOUND”,
“message”: “Not Found : The requested resource was not found”
}

Must be something else required in the payload. I am sending:

[{ “data”: “4i67-toZ_wK9JCOlcxibuSm4d8EZGUBSvljrVAysXMM”, “name”: “_acme-challenge”, “ttl”: 600 }]

_az · February 15, 2020, 1:08am

Mmmm, so, your original script used this API endpoint:

PUT /v1/domains/{domain}/records/{type}/{name}

The PATCH endpoint has this prototype however:

PATCH /v1/domains/{domain}/records

So I believe you need to chop off the /{type}/{name}, and then change your request body to something like:

{
  "name": "${DNS_REC_NAME}",
  "type": "TXT",
  "data": "${DNS_REC_DATA}",
  "ttl": "${DNS_REC_TTL}"
}

(Edit: might need to be additionally wrapped as an array [] as in your first example? Dunno, unclear).

Highly recommend lowering TTL as low as you can, “1” if possible. It makes a difference if you are running Certbot multiple times within 1 minute - Let’s Encrypt clamps TTLs to 60 seconds.

I don’t have GoDaddy so I can’t test this for you, but check the docs.

jeffsani · February 15, 2020, 12:54am

In the GoDaddy API, the lowest “custom” TTL is 600. Does the API allow you to go lower? Thanks for that catch. I am trying that now. Appreciate the assist.

_az · February 15, 2020, 12:55am

No idea. It’s no huge deal if you need to set 600. Just be wary that Let’s Encrypt does have a 1 minute DNS cache on its resolver … so wait 60 seconds between attempts.

jeffsani · February 15, 2020, 1:07am

Looks like no.

{
“code”: “INVALID_BODY”,
“fields”: [
{
“code”: “UNKNOWN_RESTRICTION”,
“message”: “must have a minimum value of 600”,
“path”: “records[0].ttl”
}
],
“message”: "Request body doesn’t fulfill schema, see details in fields"
}

I have those sleep timers in the scripts for this reason and I learned that one more or less by trial and error. Mine are set at 90 sec and that seems to be enough time for both ends.