Same for us, it looks like that the TXT record is so large that Unbound isn't able to resolve and fails at the first verification when requesting a cert with 25 domains:
[adding txt record to route53 one by one]
...
...
...
│ [Wed Dec 27 12:24:34 UTC 2023] All success, let's return
│ [Wed Dec 27 12:24:34 UTC 2023] Verifying:
│ *.abh.cloudengine.mercedes-benz.com
│ [Wed Dec 27 12:24:34 UTC 2023] Pending, The CA is processing your order,
│ please just wait. (1/30)
│ [Wed Dec 27 12:24:37 UTC 2023] Invalid status,
│ *.abh.cloudengine.mercedes-benz.com:Verify error detail:No TXT record found
│ at _acme-challenge.abh.cloudengine.mercedes-benz.com
│ [Wed Dec 27 12:24:37 UTC 2023] Removing DNS records.
│ [Wed Dec 27 12:24:37 UTC 2023] Removing txt:
│ lhnUPlA9u22VhTR2t22WL1B6XxVJM3O745H-mhUKboE for domain:
│ _acme-challenge.abh.cloudengine.mercedes-benz.com
...
...
...
failure
Which indicates that unbound is not able to get the TXT record. But it did in the past since we successfully created this certificate in one command, with 25 domains, 3 months ago.
Now, to troubleshoot this issue we created a large fake TXT record: _acme-challenge.abhtest.cloudengine.mercedes-benz.com
Unbound 1.16 successfully return the TXT records
Unbound 1.18 and 1.19 doesn't.
We then ran unbound 1.19 locally via docker using unboundtest Dockerfile and the same unbound.conf:
Unbound 1.19 running locally also fails to return the TXT record. We can see the TXT record in the debug log, but not in the answer.
Query results for TXT _acme-challenge.abhtest.cloudengine.mercedes-benz.com
Response:
;; opcode: QUERY, status: NOERROR, id: 46640
;; flags: qr tc rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version 0; flags: do; udp: 512
;; QUESTION SECTION:
;_acme-challenge.abhtest.cloudengine.mercedes-benz.com. IN TXT
----- Unbound logs -----
...
...
Dec 27 13:39:13 unbound[13:0] debug: process_response: new external response event
Dec 27 13:39:13 unbound[13:0] info: scrub for cloudengine.mercedes-benz.com. NS IN
Dec 27 13:39:13 unbound[13:0] info: response for _acme-challenge.abhtest.cloudengine.mercedes-benz.com. TXT IN
Dec 27 13:39:13 unbound[13:0] info: reply from <cloudengine.mercedes-benz.com.> 205.251.196.169#53
Dec 27 13:39:13 unbound[13:0] info: incoming scrubbed packet: ;; ->>HEADER<<- opcode: QUERY, rcode: NOERROR, id: 0
;; flags: qr aa ; QUERY: 1, ANSWER: 21, AUTHORITY: 4, ADDITIONAL: 0
;; QUESTION SECTION:
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. IN TXT
;; ANSWER SECTION:
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "EHQYOqAeQgjOEpYljeyOvHTTKFc2XvLRxz3L3t0GpJg"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "Mo4Jj06GlYbnTpSoGrW9hfUxQmwaACajaAjVQNoDBq0"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "NxTMWNG8vB3_sqBFl-hYLAqNfgYF-CG3EsOXYtNYiTY"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "PbJXwoJr8Xmneky7VBgOf-WfVaQFuu50AaJB-R-u4PY"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "PeTIYmnDU2cyq-l_VljNIYO7tRjdWI5yzpexoDP3Z1U"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "Q1QrrSEEInPrag2g7y4_EH"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "Q1QrrSEEInPrag2g7y4_EH-GTcUmL8XlcWv6SqDdCsE"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "QSadJSxPUioThP2XHNH1aXvJKEjyPbkttdINObZZGfA"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "YTJxX-cdB5bXJQ2oR03rhN1Au1BZFZS955DrnhDbOBI"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "ex76rM--NrtcwTlx1rqpxtsk_0fv4oSEVcfxjiqm8VQ"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "jakxcJHi_sAFnE64fjyVh1fhPk3SLOLfIrNssr5YX6Q"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "qAacYHuljj2mkA82MkYEUbACRVcWkLYNkUU8lwrLHAY"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "rbV3iZeujOvzfsl7Vpj9vM0L0CMoPaPLzXHb0yM3DB4"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "yfu8J7zX1TqTtBau9Mdm7aBui3Sba8BlG5XYCjMOWkw"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "3O6-HP0qVo03wno7w7dLPuSCDfZKBXJM1nNbQYY-Y1Q"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "6dWY3tFbyPWIgkL46ok1TE63UFqGnfQzaRdd7a1YPUI"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "A4EGy0uFH_79sdkVkMJhT9_U4Ltkf0-6Uoqup-SLI70"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "AYuYR7_0pTjkDaHa5-vjhsMDWDvGp7ZgJP2HqzWzXSA"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "C6CtkcfTd21q5m2FB-QH0vbArg_g0QNQaeIUHIc91vI"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "CbwkUnUZz6-plPNxhpkCSXbCZDt87gMf-JED4pIUv0E"
_acme-challenge.abhtest.cloudengine.mercedes-benz.com. 0 IN TXT "DeJjJ9cImQwPPFUMsI39UBFVtj2Fsn6L9uGx4m8qV5A"
;; AUTHORITY SECTION:
cloudengine.mercedes-benz.com. 0 IN NS ns-1814.awsdns-34.co.uk.
cloudengine.mercedes-benz.com. 0 IN NS ns-603.awsdns-11.net.
cloudengine.mercedes-benz.com. 0 IN NS ns-63.awsdns-07.com.
cloudengine.mercedes-benz.com. 0 IN NS ns-1193.awsdns-21.org.
;; ADDITIONAL SECTION:
;; MSG SIZE rcvd: 1362
Dec 27 13:39:13 unbound[13:0] debug: iter_handle processing q with state QUERY RESPONSE STATE
Dec 27 13:39:13 unbound[13:0] info: query response was ANSWER
Dec 27 13:39:13 unbound[13:0] debug: TTL 0: dropped msg from cache
Dec 27 13:39:13 unbound[13:0] debug: iter_handle processing q with state FINISHED RESPONSE STATE
Dec 27 13:39:13 unbound[13:0] info: finishing processing for _acme-challenge.abhtest.cloudengine.mercedes-benz.com. TXT IN
Dec 27 13:39:13 unbound[13:0] debug: mesh_run: iterator module exit state is module_finished
Dec 27 13:39:13 unbound[13:0] debug: validator[module 0] operate: extstate:module_wait_module event:module_event_moddone
Dec 27 13:39:13 unbound[13:0] info: validator operate: query _acme-challenge.abhtest.cloudengine.mercedes-benz.com. TXT IN
Dec 27 13:39:13 unbound[13:0] debug: validator: nextmodule returned
Dec 27 13:39:13 unbound[13:0] debug: val handle processing q with state VAL_INIT_STATE
Dec 27 13:39:13 unbound[13:0] debug: validator classification positive
Dec 27 13:39:13 unbound[13:0] info: no signer, using _acme-challenge.abhtest.cloudengine.mercedes-benz.com. TYPE0 CLASS0
Dec 27 13:39:13 unbound[13:0] debug: val handle processing q with state VAL_FINISHED_STATE
Dec 27 13:39:13 unbound[13:0] debug: TTL 0: dropped msg from cache
...
...
We then modified the unboundtest configuration unbound.conf to add the following line
max-udp-size:4096
Unbound 1.19 running locally successfully returned the TXT record.
Indeed, Unbound 1.16 default max-udp-size was 4096 and it was changed in this commit to 1232 which is used by 1.18 and 1.19
19 January 2023: Wouter
- Set max-udp-size default to 1232. This is the same default value as
the default value for edns-buffer-size. It restricts client edns
buffer size choices, and makes unbound behave similar to other DNS
resolvers. The new choice, down from 4096 means it is harder to get
large responses from Unbound. Thanks to Xiang Li, from NISL Lab,
Tsinghua University.
Is it possible that Let's Encrypt unbound configuration is misbehaving when:
max-udp-size: 1232
edns-buffer-size: 512
It is working with 1.16:
max-udp-size: 4096
edns-buffer-size: 512
Now that max udp-size is 1232 does it mean the response is discarded and the TCP fallback is not being used?
Locally I have no issue using dig to query the TXT records without edns:
$ dig @ns-63.awsdns-07.com -t TXT _acme-challenge.abhtest.cloudengine.mercedes-benz.com +noedns
When setting buffsize to 512 and ignore truncation, it fails:
$ dig @ns-603.awsdns-11.net. -t TXT _acme-challenge.abhtest.cloudengine.mercedes-benz.com +bufsize=512 +edns +ignore
No problem with 4096:
$ dig @ns-603.awsdns-11.net. -t TXT _acme-challenge.abhtest.cloudengine.mercedes-benz.com +bufsize=4096 +edns +ignore
I have been thinking about this for the last 4 days and my sanity is slowly going away. At this stage I am not sure if there is an issue between Let's Encrypt unbound and route53, or is there an issue with some ns1-4.corpinter.net NS.
With more and more domain adding constant TXT records to verify their ownership of a domain name, it is not uncommon to have bigger and bigger TXT records. Just look at google.com or mercedes-benz.com TXT records to see all the something-verification=123456789.