Hi, I love to use the wildcard certificates with the new acme protocol and letsencrypt service. But I have trouble with it.
When I try to request a certificate for a domain and wildcards for this domain with -d domain.tld -d *.domain.tld there are two different challenges issued for the same domain (domain.tld) by the service. So before testing there are two TXT records with different challenges in the DNS for this domain. That should not be a problem but it seems to be. It looks as if the service could not pick the correct challenge out of the two available all the time. Once it seem to pick the correct one and the process succeeds, other times it picks the wrong one and fails.
I use certbot in manual mode with scripts for auth an cleanup like outlined in the docs. Those scripts use virtualmin to edit the DNS and assure that the challenge is visible in all primary and secondary name servers before continuing the process. But still I get once and awhile an “Incorrect TXT record” error showing one of the challenges. This leads me to the former assumption. Here a (cleaned) trace of a failed request:
Saving debug log to /var/log/certbot/letsencrypt.log
Plugins selected: Authenticator manual, Installer None
Renewing an existing certificate
Performing the following challenges:
dns-01 challenge for domain.tld
dns-01 challenge for domain.tld
Output from dns-auth-hook.sh:
Setting challenge po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI for domain domain.tld ...
Updating server domain.tld ..
...
Challenge po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI found at d.ns14.net ...
Challenge po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI found at c.ns14.net ...
Challenge po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI found at ns1.heinzsoftware.net ...
Challenge po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI found at b.ns14.net ...
Output from dns-auth-hook.sh:
Setting challenge zq8hPcK1u04N7CI0uCIrvPO-qSSIkzHLpNo-ylZZ0_s for domain domain.tld ...
Updating server domain.tld ..
...
Challenge zq8hPcK1u04N7CI0uCIrvPO-qSSIkzHLpNo-ylZZ0_s found at b.ns14.net ...
Challenge zq8hPcK1u04N7CI0uCIrvPO-qSSIkzHLpNo-ylZZ0_s found at c.ns14.net ...
Challenge zq8hPcK1u04N7CI0uCIrvPO-qSSIkzHLpNo-ylZZ0_s found at ns1.heinzsoftware.net ...
Challenge zq8hPcK1u04N7CI0uCIrvPO-qSSIkzHLpNo-ylZZ0_s found at d.ns14.net ...
Waiting for verification...
Cleaning up challenges
Output from dns-cleanup-hook.sh:
Updating server domain.tld ..
...
Output from dns-cleanup-hook.sh:
Updating server domain.tld ..
...
Running post-hook command: /etc/certbot/restart-services.sh
Output from restart-services.sh:
Restarting apache ...
...
Failed authorization procedure. domain.tld (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect TXT record "po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI" found at _acme-challenge.domain.tld
IMPORTANT NOTES:
- The following errors were reported by the server:
Domain: domain.tld
Type: unauthorized
Detail: Incorrect TXT record
"po0hy6nSYcTB1P8hNpqkMj9AM3ysQGtnBLTfhbEG9GI" found at
_acme-challenge.domain.tld
To fix these errors, please make sure that your domain name was
entered correctly and the DNS A/AAAA record(s) for that domain
contain(s) the right IP address.
I ran the same command a few minutes later and the process succeeded.
I would check that you're not accidentally replacing the previous value. Both of the values should be present in the zone at the end of the second invocation of the hook. Could you add a dig +trace _acme-challenge.XXX at the end of the script to show what ALL of the records are for that DNS name after each hook invocation?
I’m sure they are present. I already checked it with dig during the process.
Here the trace. This one is of a successful request:
dig +trace _acme-challenge.magazin-kettenblatt.de TXT
; <<>> DiG 9.11.2 <<>> +trace _acme-challenge.magazin-kettenblatt.de TXT
;; global options: +cmd
. 412970 IN NS i.root-servers.net.
. 412970 IN NS c.root-servers.net.
. 412970 IN NS a.root-servers.net.
. 412970 IN NS h.root-servers.net.
. 412970 IN NS f.root-servers.net.
. 412970 IN NS k.root-servers.net.
. 412970 IN NS j.root-servers.net.
. 412970 IN NS l.root-servers.net.
. 412970 IN NS g.root-servers.net.
. 412970 IN NS b.root-servers.net.
. 412970 IN NS m.root-servers.net.
. 412970 IN NS e.root-servers.net.
. 412970 IN NS d.root-servers.net.
. 517289 IN RRSIG NS 8 0 518400 20180501050000 20180418040000 39570 . NrEq63W3bsuDnSYGM0f8gmn2VVLtjgPlag9QddzOPrHfCjuAnbCh4az/ y5iXc4VcY56AhDcWHvN5up8fejeOb7hXCig6R4MTl0+XIUJKuFlod6UV jlsMW5tcQZzsQ76wDFlAavtnaP+QD02QRgxsCOqsx29BHQElVPlu0wXe SAOGadzzLfnCbV+lqZehFSy3xNLqP+kvt/lCzTrpMtwmT8vjVMJ69DH5 wfBrbLu1X8Jk3lIu8vYR8n8ztGVo2dZPiPRlQoAg0BpBmVYRsVHAcmNb vbmybzoTCVGXkAh/K25xtcqHUYng7X7a52FmWyPj5/okOKwBFAcD6Euz 2N5iRA==
;; Received 1125 bytes from 127.0.0.1#53(127.0.0.1) in 0 ms
de. 172800 IN NS a.nic.de.
de. 172800 IN NS f.nic.de.
de. 172800 IN NS l.de.net.
de. 172800 IN NS n.de.net.
de. 172800 IN NS s.de.net.
de. 172800 IN NS z.nic.de.
de. 86400 IN DS 39227 8 2 AAB73083B9EF70E4A5E94769A418AC12E887FC3C0875EF206C3451DC 40B6C4FA
de. 86400 IN RRSIG DS 8 1 86400 20180501050000 20180418040000 39570 . uJQDylY2iaqdTNm7KBqYWPs+8/y9u/zCGJLdrF6KC8sDyyhIjuhjrRms UFoUaYQStv/GMySmJf9Kn9kO7tLgHYj6JOyu1ucLiwE2DXPz93sfK4Ba mMDsTISg5yuVfTC8iBiWGt82gyPCkYWRTmyUyZNITkKBlybEWOXvBEd2 7YnnCj5Lgv9FmnYGe752Pk1jpHrcA/MuOR+EHmirMZBRdKIfsGrkaeqV 17/oclFUJEC5GTfn9yYIkeBPBw8EHZCTQBEcVrwLTtg06OP1xfRl8MYH 7e1bLFxC18JcNDDRhy7CH1LEB4bmx9CyjXNcp0SUnrmFf4qJB5xMKoQ2 GV9P2A==
;; Received 744 bytes from 199.7.83.42#53(l.root-servers.net) in 32 ms
magazin-kettenblatt.de. 86400 IN NS b.ns14.net.
magazin-kettenblatt.de. 86400 IN NS c.ns14.net.
magazin-kettenblatt.de. 86400 IN NS d.ns14.net.
magazin-kettenblatt.de. 86400 IN NS ns1.heinzsoftware.net.
h319dm5gc3edek691vqbhehot7vggj2b.de. 7200 IN NSEC3 1 1 15 BA5EBA11 H31FCTNLR9QCVI0VN44S89NBKM33P0MQ NS SOA RRSIG DNSKEY NSEC3PARAM
h319dm5gc3edek691vqbhehot7vggj2b.de. 7200 IN RRSIG NSEC3 8 2 7200 20180425100055 20180418100055 27229 de. u7MlhviLVnX685ofBFgU7mmJisA7ayrph8WvH5waal72ILDCoxJVgOO7 iM82wAtTZ/DXiMnvt6JY+2WLVBFmYU8adWHDiW+1IWGgIolvOE6AE6A2 bJVugQaqP0a9KXoYIobDQCyL7M6wa4RJ+niny+XlZ1PxL07fqBTYnxxE Sk8=
4s7i5c2j6k3s2mi2d2bip7o3gd85o7qn.de. 7200 IN NSEC3 1 1 15 BA5EBA11 4S7QGIIJCDI5JLAO8O308AQ3J5CDJ0VU NS DS RRSIG
4s7i5c2j6k3s2mi2d2bip7o3gd85o7qn.de. 7200 IN RRSIG NSEC3 8 2 7200 20180425100055 20180418100055 27229 de. gv2RqTZxtqHWBtzDsrf/IrsunFDZThBVS8F3ZLTynsqlKC+nu9RzDLpf Fn3denrZ0w/QZiaIovuP6wlWpVHU8SSh+gUXm7PxqRVp4XLbaMBWrld1 EHe8RIg1jTFShkety1EUiGuUjF461AsMz+WnNoG0iVjciv7eEYGEIMki 5rU=
;; Received 646 bytes from 2001:67c:1011:1::53#53(n.de.net) in 4 ms
_acme-challenge.magazin-kettenblatt.de. 10 IN TXT "MZGeRnPiSSRKFzx959RGaYU-ENxRAhmgOt-IiV95I-I"
_acme-challenge.magazin-kettenblatt.de. 10 IN TXT "NpxvpOTjs6rl29BYouXXxo94lOQnaY-9iUpuUA1t0CI"
;; Received 179 bytes from 217.160.113.32#53(b.ns14.net) in 7 ms
If I knew the name server address letsencrypt is asking for the challenges, I could restrict my waiting for the challenges to appear on this server. Is this known to the public?
By the way: why is it necessary to ask for a challenge twice for the same domain in one request?
update: It seems to be necessary to wait for about 30 seconds after the last challenge is set to make the change propagation thru the DNS hierarchy possible. This at least did the trick for me. The best thing would be if I could ask the same name server as the letsencrypt service does for the appearance of the challenges. This would be independent of timing issues.
One is for the base domain and one is for the wildcard (which isn't required by industry rules, but is required by Let's Encrypt's current architecture). This is being discussed at
Let's Encrypt run their own recursive DNS servers, with little to no caching, which will query one of your authoritative nameservers randomly-ish. (Or more than one, if some of them don't work.) You can't do much more direct investigation than querying your own authoritative nameservers yourself.
You can use https://unboundtest.com/ or your own Unbound installation with a similar configuration to more or less mimic the precise resolution behavior.
Exactly this was my hope but it does not seem to be the case. When I wait to see the challenges appear on all of the name servers declared in the domain, the letsencrypt service should be able to see them too if your claim is correct. But waiting for 30 seconds remedies the problem, This shows that there must be more DNS propagation until letsencrypt sees it.
In fact a delay seems to be programmed in all of the available DNS plugins for other services. So it is not only my problem. It would be nice if there was a hint for that in the docs.
The delay exists there in lieu of actually polling each nameserver to confirm that the record is being served by each authoritative nameserver. However, this can be problematic on anycast (or load-balanced, if anybody even does that) nameservers, because one geographical perspective may not be accurate for the perspective of the VA, so a fixed delay may in some cases be more reliable. The reason sleeps are common across providers is that they all share the general characteristics that DNS updates are eventually consistent.
Since your script is ostensibly checking each of your nameservers, the only explanation I see is that your script is getting false positives on the readiness of each of the authoritative nameservers. Maybe there is more going on behind the scenes on each nameserver IP than meets the eye?
You can do a packet capture on each of your nameservers and you will see that the validation authority is directly connecting to your nameservers.
I don't think it gets false positives as it is looking for the exact challenge pattern requested for a domain. This should be unique in this context, right? There is definitely something weird going on.
Ah, OK! I suspect our secondaries to be the cause of the trouble. Maybe I see changes to them earlier than the rest of the world. As I have only AXFR access to them I'm not able to investigate that further. May be it's time to change the secondary provider. But still thanks for the information.