Must the DNS TXT record, which was provisioned for fulfilling the DNS challenge, be kept alive beyond successful validation - for the entire lifetime (90 days) of one certificate? I mean one certificate, without renewal. For renewal it's clear: New order -> new challenge.
I found the following information and conclude that the DNS record can be removed right after validation. Can anyone confirm?
Successfully completed challenges get cached for 30 days. (FAQ - Let's Encrypt)
Once you successfully complete the challenges for a domain, the resulting authorization is cached for your account to use again later. Cached authorizations last for 30 days from the time of validation. If the certificate you requested has all of the necessary authorizations cached then validation will not happen again until the relevant cached authorizations expire.
The client SHOULD de-provision the resource record(s) provisioned for this challenge once the challenge is complete, i.e., once the "status" field of the challenge has the value "valid" or "invalid".
That's pretty clear. What's actually confusing me, is the following from LE docs, especially the second sentence (Challenge Types - Let's Encrypt):
You can have multiple TXT records in place for the same name. For instance, this might happen if you are validating a challenge for a wildcard and a non-wildcard certificate at the same time. However, you should make sure to clean up old TXT records, because if the response size gets too big Let’s Encrypt will start rejecting it.
Why can the size get to big at all, if one should deprovision TXT records after successful validation? (Two records for wildcard and non-wildcard won't make it big...) Is there any need to keep old records beyond the process of fulfilling the challenge? Will LE re-check the challenge after some time for the same certificate (not a new order)?
Perhaps now they do. The last time I checked, not many clients handled this automatically. It's not even straightforward with Certbot, as that must often be handled through a separate manual-cleanup-hook that tutorials for DNS management don't typically address.
The certbot supported DNS plugins handle creating and deleting the TXT Records.
There is a manual method for the DNS challenge as well. In that case if you want automated renewals you supply a pre authorization hook and the clean up hook. If you don't want automated renewals then you are responsible for adding the TXT record manually and hopefully deleting it. We of course discourage manual certs without automated renewals But it is sometimes handy for one off certs.
The certbot developed/bundled ones now do this. I don't believe they always did, but haven't combed through the changelog enough. The manual method, which is needed for many providers, requires the separate hook - which many people are oblivious to. That hook introduced in 2017. dns-Lexicon added it in 2018. Acme.sh introduced cleanup of txt records in 2019. IIRC, 2019/2020 is when most of the various clients started introducing that, because "oh sh!t."
This is why you'll often see people posting issues on the forum because there are too many TXT records, though their problems are often related to overstuffing their DNS provider's records instead of ISRG being able to parse them.
I haven't audited clients for several years, but this was a missing feature for quite some time in pretty much every client.
In the past year this has been rare and usually related to Cloudflare. Their interface apparently doesn't show any TXT values when there are too many which makes it difficult to clean up.
I appreciate if it was a problem years ago but doesn't seem to be much of one now.
I think what mholt was reacting to was your comment below which sounded like Certbot needed this. Or, maybe it was just me that interpreted it that way.
Some CA's don't cache validations, so for them each order results in a whole new set of challenge responses to complete.
Outside of cached validations, every TXT challenge is a different value every time and no LE don't go back and double check TXT records if the validation is currently cached.
FWIW, Certify The Web has always had TXT record cleanup but it hasn't always been successful at it - supporting lots of DNS providers (often with their own constantly changing APIs) is a minefield and nowadays I leave most DNS API stuff to Posh-ACME internally. If you're only dealing with DNS for your own domain that's less of a problem. If you're doing it on a large scale (e.g. main subdomains) I'd suggest using CNAMEs to a zone dedicated to auth challenge responses, this avoids frequently messing with your main DNS zone unnecessarily.
Wow! Such an active community. This was my first post. I did not expect to get so many replies :-).
Many thanks to you all. I could have marked many of the answers as solution... Finally, I marked the first answer, which answered my question.
That is absolutely crucial IMO.