Wildcard certificate on multiple web servers using DNS TXT validation

[nodesocket]

Hello,

I generated a wildcard certificate successfully using:

certbot-auto certonly --agree-tos --manual --server https://acme-v02.api.letsencrypt.org/directory -d "*.my-domain.io"

And TXT DNS validation on one of my web servers (we have hundreds of subdomains) and four web servers. However, when I run the certbot-auto command on the next web server it is prompting to create yet another DNS TXT record which is different from the first.

Is it ok if I update the _acme-challenge DNS TXT record each time for each web server? When certbot-auto runs renew does it check the DNS TXT record each time? This will cause a problem as it won’t match.

[rg305]

Each request and subsequent renewal should require a new set of TXT records to prove control.
I could see how running this simultaneously on multiple systems could become a fight for TXT record control.
I’m not sure how that contention can be managed effectively short of having all the systems initiate obtaining certs on a random +/- window (say +/- 2 hours) and then renew their certs at more exact intervals (like when 30 days remain). This should line them up in their success order and continue to renew in that same order.

Imaging that only one server would update each day - then you would have 60 days worth of no contention; or 60 servers each with its’ own day for renewing.

[nodesocket]

Thanks for the reply. If renewal on each web server checks the DNS TXT, and it currently uses a single record _acme-challenge.my-domain.io even staggering won’t work automated. I’d have to know when each web server is going to renew and update the DNS TXT record to match.

Does it make sense instead of using _acme-challenge.my-domain.io to use a semi-unique DNS TXT record? Something like _acme-challenge-dT4B1C4e.my-domain.io that way each web server has a separate validation DNS TXT record?

[rg305]

It has to follow the RFC.
But I get your point, even adding just one single char would allow for 39 distinct concurrent validations.
We need to bring in the guys that know the nitty gritty details…
Maybe @schoen can assist with this:
The question being: How does LetsEncrypt handle concurrent dns validations?

[mnordhoff]

@nodesocket

Is it ok if I update the _acme-challenge DNS TXT record each time for each web server?

Yes. You can change or delete the TXT record as soon as a validation is completed. An old TXT record never matters again, so you can do anything you want.

@nodesocket

When certbot-auto runs renew does it check the DNS TXT record each time? This will cause a problem as it won’t match.

"cerbot-auto renew", by default, tries to renew any certificates expiring in less than 30 days. It's recommended to run it twice a day. (The other 59 or so days, it won't do anything.)

Authorizations are currently valid for 30 days.

So, typically, you'll have to validate again -- with a new TXT record value -- every time you renew.

(Authorizations are specific to an account. Consider whether to share one account between all of your servers, or use a different account on each server.)

@nodesocket

If renewal on each web server checks the DNS TXT, and it currently uses a single record _acme-challenge.my-domain.io even staggering won’t work automated. I’d have to know when each web server is going to renew and update the DNS TXT record to match.

You can't know the new TXT record value before starting a new validation. And you should clean up and delete old TXT records. So that sort of conflict isn't a factor.

@nodesocket

Does it make sense instead of using _acme-challenge.my-domain.io to use a semi-unique DNS TXT record? Something like _acme-challenge-dT4B1C4e.my-domain.io that way each web server has a separate validation DNS TXT record?

The existing protocol and software always use _acme-challenge.my-domain.io, though you can use CNAME records.

@rg305

The question being: How does LetsEncrypt handle concurrent dns validations?

As far as Let's Encrypt goes, concurrent DNS validations are pretty much fine. If you have multiple TXT records, Boulder will go through the list and see if one of them matches.

In the current implementation, the DNS response can't be larger than 4096 bytes, RRSIG records not counted. No one should need scores of TXT records to exist simultaneously.

DNS providers and ACME clients can be a problem. DNS providers don't always support an atomic, non-racey way to add and remove records from a record set. ACME clients don't always check, just overwriting or deleting the record set. Any client that supports wildcards has probably improved or entirely fixed this, since so many people now do two simultaneous validations for the same name.

@nodesocket

It would be best to use a DNS provider and ACME client that support fully automated, race condition-free TXT record changes. Remember that you can delegate or CNAME the subdomain to a different DNS provider. (Usually.)

Also consider sharing the same certificate on all servers, perhaps centrally managed by one server and copied to the others.

Integration Guide

[jsha]

The nice thing about the DNS challenge is that it doesn’t need to run on the web server. I think in your situation the best plan would be to have one machine that does the DNS validation and issues the certificate, then pushes a copy out to each of your four web servers and instructs those servers to reload their certificates. Ideally this process would be automated (if your DNS provider supports automation), but you can also do it manually.

[rg305]

The question is: How does LetsEncrypt handle concurrent dns validations?
And the answer seems to be:
Use the same LE account on multiple systems to reduce (part of the) validation.
Centralize DNS validations to only one system.
Delete any “old” records at will.

That seems to be less than ideal for someone who can’t do any of those options.
…
Maybe DNS scavenging could automatically delete stale TXT records in the challenge zone…

And we haven’t really covered rate limits accurately in the response.

[nodesocket]

@mnordhoff thanks so much for the reply. So part one of my problem was indeed solved, just by adding multiple values to the single DNS TXT _acme-challenge.my-domain.io record.

Part two though is still an issue I believe. Currently we run certbot-auto rewnew in cron. After the 30 days, is this command going to fail or block prompting with an updated DNS TXT record? Is there a way to automate this process?

[Osiris]

@nodesocket

Part two though is still an issue I believe. Currently we run certbot-auto rewnew in cron. After the 30 days, is this command going to fail or block prompting with an updated DNS TXT record?

Using the manual plugin without scripts (see the manual about scripting and the manual plugin), the renew command will fail to run.

Renew is meant to be fully automates, so you'll need to have that in order first. See the link above

[jsha]

@nodesocket

After the 30 days, is this command going to fail or block prompting with an updated DNS TXT record? Is there a way to automate this process?

Yes, when the certificate is due for renewal, your Certbot command with --manual will fail. Certbot has a bunch of plugins to automate updating DNS for various providers. See User Guide — Certbot 2.7.0.dev0 documentation for a list. Unfortunately, right now neither of the main recommended installation methods for Certbot actually allow you to use those plugins. If your DNS provider is on that list, my recommendation would be to follow the instructions in the Certbot docs for setting up a development environment, which will include the plugins. The Certbot team is actively working on improving their packaging so they can make it easier to install the DNS plugins.