Some background. We are running an Ookla speedtest daemon on more than one server. This also requires that we have Apache running. Our SOC requires all public facing services use SSL. The certificate used for the Ookla daemon must have all the hostnames used in it. Each host has a unique name and a shared DNS RR name. I did manage to get this all working, the systems are live and everyone is happy … well mostly.
I have run into the problem that certbot renewals will hit the DNS RR hostname and then get the wrong server with a unique + RR hostname returned. This causes the renewal to fail. I am thinking I could probably use DNS Challenge with a TXT record for just the RR hostname. However, I want to add this without taking down the live hosts. If they are not accessible via SSL for too long then Ookla drops the server and we have to put in a request to get it reviewed and back in Ookla’s database. This can take more than a day to resolve and I cannot have our Speedtest in HAZCON that long without upper management getting antsy and making my life miserable.
On the other hand, am I over thinking this and just need to add the Apache plugin? Or will that too hit the DNS RR issue? Thanks.
Added later:
For more context on the certificates, here is what I did to get them on each server (I am required by company policy to not place IP addresses or host names of our servers in publicly accessible forums. Not my call.):
Server1 > service apache24 stop ; certbot certonly --standalone -d hostname,speedtest.foo.bar ; service apache24 start
Server2 > service apache24 stop ; certbot certonly --standalone -d hostname,speedtest.foo.bar ; service apache24 start
Server3 > et cetera
Where speedtest.foo.bar is a pseudonym for the DNS RR domain. The updates tend to fail when checking the speedtest.foo.bar domain.
That's what it seems like to me. The Apache plugin should be able to perform live validation and renewal of both names without interrupting the service.
Okay, so I install the Apache plugin, certbot runs renewal on Server1, hits the RR DNS and gets sent to Server3. Server3 has a certificate with its unique hostname+RRhostname that then generates this error:
Attempting to renew cert (server1.foo.net) from /usr/local/etc/letsencrypt/renewal/server1.foo.net.conf produced an unexpected error: Failed authorization procedure. speedtest.foo.com (tls-sni-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: Incorrect validation certificate for tls-sni-01 challenge. Requested f5b55612b49d802fd705110b35c2ac36.2c2b9a12ee11f8ccf15fc7816976d267.acme.invalid from ###.###.###.###:443. Received 2 certificate(s), first certificate had names "server3.foo.net, speedtest.foo.com". Skipping.
All renewal attempts failed. The following certs could not be renewed:
/usr/local/etc/letsencrypt/live/server1.foo.net/fullchain.pem (failure)
I have to reload the Ookla service to load the new certificate anyway. So a short interruption for a few seconds reload not a problem. The problem is when it hits the DNS RR and fails like that shown above. I’m not sure the Apache plugin will solve that and I don’t want to mess with the servers until I know I have a working resolution that will limit down-time.
Ah, now I know what you mean by “then get the wrong server”. This is because of your use of TLS-SNI, which is being disabled imminently anyway.
Check that your Certbot isn’t ancient:
certbot --version
and then try using this as part of your Certbot invocation (still with the apache plugin):
--preferred-challenges http-01
Edit: upon re-reading your last post I realize that more than two endpoint servers exist. So the above advice isn’t going to work. I’m having a hard time picturing your scenario in my head, but your initial instinct of using DNS is probably right. However, Certbot doesn’t support mixing challenge types in a single certificate order.
So, instead of modifying only our RR DNS zone with a TXT record and using “–preferred-challenges dns” (Which is what I would prefer as it should not require me touch the servers if I can just get the TXT record for that zone). I need to modify each server and set up the Apache plugin and this will absolutely not hit the DNS RR problem? I mean, the DNS solution would be most excellent as it would require one change in one place, DNS. As I also manage our DNS I would just need an “okay” from my manager to implement that and I could do it at any time. We have a 5 minute TTL on the RR zone, so it would also propagate quickly.
To modify the live, customer facing servers requires me to open a ticket with our TAC, schedule possible down-time, submit a MOP, and do the work during our midnight to six AM maintenance window. You can see why I would rather do the DNS challenge if I can do that without taking down a server.
Edit: … and you answered part of the question as I was typing this. Thanks.
Worth mentioning since often people don’t realize it, but these are not long-term DNS records, they do change at every renewal and therefore have to be automated via Certbot.
But, the webserver remains totally uninvolved, which is what you want.
https://github.com/joohoi/acme-dns is another DNS approach that involves a permanent delegation of the _acme-challenge CNAME to a separate DNS server. Whether your boss will like it depends on the reason for their objection.
I was just informed why. We cannot enable RFC 2136 Dynamic Updates on any of our publicly facing DNS servers per SOC policy. If Let’s Encrypt servers could talk to it to get a TXT record then it is publicly facing and cannot use DDNS per that security policy. So, unless I can get a permanent TXT record to validate each domain for updates I cannot resolve this with Let’s Encrypt’s DNS “solution”.
Maybe you can do "both".
You could "satisfy" this policy by only allowing indirect access to the delegated zone.
That is, Internet DNS systems can only reach your external DNS servers which can have conditional forwarding set to the _acme-challenge.your-domain.tld zone (to an internal DNS server).
This would provide the DNS answer while NOT allowing updates to the zone from outside your network/control.