I'm unable to perform automated (via cron) certificate renewal using Route53 DNS challenge-authentication because the identically named CNAME record for this domain already exists, so certbot's route53 plugin cannot create a new record because of the record name conflict.
The existing CNAME record is needed so that the CDN could use it to issue/renew their own certificates, and unfortunatelly the record name they're checking is again _acme-challenge.
The error message I'm getting is:
"PluginError: An error occurred (InvalidChangeBatch) when calling the ChangeResourceRecordSets operation: [RRSet of type TXT with DNS name _acme-challenge.example.com. is not permitted because a conflicting RRSet of type CNAME with the same DNS name already exists in zone example.com.]
To use certbot-dns-route53, configure credentials as described at Configuration — Boto3 Docs 1.24.13 documentation and add the necessary permissions for Route53 access."
Is there any way to keep the CNAME record required by the CDN and make the Route53 plugin work on the same domain?
I can't think of one. But, at the risk of being off-topic ... can you use the HTTP challenge for your new cert instead? Let's Encrypt allows up to 100 names in the same cert.
A domain is a very large space... So, I'd have to say "YES, that is possible".
But it won't be possible for the exact same subset/FQDN of that domain; As there can't be any other records entered where a CNAME already exists for that FQDN.
Perhaps you can include the removal of the current CNAME record prior to your use and then replace the CNAME record after your use???
That said, there are three different types of authentication methods [@MikeMcQ already mentioned the most popular one "HTTP" - there is also "TLS-ALPN"], but neither of the others work with ROUTE53 (DNS auth), so they don't exactly qualify as an answer to the question you asked.
It also seems a little unusual to me that you're trying to get a certificate for the same exact name from two places at once: from your CDN and from whatever system you're running certbot on. Is the server "behind" the CDN? If so, I would expect that usually the server would be using a different name than the public name that the CDN is serving. And if not, can you explain how the same name is sometimes being served to the CDN and sometimes being served somewhere else?
That is, if you describe your infrastructure a bit more completely, then maybe people here could help you solve your problem in a different way. For instance, even if you do need a certificate for the same name in two places, you might be able to create it in one place and then securely copy it wherever else it needs to go.
So you have an _acme-challenge CNAME pointing to somewhere else, which is presumably a TXT record - can you update the "somewhere else"?
Who is your CDN?
Both systems can be trying to get a similar cert ... when it is a...
@MikeMcQ Occured to me as well, but back then when I was trying to figure this out, the only way to acquire the wildcard cert from LetsEncrypt was using DNS verification. Has anything changed since?
The CDN in question is Fastly.
And although the certificate they're issuing is not LetsEncrypt, for some reason they're asking for the identically named record... which is of CNAME type unlike what Certbot does (TXT, right?). If it were TXT I was kinda hoping there's a way to expand/update the resource record value with the new verification string. Sadly, the route53 plugin offers little flexibility by having only the 2-3 options it has.
@rg305 It is a wildcard cert, and yes, the origin server is behind the CDN with HTTP(S) access allowed only to the CDN. Fastly uses HTTPS to connect to the origin server(s), hence the need for the SSL certificate. Unrothodox practice perhaps, but for me it gets the job done.
Forgive the lack of the infrastructure setup details, but the entire SSL renewal and deployment process is cron/Ansible automated, so I'm looking for the quickest and simplest way to resolve this, and one that fits into the setup. Also, I'm seeing so many angles to approach this that I opted for the above non-Tl;dr problem description.
However, I believe Rudy's comment guided me to a possible solution - removing the CNAME _acme-challenge record first, fol. by re-adding it after Certbot run exits. It'd be a small hack using the AWS API, first to check if _acme-challenge record already exists with the value queried and remove it, so that certbot could do its job.
When I come to think of it, it'd be a simple addition to any DNS authentication plugin to at least check for existing resource records of the _acme-challenge name and handle "the exception thrown".
I thank you all for the input, and any further comments on the topic are appreciated.
I'll report back when and if I manage to sort this out using the method above.
No. DNS challenge needed for wildcard.
Any CA using the ACME protocol would use the same name. You can use CNAME to redirect to another DNS similar to HTTP challenge requests redirected with HTTP 301, 302, .... When you added the CNAME for Fastly you were delegating all ACME DNS challenges to a system they can manage. That is, when Fastly starts a DNS challenge it sees your CNAME and then looks for the TXT record in that destination. Hope this clarifies.
Yeah, I liked that idea too. It's too bad you can't just get the cert files from Fastly since it sounds like your cert will be identical (apart from being from a diff CA).
It is likely not identical; They are more likely to be combining many domains onto single certs.
So, if you did have their cert, you would likely have much more than you should.
Fair point. They may be combining many customer wildcards into one cert. Hard to know from the info given.
I see the issue, so Fastly are requiring the _acme-challenge record for themselves: Serving HTTPS traffic using Fastly-managed certificates | Fastly Help Guides
You do also have the option of providing your own certs to Fastly, so you'd remove the CNAME altogether and upload certs via their API when they renew. Serving HTTPS traffic using certificates you manage | Fastly Help Guides
Neither option is particularly simple but if they're going to squat the _acme-challenge CNAME that makes it easy for them and more complex for you. I believe Cloudflare does this is a more subtle way by using (transient) TXT records.
The approach of removing Fastly's CNAME record, fol. by re-adding it after Certbot client does its thing works fine by expanding the renewal command with '--pre-hook' and '--deploy-hook', along with appropriate pre/post scripts needed to query if the record exists and add/remove it as needed.
Felt like a nuisance to deploy aws-cli, but it was actually without hassle - it doesn't require super user access and can be installed to an unprivileged user's HOME folder. One could argue that adding the full-blown aws-cli for this purpose is a security concern, but with the account credentials already there for route53-plugin to use, it's just as big of a concern as the plugin itself, the security of the system as a whole and the AWS accunt privileges.
Thank you all for your suggestions and I hope someone finds this useful.