DNS challenge failure - Azure child zone

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: accounts.hillmangroup.net

I ran this command: I'm using Certify The Web app

It produced this output:
[ERR] DNS update failed: Azure DNS API :: Operation returned an invalid status code 'NotFound'

My web server is (include version): N/A

The operating system my web server runs on is (include version): N/A

My hosting provider, if applicable, is: N/A (azure for DNS)

I can login to a root shell on my machine (yes or no, or I don't know): N/A

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):

I am using Certify The Web desktop app for testing/validation of DNS challenges against Azure hosted DNS via a service principal. In order to limit the access of this service principal I created a child domain (_acme-challenge.accounts.hillmangroup.net).

If I run the 'test' function in the app - it creates and deletes the expected TXT record in the child domain without issue. Confirmed via Azure audit logs. When I attempt to issue a cert - it fails with error:

DNS update failed: Azure DNS API :: Operation returned an invalid status code 'NotFound'

I can generate a cert without fail if I do not create a child zone - using same service principal. I've opened a case with MS and posting here on the chance this has a simple explanation.

1 Like
4

Hi @MrNeuman and welcome to the LE community forum :slight_smile:

What does the DNS zone look like?
Are there any CNAME or "@" or "*" entries?
Is there any entry for "accounts"?
[I suspect there is, and it is causing you this trouble]
As shown by the following inconsistency:

nslookup -q=ns accounts.hillmangroup.net ns1-09.azure-dns.com
hillmangroup.net
        primary name server = ns1-09.azure-dns.com
        responsible mail addr = azuredns-hostmaster.microsoft.com
        serial  = 1
        refresh = 3600 (1 hour)
        retry   = 300 (5 mins)
        expire  = 2419200 (28 days)
        default TTL = 300 (5 mins)

nslookup -q=ns nxd-accounts.hillmangroup.net ns1-09.azure-dns.com
*** UnKnown can't find nxd-accounts.hillmangroup.net: Non-existent domain
1 Like
5

Parent zone (hillmangroup.net) has its required NS/SOA records and delegation NS records for the child zone (_acme-challenge.accounts).

The child zone (_acme-challenge.accounts.hillmangroup.net) has its required NS/SOA records only. I created an empty TXT record in this zone at one point in my testing thinking perhaps the challenge was looking to update rather than create a TXT record - but that had no effect.

1 Like
6

The child zone should be "accounts".
Then "_acme-challenge" can be a TXT record in it.

1 Like
7

The domain I ultimately need to implement this against has a load balanced front-end answering for 'accounts' - advertised with a CNAME entry. CNAME records are not allowed at the root of a zone - thus precluding me from using 'accounts' as a child zone.

1 Like
9

Why do you need to do a CNAME for a child zone?

Sounds like you only need an "@" and/or "*" record within that child zone.

I may be missing a piece of the puzzle.

1 Like
10

OK rethinking this over (and over).

You may be able to CNAME[delegate] the entire "accounts" [zone].
And also enter a TXT record for _acme-challenge.accounts.
[because the more specific answer should come first]

What is the TXT record for _acme-challenge.accounts? Here you go! [Go ask these other DNS servers...]
What is the A record for _acme-challenge.accounts? Go ask these other DNS servers...

OR

Handle all things "accounts" related within the delegated DNS system.

1 Like
11

Thanks for the effort...

CNAME required for 'accounts' due to 3rd party hosted CDN or similar service.

I have a workaround of not using a child domain at all. Unfortunately this requires me to delegate permission to the entire parent zone for the service principal - which is what I am trying to avoid.

My thinking was that the process would simply create an '@' TXT record in the child zone. Which the test process does no problem. The failure only comes when attempting to actually issue a cert.

Originally I attempted to grant permission on only the required TXT record in the parent zone - as Azure allows delegation down to an individual record level. This failed - but appeared to be an Azure permission problem of not have appropriate READ rights at the parent domain level.

1 Like
12

Sounds like you may need to create some distance between the domain and the service provider: Add a delegated subdomain (for their use).

1 Like
13

Thank you again for the feedback. Suggestions aside - as I have known workarounds - my question remains.

I can run a test in the Certify the Web app - and the TXT record is created as expected. But it fails when the LE cert is requested. Is this expected in this context of having the '_acme-challenge.x' as a zone of its own? Is this an LE concern or a Certify the Web concern?

Test results:

Azure DNS API :: DNS TXT Record Created: _acme-challenge-test.accounts.hillmangroup.net in root domain _acme-challenge.accounts.hillmangroup.net with value: dv4q9bCSBJHQ8DcSwLk_mZLTFGbwILL_hLTVh_R60_Q

Cert requested results:
2021-12-28 12:59:39.112 -05:00 [INF] Beginning Certificate Request Process: using ACME Provider:Certes
2021-12-28 12:59:39.113 -05:00 [INF] Requested identifiers to include on certificate: accounts.hillmangroup.net
2021-12-28 12:59:39.113 -05:00 [INF] Beginning certificate order for requested domains
2021-12-28 12:59:39.113 -05:00 [INF] BeginCertificateOrder: creating/retrieving order. Retries remaining:2
2021-12-28 12:59:39.702 -05:00 [INF] Created ACME Order: https://acme-v02.api.letsencrypt.org/acme/order/337179820/50364815510
2021-12-28 12:59:39.856 -05:00 [INF] Fetching Authorizations.
2021-12-28 12:59:40.310 -05:00 [INF] Got http-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/62244980360/BMdPAw
2021-12-28 12:59:40.458 -05:00 [INF] Got dns-01 challenge https://acme-v02.api.letsencrypt.org/acme/chall-v3/62244980360/GJ6oVQ
2021-12-28 12:59:40.458 -05:00 [INF] Attempting Domain Validation: accounts.hillmangroup.net
2021-12-28 12:59:40.459 -05:00 [INF] Registering and Validating accounts.hillmangroup.net
2021-12-28 12:59:40.459 -05:00 [INF] Preparing automated challenge responses (accounts.hillmangroup.net)
2021-12-28 12:59:40.465 -05:00 [INF] DNS: Creating TXT Record '_acme-challenge.accounts.hillmangroup.net' with value 'xxxxxxx', in Zone Id '_acme-challenge.accounts.hillmangroup.net' using API provider 'Azure DNS API'
2021-12-28 12:59:41.462 -05:00 [ERR] DNS update failed: Azure DNS API :: Operation returned an invalid status code 'NotFound'
2021-12-28 12:59:41.462 -05:00 [INF] Requesting Validation: accounts.hillmangroup.net
2021-12-28 12:59:41.462 -05:00 [INF] Azure DNS API :: Operation returned an invalid status code 'NotFound'
2021-12-28 12:59:41.921 -05:00 [INF] Azure DNS API :: Operation returned an invalid status code 'NotFound'
2021-12-28 12:59:41.921 -05:00 [INF] Azure DNS API :: Operation returned an invalid status code 'NotFound'

1 Like
14

Can you place a test TXT record at the expected location?
Can you issue a cert manually?

This makes NO sense:

It says it created the _acme-challenge-test within the _acme-challenge zone.
That looks like a fail to me.

2 Likes
15

The ugly explanation for the 'successful' creation was revealed by the Azure audit logs.

The app is creating a TXT record of name '_acme-challenge-test.FQDN' in whichever zone is configured in its Domain Authorization panel.

So in my test run - with app configued to use the zone '_acme-challenge.hillmangroup.net'. It created a TXT record of:

_acme-challenge-test.accounts.hillmangroup.net._acme-challenge.accounts.hillmangroup.net

Lovely.

I was hopeful the DNS-01 challenge would create a TXT record of '@' within a dedicated subdomain of name '_acme-challenge'. This would be helpful from a delegation of permissions standpoint within Azure DNS. Many of the docs on this topic suggest using the Azure 'DNS Zone Contributor' role within the domain you are issuing certs against. This role exposes more permissions than I care to grant to an API - especially at the domain level.

2 Likes
16

Hi, I'm the developer of Certify The Web. The app very recently (2 weeks ago) added CNAME delegation support for DNS challenges but it's unclear to me if you are using this feature or not, your description suggest you cannot use CNAMEs in your domain. I'm unclear on whether you are actually trying to get a cert for hillmangroup.net or accounts.hillmangroup.net.

> Terminology tip: Delegation here is not about setting up NS records to handle subdomain queries from another zone etc, it's just about redirecting TXT record queries.

ACME dns validations expect to find a TXT record called _acme-challenge.your.domain.com, so for accounts.hillmangroup.net it needs to resolve _acme-challenge.accounts.hillmangroup.net to a TXT record. Normally the app expects that domain to be in the zone you have provided API credentials for. You cannot rewrite the TXT record name to a different zone without extra configuration (a CNAME delegation rule).

Did you create a child zone called _acme-challenge? If so I'd recommend renaming that as auth or something else as calling it _acme-challenge would be incredibly confusing :slight_smile:

Here's how I'd set it up to use another domain/subdomain for DNS validation for a cert for accounts.hillmangroup.net, but using auth.hillmangroup.net specifically for the DNS challenges

  • In the DNS control panel for hillmangroup.net add a CNAME _acme-challenge.accounts pointing to _acme-challenge.accounts.auth.hillmangroup.net, make sure there are no stray _acme-challenge records or name conflicts in this zone.

Create a new managed certificate in the app:

  • Set the certificate domain to be accounts.hillmangroup.net
  • On the authorization tab, select dns-01 > Azure DNS.
  • Select/add your credentials for the service account which can manage the auth.hillmangroup.net zone
  • Click ... to populate the zone dropdown list and choose ``auth.hillmangroup.net` as the zone you are going to write to.
  • set CNAME delegation rule to *.accounts.hillmangroup.net:*.auth.hillmangroup.net
  • Save then click Test to ensure the records can be written/deleted from the target DNS zone.

As mentioned, this is a new feature, so there could be bugs but it's important that your configuration is correct and that the expected outcome is clear.

If on the other hand you are entirely unable to use CNAMEs due to other constraints, start again without attempting delegation to another domain.

2 Likes
17

Note for the above I'm assuming you actually want a cert for accounts.hillmangroup.net if you just want a cert for hillmangroup.net then remove all .accounts references.

2 Likes
18

Thank to all for your feedback & suggestions.

I was attempting to use a child zone for '_acme-challenge.accounts' to obtain a cert for 'accounts.hillmangroup.net'. I was not using the CNAME feature of the app.

The fact that the test worked successfully is where things got confusing. I didn't dig into the details enough to see what TXT record was actually being created.

In hindsight it's clear that TXT records manipulated by the DNS challenge should not be created as an '@' record in the root of a zone - as those frequently contain other critical values related to email/etc.

This was my first effort with automated certificate management. My goal was to limit DNS records exposed to the Azure service principal. The subdomain approach was my first thought and the app enabled me to really dig into this and understand how the process works.

I pivoted to creating a custom Azure role - which will be a better solution in the end since it further restricts what record types the service principal can manipulate.

3 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.