Dns update for wildcard certificate not working anymore

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: backprod.de (same error on other domains)

I ran this command: sudo certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/renewal/rfc2136.ini -d 'backprod.de' -d '*.backprod.de' or
service certbot restart (this will also try to renew the abovementioned domain)

The operating system my web server runs on is (include version):
Ubuntu 22.04

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
No

I'm running a bind server on my VPS as primary nameserver with several secondaries.

Since update to ubuntu 22.04 (and with this also to a newer bind version) certbot wildcard url dns update does not work anymore. Currently I assume a change in bind but I'm not sure about this.

Log entry in certbot log:
2022-08-21 23:26:30,677:DEBUG:certbot_dns_rfc2136._internal.dns_rfc2136:No authoritative SOA record found for _acme-challenge.mydomain.de

Update command:
service certbot restart (auto) or manual
sudo certbot certonly --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/renewal/rfc2136.ini -d 'backprod.de' -d '*.backprod.de'

Setup in Bind:

Static Zone: backprod.de (no updates allowed)
Dynamic Zone auto.backprod.de (updates allowed)

The idea of this setup is that only the auto zone will be changed where all static entries in the main zone will not be modified automatically.

Alias in static zone: _acme-challenge.backprod.de. real name _acme-challenge.auto.backprod.de. (this is used for e.g. for dehydrated which is still working like before only with certbot I'm running into problems now)

This setup did work in past without problems with dehydrated script and certbot. Seems to be that an update of bind changed this.

If I take a look in the bind logs I will find this:
22-Aug-2022 22:10:12.712 update-security: info: client @0x7f4b9aedd768 127.0.0.1#49886/key mykey: update 'backprod.de/IN' denied

Well this message makes more sense. But I would assume that mydomain.de should not be updated because it's static and I never set the option in past that it is alowed to update this zone. Instead auto.mydomain.de should be updated. I'm not sure if this was handeled different by bind in past. But definately there was never an explicit authorization to update this zone. So the only zone which certbot should be able to update in past is the auto subzone.

Well after allowing updates in backprod.de for test purposes I'm getting the followin error messages which make no sense at all:

No authoritative SOA record found for _acme-challenge.backprod.de
Received authoritative SOA response for backprod.de - Well for me that sounds contradictionary to the message above. This is the SOA for every subdomain and _acme-challenge is included.
Successfully added TXT record _acme-challenge.backprod.de - Well that sounds good
DNS problem: NXDOMAIN looking up TXT for _acme-challenge.backprod.de - check that a DNS record exists for this domain - Well this should exist because certbot is telling me that the record is created. But this seems to be wrong. I don't see the "created" record in bind. So I assume there is no record created althoug the message above is a success message.

If I manually create the text record at _acme-challenge.auto.backprod.de. certbot is telling me:
Certbot failed to authenticate some domains (authenticator: dns-rfc2136). The Certificate Authority reported these problems:

  • Domain: backprod.de*
  • Type: unauthorized*
  • Detail: Incorrect TXT record "test" found at _acme-challenge.backprod.de*

Yeah, sure. If this entry is not modyfied by certbot this could not be working - not a suprise. For me this is looking like in past certbot was able to change the entry in auto.backprod.de via the alias. Now it tries to change the previous static master zone and fails for whatever reason. Certbot should also delete this text entry with every run but this also does not happen.

I also tried to remove the alias entry but than I'm getting:
Received response from server: SERVFAIL

So all this error messages of certbot are not very helpful.

Certbot should create this entry. That's the main task of certbot and what it's purpose. But this does not seem to be working anymore. Not sure if a change in bind caused this and if this has to be fixed in bind or certbot. Any ideas why this worked in past and it's not working anymore?

I didn't think that the RFC2136 plugin ever followed CNAME entries in the past. There are a few long standing feature requests and PRs for this functionality.

Are you sure you didn't have a zone for _acme-challenge.backprod.de. and Certbot was updating that?

The way the RFC2136 plugin logic works, it will search for, in order:

  1. An SOA record for _acme-challenge.backprod.de
  2. An SOA record for backprod.de
  3. An SOA record for de

and will try to create the relative record in the first authoritative (AA) zone it finds. It makes sense that Certbot tried to create a TXT record in (2).

Certbot has never had any functionality which would enable it to see that _acme-challenge.backprod.de is an alias of _acme-challenge.auto.backprod.de, or the functionality to follow such an alias.

5 Likes

Which version of certbot are you running?

3 Likes

Thank you for your reply. Well that would explain the message order
No authoritative SOA record found for _acme-challenge.backprod.de
Received authoritative SOA response for backprod.de
because it checks for the _acme-challenge zone first (which does not exist). After that it checks for zone backprod which does exist.

Yes absolutely. The same is valid for the second domain torstens-buecherecke.de

Due to the fact I'm using btrfs filesystem with snapshots I can check all old logs and all old files if that helps. So I checked the status messages for beginning of this year and found the auto zone entries for the domain torstens-buecherecke.de (I switched to this domain because for backprod the problem seems to be another one because it still exists before the change to ubuntu 22.04). So I continued my seach for the domain torstens-buecherecke.de.

The setup idea is the same for this domain. So there is also the auto zone for which is dynamic and the standard zone which is not.

This is looking pretty much like certbot worked with the auto zone before without problems and same setup (at least I have not changed anything):

Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2021-11-17T20:46:42Z",
  "identifiers": [
    {
      "type": "dns",
      **"value": "*.torstens-buecherecke.de"**
    },
    {
      "type": "dns",
      "value": "torstens-buecherecke.de"
    }
  ],
  "authorizations": [
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/48052101030",
    "https://acme-v02.api.letsencrypt.org/acme/authz-v3/48052101040"
  ],
  "finalize": "https://acme-v02.api.letsencrypt.org/acme/finalize/35589629/38654796500"
}
2021-11-10 21:46:43,223:DEBUG:acme.client:Storing nonce: 01021bpW0bcAGym4E3O53SHhN3WJzCBcW7FefTKD5QVZySA
2021-11-10 21:46:43,224:INFO:certbot.auth_handler:Performing the following challenges:
2021-11-10 21:46:43,224:INFO:certbot.auth_handler:dns-01 challenge for torstens-buecherecke.de
2021-11-10 21:46:43,224:INFO:certbot.auth_handler:dns-01 challenge for torstens-buecherecke.de
2021-11-10 21:46:43,236:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**No authoritative SOA record found for _acme-challenge.auto.torstens-buecherecke.de**
2021-11-10 21:46:43,241:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Received authoritative SOA response for auto.torstens-buecherecke.de**
2021-11-10 21:46:43,268:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Successfully added TXT record**
2021-11-10 21:46:43,272:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**No authoritative SOA record found for _acme-challenge.auto.torstens-buecherecke.de**
2021-11-10 21:46:43,274:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Received authoritative SOA response for auto.torstens-buecherecke.de**
2021-11-10 21:46:43,280:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Successfully added TXT record**
2021-11-10 21:46:43,282:INFO:certbot.plugins.dns_common:Waiting 60 seconds for DNS changes to propagate
2021-11-10 21:47:43,320:INFO:certbot.auth_handler:Waiting for verification...
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "torstens-buecherecke.de"
  },
  "status": "valid",
  "expires": "2021-12-10T20:47:45Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "**valid**",
      "url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/48052101040/nLwuNg",
      "token": "jO2VRdK9J-opxPrYhLK4PoGuvJsOH-D_csLAF7x4Nik",
      "validationRecord": [
        {
          "hostname": "torstens-buecherecke.de"
        }
      ],
      "validated": "2021-11-10T20:47:43Z"
    }
  ]
}
2021-11-10 21:47:48,439:DEBUG:acme.client:Storing nonce: 0102l8ZdhkdTk9mOgQ1qoNesOVylC4gABtZHJcJSeWd43Xw
2021-11-10 21:47:48,440:DEBUG:certbot.error_handler:Calling registered functions
2021-11-10 21:47:48,440:INFO:certbot.auth_handler:Cleaning up challenges
2021-11-10 21:47:48,444:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**No authoritative SOA record found for _acme-challenge.auto.torstens-buecherecke.de**
2021-11-10 21:47:48,454:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Received authoritative SOA response for auto.torstens-buecherecke.de**
2021-11-10 21:47:48,467:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Successfully deleted TXT record**
2021-11-10 21:47:48,471:DEBUG:certbot_dns_rfc2136.dns_rfc2136:No authoritative SOA record found for _acme-challenge.auto.torstens-buecherecke.de
2021-11-10 21:47:48,473:DEBUG:certbot_dns_rfc2136.dns_rfc2136:**Received authoritative SOA response for auto.torstens-buecherecke.de**

I can't tell which version was running when creating this log entry. Currently without the snap the command "apt-cache policy certbot | grep Installed" is telling me 1.21.0-1build1. So I'm not sure if the version changed from Ubuntu 20.04.

Before I created this help request I installed the snap version like described and removed the old version before. But that did not help. I'm not a fan of snap at all but if it woul help I would stay with that version.

Based on the information above this should never have worked because certbot whould never update the auto zone because the domain is just torstens-buecherecke.de. So why did it work for months or even years without problems? I also checked the config files but I don't see where I should have made a change to get this working like this in past.

1 Like

I can't speak to your DNS questions. But, at the top of the log (like 2nd line) did it have something like this version info

2022-08-22 17:46:04,192:DEBUG:certbot._internal.main:certbot version: 1.29.0
3 Likes

I don't see this in my log. What I see is DEBUG:certbot.main:certbot version:but this seems to be not the version of certbot but only the version which is set in the config file via version =

In my case this is version = 0.40.0 but I could write everyhing in there this just seems to be a text. This is also shown in the new logs so this log entry not very helpful.

To be honest, this log file seems impossible to me.

The only way I can think to make Certbot behave this way would be to modify Certbot's source code directly.

I thought for a moment that your old installation may have been relying on the short-lived dns_rfc2136_base_domain feature, which was added in v0.35.0 and removed again in v0.35.1 due to being buggy. However, your Certbot log does not match up with how that feature worked.

3 Likes

Well I definately don't do that. So it worked in past and now it don't. Seems to be that was a feature which was not planned to deliver. :wink: Is there any way to see the certbot version with just the complete backup of filesystem with a non active running system? In this case I can tell which version was running.

I have created an additional zone now for _acme-challenge.torstens-buecherecke.de with that the update is working. Due to the fact I have to remove the cname entry to get this zone up and running I have to change dehydrated code now because this worked based on cname entry. I hope that this will not cause other problems.

If the package was installed with apt on Ubuntu, you could try something like:

# find /usr/lib/python3/dist-packages/certbot*.egg-info -maxdepth 0
/usr/lib/python3/dist-packages/certbot-0.40.0.egg-info
/usr/lib/python3/dist-packages/certbot_dns_rfc2136-0.35.1.egg-info
4 Likes

System Status from 01/2022:
/btrfs/sync1/.snapshots/12530/snapshot/usr/lib/python3/dist-packages/certbot-0.40.0.egg-info
/btrfs/sync1/.snapshots/12530/snapshot/usr/lib/python3/dist-packages/certbot_apache-0.39.0.egg-info
/btrfs/sync1/.snapshots/12530/snapshot/usr/lib/python3/dist-packages/certbot_dns_rfc2136-0.35.1.egg-info

Now on the current system:
/usr/lib/python3/dist-packages/certbot-1.21.0.egg-info
/usr/lib/python3/dist-packages/certbot_apache-1.21.0.egg-info
/usr/lib/python3/dist-packages/certbot_dns_rfc2136-1.21.0.egg-info

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.