Certbot dns-rfc2136 matches wrong BIND view despite valid key

Help
1

My domain is: newtrail.galaxyproject.org

I ran this command: /opt/certbot/bin/certbot certonly --test-cert --non-interactive --dns-rfc2136 --dns-rfc2136-credentials /etc/letsencrypt/dns-rfc2136-credentials.ini --register-unsafely-without-email --agree-tos -d newtrail.galaxyproject.org

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Obtaining a new certificate
Performing the following challenges:
dns-01 challenge for newtrail.galaxyproject.org
Cleaning up challenges
Encountered exception during recovery: 
Traceback (most recent call last):
  File "/opt/certbot/local/lib/python2.7/site-packages/certbot/error_handler.py", line 124, in _call_registered
    self.funcs[-1]()
  File "/opt/certbot/local/lib/python2.7/site-packages/certbot/auth_handler.py", line 243, in _cleanup_challenges
    self.auth.cleanup(achalls)
  File "/opt/certbot/local/lib/python2.7/site-packages/certbot/plugins/dns_common.py", line 77, in cleanup
    self._cleanup(domain, validation_domain_name, validation)
  File "/opt/certbot/local/lib/python2.7/site-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 79, in _cleanup
    self._get_rfc2136_client().del_txt_record(validation_name, validation)
  File "/opt/certbot/local/lib/python2.7/site-packages/certbot_dns_rfc2136/dns_rfc2136.py", line 163, in del_txt_record
    .format(e))
PluginError: Encountered error deleting TXT record: The peer didn't know the key we used
Encountered error adding TXT record: The peer didn't know the key we used

My web server is (include version):

nginx version: nginx/1.14.0 (Ubuntu)
built with OpenSSL 1.1.1  11 Sep 2018
TLS SNI support enabled

The operating system my web server runs on is (include version): Ubuntu 18.04.3 LTS

My hosting provider, if applicable, is: AWS EC2

I can login to a root shell on my machine (yes or no, or I don’t know): yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot): certbot 0.40.1

Running BIND 9.11.3, the relevant configuration sections (the key is not my real key) are:

key "certbot." {
    algorithm hmac-sha512;
    secret "4N9qH7FfZxGpxPo6iIFraJAT2kwcaqF2S2ClxmhT3iKw85NGSUTTdfM6bWWa1qMGIjqCD/G+9gq6KW/aZvaZ4g==";
};

acl "galaxy" {
    10.0.0.0/8;
    127.0.0.1/32;
    ::1/128;
};

options {
    directory "/var/cache/bind";
    dnssec-validation auto;
    auth-nxdomain no;
    listen-on-v6  {
        any;
    };
    listen-on  {
        any;
    };
    notify explicit;
    recursion no;
    allow-recursion  {
        none;
    };
    allow-transfer  {
        key axfr-galaxy.;
        key axfr-global.;
    };
    allow-query  {
        any;
    };
    query-source address * port *;
    max-ncache-ttl 3600;
    lame-ttl 86400;
    cleaning-interval 120;
};

view "galaxy" {
    match-clients {
        !key certbot.;
        !key axfr-global.;
        key axfr-galaxy.;
        galaxy;
    };
    recursion yes;
    allow-recursion { any; };
    also-notify {
        3.136.94.7 key axfr-galaxy.;
        128.118.250.8 key axfr-galaxy.;
    };

    /* includes */
    include "/etc/bind/named.conf.default-zones";

    /* zones */
    zone "galaxyproject.org" IN {
        type master;
        file "/etc/bind/zones/galaxyproject.org.private.zone";
    };
};

view "global" {
    match-clients {
        key certbot.;
        !key axfr-galaxy.;
        key axfr-global.;
        !galaxy;
        any;
    };
    recursion no;
    allow-recursion { none; };
    also-notify {
        3.136.94.7 key axfr-global.;
        128.118.250.8 key axfr-global.;
    };

    /* zones */
    zone "galaxyproject.org" IN {
        type master;
        file "/etc/bind/zones/galaxyproject.org.zone";
        update-policy {
            grant certbot. zonesub TXT;
        };
    };
};

The certbot credentials file contains:

dns_rfc2136_name = certbot.
dns_rfc2136_secret = 4N9qH7FfZxGpxPo6iIFraJAT2kwcaqF2S2ClxmhT3iKw85NGSUTTdfM6bWWa1qMGIjqCD/G+9gq6KW/aZvaZ4g==
dns_rfc2136_port = 53
dns_rfc2136_algorithm = hmac-sha512
dns_rfc2136_server = 3.14.178.173

BIND logs:

11-Dec-2019 14:57:51.981 debug 2: tsig key 'certbot': unknown key
11-Dec-2019 14:57:51.982 error: client @0x7f1a580d5a50 3.14.178.173#48752: view galaxy: request has invalid signature: TSIG certbot: tsig verify failure (BADKEY)

The relevant bit there is view galaxy: this request should match the global view, but it does not. I am not sure where key 'certbot' comes from, but if I rename the key to certbot (without the trailing period) in all places, I get the same error. Using the same key with BIND’s nsupdate works (or would if not for an unrelated filesystem permission issue that I’ll fix, but certbot doesn’t get this far):

# cat certbot.key
key "certbot." {
    algorithm hmac-sha512;
    secret "4N9qH7FfZxGpxPo6iIFraJAT2kwcaqF2S2ClxmhT3iKw85NGSUTTdfM6bWWa1qMGIjqCD/G+9gq6KW/aZvaZ4g==";
};
# cat ddns-test.txt
server 3.14.178.173 53
debug
zone galaxyproject.org
update add _acme-challenge.newtrail.galaxyproject.org. 180 TXT "ddns-test"
show
send
# nsupdate -k certbot.key ddns-test.txt 
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:      0
;; flags:; ZONE: 0, PREREQ: 0, UPDATE: 0, ADDITIONAL: 0
;; ZONE SECTION:
;galaxyproject.org.		IN	SOA

;; UPDATE SECTION:
_acme-challenge.newtrail.galaxyproject.org. 180	IN TXT "ddns-test"

Sending update to 3.14.178.173#53
Outgoing update query:
;; ->>HEADER<<- opcode: UPDATE, status: NOERROR, id:  59627
;; flags:; ZONE: 1, PREREQ: 0, UPDATE: 1, ADDITIONAL: 1
;; ZONE SECTION:
;galaxyproject.org.		IN	SOA

;; UPDATE SECTION:
_acme-challenge.newtrail.galaxyproject.org. 180	IN TXT "ddns-test"

;; TSIG PSEUDOSECTION:
certbot.		0	ANY	TSIG	hmac-sha512. 1576077745 300 64 FOO BAR== 59627 NOERROR 0 


Reply from update query:
;; ->>HEADER<<- opcode: UPDATE, status: SERVFAIL, id:  59627
;; flags: qr; ZONE: 1, PREREQ: 0, UPDATE: 0, ADDITIONAL: 1
;; ZONE SECTION:
;galaxyproject.org.		IN	SOA

;; TSIG PSEUDOSECTION:
certbot.		0	ANY	TSIG	hmac-sha512. 1576077745 300 64 BAZ QUUX== 59627 NOERROR 0

BIND in this case logs:

11-Dec-2019 15:22:25.134 info: client @0x7f1a580c72c0 3.14.178.173#18699/key certbot: view global: updating zone 'galaxyproject.org/IN': adding an RR at '_acme-challenge.newtrail.galaxyproject.org' TXT "ddns-test"
11-Dec-2019 15:22:25.135 debug 1: journal file /etc/bind/zones/galaxyproject.org.zone.jnl does not exist, creating it
11-Dec-2019 15:22:25.135 error: /etc/bind/zones/galaxyproject.org.zone.jnl: create: permission denied
11-Dec-2019 15:22:25.135 info: client @0x7f1a580c72c0 3.14.178.173#18699/key certbot: view global: updating zone 'galaxyproject.org/IN': error: journal open failed: unexpected error
11-Dec-2019 15:22:25.135 debug 1: delete_node(): 0x7f1a5eefe210 _acme-challenge.newtrail.galaxyproject.org (bucket 6)

Here it’s landed in the right view. So it looks like nsupdate is doing something different in regards to how the key is sent. I haven’t dug in to the packets or DNSSEC spec yet to figure out what the difference is, but it looks like Certbot is missing something here?

1 Like
2

I tried to reproduce your issue with your configuration but wasn’t able to. My updates went to the global view:

Dec 11 21:31:31 bc048b1e1be8 named[12362]: client @0x7f05840f20a0 127.0.0.1#40772/key certbot: view global: updating zone 'galaxyproject.org/IN': adding an RR at '_acme-challenge.newtrail.galaxyproject.org' TXT "8YL6hx4EBSyEz5qIdF4OF0Wh3ZEIn-QcJ7BLTH79Sf8"
Dec 11 21:31:31 bc048b1e1be8 named[12362]: /etc/bind/zones/galaxyproject.org.zone.jnl: open: permission denied
Dec 11 21:31:31 bc048b1e1be8 named[12362]: client @0x7f05840f20a0 127.0.0.1#40772/key certbot: view global: updating zone 'galaxyproject.org/IN': error: journal open failed: unexpected error
Dec 11 21:31:31 bc048b1e1be8 named[12362]: client @0x7f057c002d20 127.0.0.1#40774/key certbot: view global: updating zone 'galaxyproject.org/IN': deleting an RR at _acme-challenge.newtrail.galaxyproject.org TXT

The two changes I had to make to your configuration to get to this point were:

  • Uppercasing hmac-sha512 in the plugin ini file
  • Removing references to axfr-{galaxy,global} since they’re undefined in your example

and the remaining error about journal permissions was fixed via directory ownership on /etc/bind/zones.

Packages:

bind9               9.11.5.P4+dfsg-5.1
certbot             1.0.0
certbot-dns-rfc2136 1.0.0
2 Likes
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

5

Thanks @_az for testing out my config, and sorry for not following up. I was pulled away and have just now returned to this work.

I’ve discovered the problem: apparently the dns_rfc2136_algorithm value is case sensitive: I needed to use HMAC-SHA512 instead of hmac-sha512. I had used lowercase since BIND uses lowercase and hadn’t considered that it might matter to Certbot.

5 Likes
6

@natefoo Thanks for coming back with your solution.

Hey… you didn’t post your real dns_rfc2136_secret up there, did you? Might want to regenerate that if it’s in use on your server.