The peer didn't like the signature we sent

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: mousetech.com

I ran this command: certbot -v certonly --dns-rfc2136 --dns-rfc2136-credentials /root/certbot-credentials.ini -d '*.mousetech.com'

It produced this output:
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None
Requesting a certificate for *.mousetech.com
Performing the following challenges:
dns-01 challenge for mousetech.com
Cleaning up challenges
Encountered exception during recovery: certbot.errors.PluginError: Encountered error deleting TXT record: The peer didn't like the signature we sent
Encountered error adding TXT record: The peer didn't like the signature we sent
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): nginx 1.20.1

The operating system my web server runs on is (include version): AlmaLinux 9.4

My hosting provider, if applicable, is: self

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no (command-line, ansible, puppet)

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 2.11.0

Additional context:

The DNS servers have been in continuous operation since the previous millenium (bind 8 or earlier), so there's likely some cruft in them, Current release is bind 9.16.23.

Renewals fail with the same error on host-specific entries (e.g.,: gourmetj.mousetech.com", not just the domain wildcard.

The server containing nginx and primary dns was just migrated from CentOS 7 to AlmaLinux 9. No problems with renewals before that.

An nsupdate test to add a new TXT record reports no errors..

As I understand it, the failing operation uses a library python module not unique to letsencrypt, but nobody has explained what the error means or how to fix it.

2 Likes
2

I haven't used RFC 2136 myself, and haven't seen that error before myself. I searched the forums and saw this post that might be related?

But I'll see what I can do, which may not be much.

Well, that does seem likely to be related.

Can you post more details, like:

  1. the command you're running to test and its output
  2. anything related to the configuration of the RFC2136 Certbot plugin, and of Bind to accept it, other than the actual credentials of course
  3. Any logs you see on the Bind side when certbot is trying to update the TXT record
4 Likes
3

I've got more info, but the message board says "no more than 2 users in a post" and rejects it both from email "reply all" and the web reply.

I have no idea where it's finding those 3+ users.

4

Are there @ signs in the text you're trying to post, so it thinks you're trying to tag people to add to the conversation, or something like that?

Post code/config/log/etc. excerpts between lines of three backticks

```
like this
```

Which will turn it into a nice block

like this

Which the forum won't try to interpret further.

5 Likes
5

Thanks for the quick response. Incidentally, the server upgrade was complete. I built a whole new OS disk from scractch rather than upgrading in-place. So internal machine signatures and ssh keys are now different.

I'll check the link, although I had a merry time figuring out that certbot wanted to target my internal domain referencing its internal IP address when I first got wildcards working, so I'm fairly confident on that.

On Wed, 2024-07-10 at 15:17 +0000, Peter Cooper Jr. via Let's Encrypt Community Support wrote:

Here's a sample from the DNS logs:

Jul 09 17:42:59 www5.mousetech.com named[564422]: client @0x7f87200211c8 96.90.14.153#51122/key rndc-key: view external: updating zone 'mousetech.com/IN': adding an RR at 'www.mousetech.com' TXT "testing"
Jul 09 17:42:59 www5.mousetech.com named[564422]: zone mousetech.com/IN/external: sending notifies (serial 2024062402)
Jul 09 17:43:54 www5.mousetech.com named[564422]: client @0x7f87200211c8 96.90.14.153#32952/key rndc-key: view external: updating zone 'mousetech.com/IN': deleting rrset at 'www.mousetech.com' TXT
Jul 09 17:43:54 www5.mousetech.com named[564422]: zone mousetech.com/IN/external: sending notifies (serial 2024062403)
Jul 09 17:47:41 www5.mousetech.com named[564422]: client @0x7f872002c2f8 10.0.1.3#45223 (mousetech.com): view internal: transfer of 'mousetech.com/IN': IXFR started (serial 2024062401 -> 2024062403)
Jul 09 17:47:41 www5.mousetech.com named[564422]: client @0x7f872002c2f8 10.0.1.3#45223 (mousetech.com): view internal: transfer of 'mousetech.com/IN': IXFR ended: 1 messages, 8 records, 300 bytes, 0.001 secs (30000>
Jul 09 17:55:59 www5.mousetech.com named[564422]: dumping master file: rename: mousetech.com-external: permission denied
Jul 09 18:10:43 www5.mousetech.com named[564422]: dumping master file: rename: mousetech.com-external: permission denied
Jul 09 18:17:59 www5.mousetech.com named[564422]: client @0x7f8724020028 10.0.1.3#42527 (internal.mousetech.com): view internal: transfer of 'internal.mousetech.com/IN': AXFR-style IXFR started (serial 2024070502)
Jul 09 18:17:59 www5.mousetech.com named[564422]: client @0x7f8724020028 10.0.1.3#42527 (internal.mousetech.com): view internal: transfer of 'internal.mousetech.com/IN': AXFR-style IXFR ended: 1 messages, 28 records>
Jul 09 18:23:19 www5.mousetech.com named[564422]: dumping master file: rename: mousetech.com-external: permission denied

Jul 10 10:02:02 www5.mousetech.com named[564422]: dumping master file: rename: mousetech.com-external: permission denied
Jul 10 10:12:21 www5.mousetech.com named[564422]: client @0x7f8737ac9728 10.0.1.5#52355/key rndc-key: view external: updating zone 'mousetech.com/IN': adding an RR at '_acme-challenge.mousetech.com' TXT "192.168.1.1"
Jul 10 10:12:21 www5.mousetech.com named[564422]: zone mousetech.com/IN/external: sending notifies (serial 2024062404)
lines 1074-1107/1107 (END)

There's a lot of "dumping permission denied", but I think that's just a journaling issue unrelated to certbot. I've see it before and fixed it. As I read the logs, they appear to confirm the TXT injection and its replication to the backup server.

The test for injecting the TXT looks like this:

nsupdate -k /etc/rndc.key
> server 10.0.1.5
> update add _acme-challenge.mousetech.com 86400 TXT 192.168.1.1
> send
> quit

There's an option for dig/nslookup to display the TXT, but I haven't tried it because when nsupdate is successful, it terminates silently, but it complains bitterly when it fails. It waits until you issue the "send" command to tell you, though!

2 Likes
6

Double check that your server time is accurate and synced with internet time. Check also that if you have recently run updates on the server (since the last renewal) that the tool versions still support the signature type your bind understands.

3 Likes
7

Clock is OK. Tried a system update, and found conficting versions of nginx, so I deleted nginx and nginx-core and re-installed to clear the conflicts. Also switched off selinux. I keep forgetting that new installs have it on by default.

Earlier I stashed my /etc/letsencrypt, uninstalled and re-installed certbot. overlaid the stashed letsencrpt.

This is the end result of all of the above:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator dns-rfc2136, Installer None

Please choose an account
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
1: www5.mousetech.com@2024-07-10T14:17:51Z (c3c6)
2: www5.mousetech.com@2018-08-23T10:16:32Z (0cec)
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Select the appropriate number [1-2] then [enter] (press 'c' to cancel): 1
Requesting a certificate for *.mousetech.com
Performing the following challenges:
dns-01 challenge for mousetech.com
Cleaning up challenges
Encountered exception during recovery: certbot.errors.PluginError: Encountered error deleting TXT record: The peer didn't know the key we used
Encountered error adding TXT record: The peer didn't know the key we used
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
1 Like
8

Ok but clobbering your entire system to fix a small part of the domain validation process seems quite extreme and for many users would likely to result in missing configuration, however it seems your pretty technical so dig into the dns-rfc2136 plugin, find out how it works and go from there.

I'd suggest reviewing the credentials/config for the plugin: Welcome to certbot-dns-rfc2136’s documentation! — certbot-dns-rfc2136 0 documentation

4 Likes
9

Well, here's how the clobbering went:

  1. Goal: migrate from CentOS 7 to AlmaLinux since CentOS7 is now going end-of-life.
  2. options: attempt using in-place tools or rebuild server. The choice of rebuilding was made because A) I had to leap multiple versions and B) I wanted to get rid of decades of lint hiding in obscure corners of the system. Rebuilding was easy because I use provisioners.
  3. Jack in a blank drive to my build (non-production server).
  4. PXE-boot the build server. PXE runs kickstart which installed a basic OS (AlmaLinux 8.9)
  5. Run Ansible second-stage proviisioning to add stuff like the backup client, hosts file and so forth.
  6. Run Puppet to handle the more complicated configurations such as DHCP and Nginx virtual hosts.
  7. Realize to my horror I can't use AlmaLinux 8 because IBM yanked the Enterprise Storage repositories completely off-line. Meaning that I no longer could setup Ceph services and DNF/YUM wouldn't run until I disabled those repos. At that point, rather than nuke and restart, I did an in-place upgrade from AlmaLinux 8 to AlmaLinux 9. This is probably how I ended up with conflicts on the Nginx server packages.
  8. Everything is fat and happy, but my LE certs are expiring shortly. That's where I ran into the certbot problem.
  9. I couldn't reliably update the system as long as Nginx was upset so I removed nginx-core and nginx. The standard for package removal in RPMs is to retain config and data files in case you want to re-install. So I did, which ended the conflicts. Installing/updating an RPM causes the original config to be moved to an ".rpmsave" file so that you can compare your customized config to the standard one that comes with the package. In my case, it was sufficient to simply overwrite the rpm-supplied one with my rpmsave and all is happy with nginx.
  10. Since none of this helped, I simply made a copy of the entire /etc/letsencrypt directory subtree, and did an unistall/reinstall. I don't normally like this kind of voodoo, but since internal security signatures for entire OS had changed I thought there might be some internal effects on the letsencrypt files. I also regenerated the rndc key used by my DNS server for the same reason.
  11. Failing success, I restored my backup of /etc/letsencrpt.

So much for general OS stuff.

Originally, I was obtaining certs on a per-host basis but about 6 months ago I decided to move to wildcards. That was an adventure.

As I recall, you can process a cert request for a single domain using the ngnix plugin which petitions LetsEncrypt for an acme security token which it then makes a servable resource so that the certbot host can reliably determine that my server is, in fact my server by comparing its acme token with mine.

Wildcard certs cannot be provided this way, however. Instead, the token is injected as a DNS TXT record into my DNS server and a DNS inquiry is made from the certbot server. Should be the same process, essentially, but as I read it, what certbot sends me and what it gets back apparently don't agree. And the arbitator for that appears to be a third-party library that's invoked by the dns-rfc2136 plugin but isn't actually part of the plugin itself.

I suppose that with the right log settings, I could display the in-and-outbound tokens, but I'd been hoping that there was something simpler I could do.

10

Solved (I think)

on closer inspection it appears that my rndc secret was SHA-512, but the certbot credentials was supplying an SHA-256 secret. So a more accurate message would be "client's certbot secret does not match DNS rndc secret". I'm afraid "peer" doesn't make it very obvious which peer is at issue.

4 Likes
11

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.