Acme-client on OpenBSD 6.3


#1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
amarillo.colmena.biz

I ran this command:
amarillo# acme-client -vD amarillo.colmena.biz

It produced this output:
acme-client: /etc/ssl/private/server.key: domain key exists (not creating)
acme-client: https://acme-v01.api.letsencrypt.org/directory: directories
acme-client: acme-v01.api.letsencrypt.org: DNS: 2600:1400:d:18b::3a8e
acme-client: https://acme-v01.api.letsencrypt.org/acme/new-authz: req-auth: amarillo.colmena.biz
acme-client: /var/www/acme/EFC6F_PwQ4pT4oPC1W2dDAQ0mvL07mmN1Ns_vLGvIZo: created
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/TbGcCyupenXd8l7kQv4ydQqafqAgfuDoxjocn5zHpyo/4630079591: challenge
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/TbGcCyupenXd8l7kQv4ydQqafqAgfuDoxjocn5zHpyo/4630079591: status
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/TbGcCyupenXd8l7kQv4ydQqafqAgfuDoxjocn5zHpyo/4630079591: bad response
acme-client: transfer buffer: [{ “type”: “http-01”, “status”: “invalid”, “error”: { “type”: “urn:acme:error:dns”, “detail”: “DNS problem: SERVFAIL looking up CAA for amarillo.colmena.biz”, “status”: 400 }, “uri”: “https://acme-v01.api.letsencrypt.org/acme/challenge/TbGcCyupenXd8l7kQv4ydQqafqAgfuDoxjocn5zHpyo/4630079591”, “token”: “EFC6F_PwQ4pT4oPC1W2dDAQ0mvL07mmN1Ns_vLGvIZo”, “keyAuthorization”: “EFC6F_PwQ4pT4oPC1W2dDAQ0mvL07mmN1Ns_vLGvIZo.i_bjzw7xXT8oO8SS88hrtJLH4WtiB_XcvE1Xh3tIA0U”, “validationRecord”: [ { “url”: “http://amarillo.colmena.biz/.well-known/acme-challenge/EFC6F_PwQ4pT4oPC1W2dDAQ0mvL07mmN1Ns_vLGvIZo”, “hostname”:
“amarillo.colmena.biz”, “port”: “80”, “addressesResolved”: [ “149.28.44.210”,
“2001:19f0:5:3c33:5400:1ff:fe7b:a6c6” ], “addressUsed”: “2001:19f0:5:3c33:5400:1ff:fe7b:a6c6” } ] }] (871 bytes)
acme-client: bad exit: netproc(59094): 1

My web server is (include version):
OpenBSD 6.3 / httpd
The operating system my web server runs on is (include version):
OpenBSD 6.3
My hosting provider, if applicable, is:
Vultr.com
I can login to a root shell on my machine (yes or no, or I don’t know):
Yes.
I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No.


#2

Negative responses for amarillo.colmena.biz for record types it doesn’t have any records for have invalid DNSSEC in some way.

debug: NODATA response failed to prove NODATA status with NSEC/NSEC3

https://letsdebug.net/amarillo.colmena.biz/466

Try running “sudo pdnsutil rectify-zone colmena.biz”.

Or, for that matter, “sudo pdnsutil rectify-all-zones” in case other zones are invalid.

And ensure that it doesn’t happen again.

By the way, did you know colmena.biz has an odd SOA record that copies the nameserver and email address from the TLD? It’s probably more or less harmless, but still.

(Edit: Mistake.)


#3

Several issues here…

  1. I am not actually running my own DNS servers. I use those of my hosting provider. That’s what a “vanity” DNS server is. That can certainly be changed if people are unhappy about it.

  2. It would seem that surely the SOA record must be correct if the authority is willing to sign the record for DNSSEC. Doesn’t the SOA record have something to do with the delegation from the parent authority?

  3. The DNSSEC test at https://dnssec-analyzer.verisignlabs.com/colmena.biz claims that nothing is wrong with the DNSSEC for colmena.biz. Another test at http://dnsviz.net/d/colmena.biz/dnssec/ also indicates that the DNSSEC is valid for colmena.biz.

  4. The SERVFAIL issue is not a normal thing, and appears to me to be some kind of internal issue at whatever recursive DNS server is in use at letsencrypt, which was reported back to acme-client. Perhaps it is only triggered via IPv6 or under certain circumstances, or for some other reason not all users experience it.


#4

Looking up CAA records with a DNSSEC-validating resolver tends to tickle DNSSEC bugs that you may not see during normal browsing. See more details at https://letsencrypt.org/docs/caa/.

@mnordhoff, you mentioned pdnsutil. Do you have reason to believe the authoritative nameservers here are running PowerDNS? If so, they may need an upgrade, since PowerDNS has a known issue similar to this one.


#5

PowerDNS returns a servfail status if you set version to anonymous so it is possible they are using PowerDNS Authoritative Server.

$dig @ns4.firstfind.nl version.bind ch txt +norec                 
; <<>> DiG 9.11.1 <<>> @ns4.firstfind.nl version.bind ch txt +norec
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 60643
;; flags: qr aa; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;version.bind.                  CH      TXT

;; Query time: 12 msec
;; SERVER: 93.94.226.155#53(93.94.226.155)
;; WHEN: Mon May 14 22:19:02 CEST 2018
;; MSG SIZE  rcvd: 41

Cheers,
sahsanu


#6

Given the above, @justinacolmena I would recommend you contact your DNS provider and ask them to upgrade their PowerDNS install. You can link them to https://letsencrypt.org/docs/caa/ (in particular the CAA errors section) if it helps.


#7
[justina@blanco ~]$ dig @vanidad-1.colmena.biz version.bind ch txt +norec

; <<>> DiG 9.11.3-RedHat-9.11.3-6.fc28 <<>> @vanidad-1.colmena.biz version.bind ch txt +norec
; (2 servers found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 62242
;; flags: qr aa; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1680
;; QUESTION SECTION:
;version.bind.                  CH      TXT

;; ANSWER SECTION:
version.bind.           5       CH      TXT     "ChoopaDNS"

;; Query time: 100 msec
;; SERVER: 2001:19f0:ccd::2#53(2001:19f0:ccd::2)
;; WHEN: Mon May 14 23:36:41 GMT 2018
;; MSG SIZE  rcvd: 63

This looks like a valid response, not a SERVFAIL.


#8

PowerDNS uses DNSSEC signatures that are valid for 3 weeks, Thursday to Thursday. That’s usually my trick for checking. (And I run PowerDNS, so I just dig one of my own domains instead of checking a calendar to see what date Thursday is.) I don’t know of any other signers that commonly behave the same way.

colmena.biz. 38400 IN RRSIG A 13 2 38400 20180524000000 20180503000000 43819 colmena.biz. bxNwUq34XourXzODxpZbwH+ha+mikcXxz8mNHSPBW66rS3MAgH6NdQ1/ ISyn0WjaJjLyO3k0y+ibvcwwhnbbGQ==

Upgrading won’t necessarily help. The issue is that some methods of updating records automatically update the associated negative records and some don’t., requiring you to rectify. If you don’t, the negative records may no longer be valid. It’s more a design limitation than a bug. I think newer versions handle it automatically under more circumstances, but still not all.

It’s configurable. You can turn it off, customize it, or leave it at the default. I guess they’ve taken option 2, or have a more complicated DNS architecture.


#9

Some DNS providers that are unfamiliar with CAA initially reply to problem reports with “We do not support CAA records.” Your DNS provider does not need to specifically support CAA records; it only needs to reply with a NOERROR response for unknown query types (including CAA). Returning other opcodes, including NOTIMP, for unrecognized qtypes is a violation of RFC 1035, and needs to be fixed.

The servers do reply with NOERROR. I am am starting to think the problem is that I currently only have a CAA record for the bare domain “colmena.biz” – and that this is not automatically inherited for hosts or subdomains under that hierarchy. Every hostname apparently needs its own CAA record.


#10

Success.

amarillo# acme-client -vAD amarillo.colmena.biz
acme-client: /etc/ssl/private/server.key: domain key exists (not creating)
acme-client: /etc/acme/letsencrypt-privkey.pem: account key exists (not creating)
acme-client: https://acme-v02.api.letsencrypt.org/directory: directories
acme-client: acme-v02.api.letsencrypt.org: DNS: 2001:418:1446:2ba::3a8e
acme-client: https://acme-v02.api.letsencrypt.org/directory: bad CA paths
acme-client: transfer buffer: [{ "iPC4WrPA66o": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417", "keyChange": "https://acme-v02.api.letsencrypt.org/acme/key-change", "meta": { "caaIdentities": [ "letsencrypt.org" ], "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.2-November-15-2017.pdf", "website": "https://letsencrypt.org" }, "newAccount": "https://acme-v02.api.letsencrypt.org/acme/new-acct", "newNonce": "https://acme-v02.api.letsencrypt.org/acme/new-nonce", "newOrder": "https://acme-v02.api.letsencrypt.org/acme/new-order", "revokeCert": "https://acme-v02.api.letsencrypt.org/acme/revoke-cert" }] (658 bytes)
acme-client: bad exit: netproc(90862): 1
amarillo# vim /etc/acme-client.conf             
amarillo# acme-client -vD amarillo.colmena.biz  
acme-client: /etc/ssl/private/server.key: domain key exists (not creating)
acme-client: https://acme-v01.api.letsencrypt.org/directory: directories
acme-client: acme-v01.api.letsencrypt.org: DNS: 2001:418:1446:2ba::3a8e
acme-client: https://acme-v01.api.letsencrypt.org/acme/new-authz: req-auth: amarillo.colmena.biz
acme-client: /var/www/acme/bULjGPfvRhiiYFlVQE33iOIa3n0BcaonpmgSndn1Rk4: created
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/0g7j_0x5SIrgkiQUnVZBdlEVN-roNfjGtWkQJodPNFI/4646924170: challenge
acme-client: https://acme-v01.api.letsencrypt.org/acme/challenge/0g7j_0x5SIrgkiQUnVZBdlEVN-roNfjGtWkQJodPNFI/4646924170: status
acme-client: https://acme-v01.api.letsencrypt.org/acme/new-cert: certificate
acme-client: http://cert.int-x3.letsencrypt.org/: full chain
acme-client: cert.int-x3.letsencrypt.org: DNS: 2001:418:1446:290::ce0
acme-client: /etc/ssl/server.crt: created
acme-client: /etc/ssl/fullchain.pem: created
amarillo#

#11

That’s not accurate. I would say it like this: Let’s Encrypt needs to receive a valid response for each FQDN that is part of the tree climbed for CAA. For instance www.example.com, example.com, com. However, the “valid response” thing is tricky. It can be either a non-empty CAA response, or it can be an empty CAA response. However, if DNSSEC is present on the domain name, it must validate.

At any rate, the spirit of your comment is right: Your easiest workaround for now may be to simply add a CAA record on each of your hostnames.


#12

You have to ask them to fix their DNSSEC setup, then. :sweat:

The SOA DNS server name is only used for a few things. Depending on how the authoritative DNS servers are configured, they might not use it at all, or things might mostly work even if it’s invalid.

The SOA email address is only for human usage.

Neither of those tests queried the invalid records. And it can be difficult to make them do so. :slightly_frowning_face:

It’s not. Query any validating resolver for the name amarillo.colmera.biz and a record type that doesn’t exist. (Since you added a CAA record, try TXT or something.)

(Edit: I misspelled your domain in this post but not anywhere else.)


#13
[justina@blanco ~]$ dig amarillo.colmena.biz txt +dnssec

; <<>> DiG 9.11.3-RedHat-9.11.3-6.fc28 <<>> amarillo.colmena.biz txt +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 17206
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 4, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags: do; udp: 4096
;; QUESTION SECTION:
;amarillo.colmena.biz.          IN      TXT

;; AUTHORITY SECTION:
colmena.biz.            300     IN      SOA     a.gtld.biz. hostmaster.neustar.biz. 1526341930 10800 3600 604800 3600
colmena.biz.            300     IN      RRSIG   SOA 13 2 300 20180524000000 20180503000000 43819 colmena.biz. oSBpQhHdckUAUkLjA5RbipD7MpVcbizT/IXUED9y5y5tiGlByb2zPj+M nY8RGx0VzwvMjTz5VNn/mrUj/zkVrw==
colmena.biz.            300     IN      RRSIG   NSEC 13 2 3600 20180524000000 20180503000000 43819 colmena.biz. JdIcrw2btGSzLx8JZr0YjiMQ0xTRNCCoxaJroxoMGAVNCxYAOKKFi6N3 7KXixLkdl6XfmD7tBwdawarO9uJXSQ==
colmena.biz.            300     IN      NSEC    db.colmena.biz. A NS SOA MX TXT AAAA RRSIG NSEC DNSKEY CAA

;; Query time: 103 msec
;; SERVER: 192.168.44.1#53(192.168.44.1)
;; WHEN: Tue May 15 00:46:44 GMT 2018
;; MSG SIZE  rcvd: 365

See, now I remember that round-robin thing with DNSSEC, where each record in the zone is cryptographically chained somehow to the next (NSEC) – like looking up a word in the dictionary, and if it’s not in its place in alphabetical order, then it may be determined not to exist.


#14

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.