Trouble renewing a cert with Joker DNS-01 challenge

ieure · February 25, 2023, 10:52pm

I think LE is unable to communicate with my DNS provider. I'm receiving a similar SERVFAIL error with both lego and certbot with the certbot-dns-joker plugin.

I'm able to update my domain with curl and see changes reflected immediately using dig against a.ns.joker.com / b.ns.joker.com. So I think there may be a communication issue between LE and Joker's nameservers.

My certs have been failing to renew and are already expired.

Any suggestions would be appreciated.

My domain is: atomized.org

I ran this command:

certbot certonly -vvvvvvv --authenticator dns-joker --dns-joker-credentials /secrets/certbot-joker.ini -d lab.pins.atomized.org --dry-run

It produced this output:

Root logging level set at -40
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requested authenticator dns-joker and installer None
Single candidate plugin: * dns-joker
Description: Obtain certificates using a DNS TXT record (if you are using Joker for DNS).
Interfaces: Authenticator, Plugin
Entry point: dns-joker = certbot_dns_joker.dns_joker:Authenticator
Initialized: <certbot_dns_joker.dns_joker.Authenticator object at 0x7f24ffe62f70>
Prep: True
Selected authenticator <certbot_dns_joker.dns_joker.Authenticator object at 0x7f24ffe62f70> and installer None
Plugins selected: Authenticator dns-joker, Installer None
Picked account: <Account(RegistrationResource(body=Registration(key=None, contact=(), agreement=None, status=None, terms_of_service_agreed=None, only_return_existing=None, external_account_binding=None), uri='https://acme-staging-v02.api.letsencrypt.org/acme/acct/90184424', new_authzr_uri=None, terms_of_service=None), dcc47d1c6b0b8ab4b4f8aa29910a3da9, Meta(creation_dt=datetime.datetime(2023, 2, 25, 22, 23, tzinfo=<UTC>), creation_host='d31b62dad6da', register_to_eff=None))>
Sending GET request to https://acme-staging-v02.api.letsencrypt.org/directory.
Starting new HTTPS connection (1): acme-staging-v02.api.letsencrypt.org:443
https://acme-staging-v02.api.letsencrypt.org:443 "GET /directory HTTP/1.1" 200 830
Received response:
HTTP 200
Server: nginx
Date: Sat, 25 Feb 2023 22:34:24 GMT
Content-Type: application/json
Content-Length: 830
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "6RF4SIj9URQ": "https://community.letsencrypt.org/t/adding-random-entries-to-the-directory/33417",
  "keyChange": "https://acme-staging-v02.api.letsencrypt.org/acme/key-change",
  "meta": {
    "caaIdentities": [
      "letsencrypt.org"
    ],
    "termsOfService": "https://letsencrypt.org/documents/LE-SA-v1.3-September-21-2022.pdf",
    "website": "https://letsencrypt.org/docs/staging-environment/"
  },
  "newAccount": "https://acme-staging-v02.api.letsencrypt.org/acme/new-acct",
  "newNonce": "https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce",
  "newOrder": "https://acme-staging-v02.api.letsencrypt.org/acme/new-order",
  "renewalInfo": "https://acme-staging-v02.api.letsencrypt.org/get/draft-ietf-acme-ari-00/renewalInfo/",
  "revokeCert": "https://acme-staging-v02.api.letsencrypt.org/acme/revoke-cert"
}
Notifying user: Simulating a certificate request for lab.pins.atomized.org
Simulating a certificate request for lab.pins.atomized.org
Requesting fresh nonce
Sending HEAD request to https://acme-staging-v02.api.letsencrypt.org/acme/new-nonce.
https://acme-staging-v02.api.letsencrypt.org:443 "HEAD /acme/new-nonce HTTP/1.1" 200 0
Received response:
HTTP 200
Server: nginx
Date: Sat, 25 Feb 2023 22:34:24 GMT
Connection: keep-alive
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: B37CpOV1BlQZ8O_rHK4eRp7Ali2jLXgnCkNK_efezgyS3JE
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800


Storing nonce: B37CpOV1BlQZ8O_rHK4eRp7Ali2jLXgnCkNK_efezgyS3JE
JWS payload:
b'{\n  "identifiers": [\n    {\n      "type": "dns",\n      "value": "lab.pins.atomized.org"\n    }\n  ]\n}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/new-order:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC85MDE4NDQyNCIsICJub25jZSI6ICJCMzdDcE9WMUJsUVo4T19ySEs0ZVJwN0FsaTJqTFhnbkNrTktfZWZlemd5UzNKRSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9uZXctb3JkZXIifQ",
  "signature": "glJsYyj72RKmYGjFqRK2Pd-4okC-nDyhV0zBnhVILqc3pNDnoDjjLTczoCN-Ml9uKiU-9H_Xg3R4kIifNO0Lh7e3xrtM_3_W7Xkr4NIXevD2_Ett-EWVbWIiYHJdJ8pc6KZgzUElYwJpawTuovfaezWeh9gEEr6QoyRC56OfwTHwR0xobmxDEWi_bmDn---FL8bajjz7IJb_IuW6rVM3PC7cpZEfzvYDt2iDL5YMZgoDsi_Q_hTgoIsP5JTI_nlOPUHhAc3a0Su-JgY9M47FKA1jeuJBO8phcPNFCR7IM-LtnAueaeVf8FSIvcU2JmAsAGSWLflkiqrzn3ujOm6NaQ",
  "payload": "ewogICJpZGVudGlmaWVycyI6IFsKICAgIHsKICAgICAgInR5cGUiOiAiZG5zIiwKICAgICAgInZhbHVlIjogImxhYi5waW5zLmF0b21pemVkLm9yZyIKICAgIH0KICBdCn0"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/new-order HTTP/1.1" 201 357
Received response:
HTTP 201
Server: nginx
Date: Sat, 25 Feb 2023 22:34:25 GMT
Content-Type: application/json
Content-Length: 357
Connection: keep-alive
Boulder-Requester: 90184424
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/order/90184424/7434924294
Replay-Nonce: BEB9zPu8Xs87yOJAlEnUE4tEfBW-WqEq6pb7I81FqlP3k4E
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "status": "pending",
  "expires": "2023-03-04T22:34:25Z",
  "identifiers": [
    {
      "type": "dns",
      "value": "lab.pins.atomized.org"
    }
  ],
  "authorizations": [
    "https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5517538894"
  ],
  "finalize": "https://acme-staging-v02.api.letsencrypt.org/acme/finalize/90184424/7434924294"
}
Storing nonce: BEB9zPu8Xs87yOJAlEnUE4tEfBW-WqEq6pb7I81FqlP3k4E
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5517538894:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC85MDE4NDQyNCIsICJub25jZSI6ICJCRUI5elB1OFhzODd5T0pBbEVuVUU0dEVmQlctV3FFcTZwYjdJODFGcWxQM2s0RSIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My81NTE3NTM4ODk0In0",
  "signature": "j5brrXvskFMQ-lW-CsqcajmTl2wVC9sSNAoRJC8buHUg_jJsUKaADFgxgfY7_GSlTKr6M33PQOlDiUo2vX-wCAXPRAJd85U0XFWnjWQNhlVC-HRUG9Ko8XFNgRfOzBbF6b77ayqIr370ID8UB3VOiCAkn9GUD6NwAfgdDLu6koiQO1LPbncJsYXLtz63RGgnlptsERDCmB03Bfzcd4qH61himXKtEzpvtzJApCzKGjXxUF31WJkE82OGYvP33q746yt4uaZ_M_CjOtyaN7A-7gzlVurJkZVAc9WK74sa504aUPmbmC-4F4P-6RQRmSpPn1FZcVuO2gFvpLyeajQqew",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/5517538894 HTTP/1.1" 200 823
Received response:
HTTP 200
Server: nginx
Date: Sat, 25 Feb 2023 22:34:25 GMT
Content-Type: application/json
Content-Length: 823
Connection: keep-alive
Boulder-Requester: 90184424
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: B37CPdiERXSfWazkjCm0ZCF2xR-Ueh-9fbajlT1EmsHlKyg
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "lab.pins.atomized.org"
  },
  "status": "pending",
  "expires": "2023-03-04T22:34:25Z",
  "challenges": [
    {
      "type": "http-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/N4l1Hg",
      "token": "vysyqVIuHh19JN23K1s_xK-_1K7Gx9B_7wHm3iiGJZ8"
    },
    {
      "type": "dns-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/XhFF8g",
      "token": "vysyqVIuHh19JN23K1s_xK-_1K7Gx9B_7wHm3iiGJZ8"
    },
    {
      "type": "tls-alpn-01",
      "status": "pending",
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/wXtCdQ",
      "token": "vysyqVIuHh19JN23K1s_xK-_1K7Gx9B_7wHm3iiGJZ8"
    }
  ]
}
Storing nonce: B37CPdiERXSfWazkjCm0ZCF2xR-Ueh-9fbajlT1EmsHlKyg
Performing the following challenges:
dns-01 challenge for lab.pins.atomized.org
Unsafe permissions on credentials configuration file: /secrets/certbot-joker.ini
Starting new HTTPS connection (1): svc.joker.com:443
https://svc.joker.com:443 "POST /nic/replace HTTP/1.1" 200 None
Notifying user: Waiting 120 seconds for DNS changes to propagate
Waiting 120 seconds for DNS changes to propagate
JWS payload:
b'{}'
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/XhFF8g:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC85MDE4NDQyNCIsICJub25jZSI6ICJCMzdDUGRpRVJYU2ZXYXprakNtMFpDRjJ4Ui1VZWgtOWZiYWpsVDFFbXNIbEt5ZyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9jaGFsbC12My81NTE3NTM4ODk0L1hoRkY4ZyJ9",
  "signature": "Xm2XhikcJ0vZ_EN5YtFCvPIWgSBP4ogG5_FNUCM6u5zyrS1WyPPwB2BhcFdx3LhYgMlJNAkyc0GxL5iyXYzQ5skTuzInnbhhTUjkqAk07ST2eh1ec_xrW9H6ZWPtYdU17YhnqgIukutpyGc_zp_cb95RTl6o7zPZI8MUcqZBuSfhNEFOiNkTxve0SstAbgQtgM3neaKsozCky82p9bFPrC48tdi3vy52yqmzLS7oX1Nj_9Z-PuRLCa26zc1mYm3WUGN6ESL7ZPGeiz-PRYCw_98VDndgUXe-t9HPTjAahEZEIO4w1HGZz1On3TXwY3Yii1Tg_Ls58JMJqjYx48ccIA",
  "payload": "e30"
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/chall-v3/5517538894/XhFF8g HTTP/1.1" 200 192
Received response:
HTTP 200
Server: nginx
Date: Sat, 25 Feb 2023 22:36:26 GMT
Content-Type: application/json
Content-Length: 192
Connection: keep-alive
Boulder-Requester: 90184424
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index", <https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5517538894>;rel="up"
Location: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/XhFF8g
Replay-Nonce: BEB9VTiEAruKDpAVfokFHWYUluyfs7F7Oqn1cp5dpQA75zs
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "type": "dns-01",
  "status": "pending",
  "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/XhFF8g",
  "token": "vysyqVIuHh19JN23K1s_xK-_1K7Gx9B_7wHm3iiGJZ8"
}
Storing nonce: BEB9VTiEAruKDpAVfokFHWYUluyfs7F7Oqn1cp5dpQA75zs
Waiting for verification...
JWS payload:
b''
Sending POST request to https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5517538894:
{
  "protected": "eyJhbGciOiAiUlMyNTYiLCAia2lkIjogImh0dHBzOi8vYWNtZS1zdGFnaW5nLXYwMi5hcGkubGV0c2VuY3J5cHQub3JnL2FjbWUvYWNjdC85MDE4NDQyNCIsICJub25jZSI6ICJCRUI5VlRpRUFydUtEcEFWZm9rRkhXWVVsdXlmczdGN09xbjFjcDVkcFFBNzV6cyIsICJ1cmwiOiAiaHR0cHM6Ly9hY21lLXN0YWdpbmctdjAyLmFwaS5sZXRzZW5jcnlwdC5vcmcvYWNtZS9hdXRoei12My81NTE3NTM4ODk0In0",
  "signature": "iTDsUOIPcMiO4It1QgTsJ7tC86p7lOvR7xYFMkl10lcUHSD9kUfKEGRijri8kiZbQkZaLSRyqFAtT49G1FS6-cqK-H09eQ8fsETCd8lBbeDHoBaBSApc81ocQ9qrrvgsXrH16k-j7AplbmdrEVL5Ns2twgaQVBfGMLWlGYS7LzSdcI-sjsJPoZKKnO9-st0BRMjRCHrDTRQevSIqmDJ3JpFJ6nfNW2ClBLum9Q5wXKlRXDiNaXtZ4P-SLnenORJwTlbI777ve376KJq-LfieD8oRekNwQgwX3S3SQZ7uVQf8eszK7i3_35DWJ2yofdVf0ymYVuEe1WBVsqG22dF9Cw",
  "payload": ""
}
https://acme-staging-v02.api.letsencrypt.org:443 "POST /acme/authz-v3/5517538894 HTTP/1.1" 200 671
Received response:
HTTP 200
Server: nginx
Date: Sat, 25 Feb 2023 22:36:27 GMT
Content-Type: application/json
Content-Length: 671
Connection: keep-alive
Boulder-Requester: 90184424
Cache-Control: public, max-age=0, no-cache
Link: <https://acme-staging-v02.api.letsencrypt.org/directory>;rel="index"
Replay-Nonce: A272HLl1RwWrNFCpXsThtSa4o_TRtuaLYBvUOkzC5FyU3T8
X-Frame-Options: DENY
Strict-Transport-Security: max-age=604800

{
  "identifier": {
    "type": "dns",
    "value": "lab.pins.atomized.org"
  },
  "status": "invalid",
  "expires": "2023-03-04T22:34:25Z",
  "challenges": [
    {
      "type": "dns-01",
      "status": "invalid",
      "error": {
        "type": "urn:ietf:params:acme:error:dns",
        "detail": "DNS problem: SERVFAIL looking up TXT for _acme-challenge.lab.pins.atomized.org - the domain's nameservers may be malfunctioning",
        "status": 400
      },
      "url": "https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5517538894/XhFF8g",
      "token": "vysyqVIuHh19JN23K1s_xK-_1K7Gx9B_7wHm3iiGJZ8",
      "validated": "2023-02-25T22:36:26Z"
    }
  ]
}
Storing nonce: A272HLl1RwWrNFCpXsThtSa4o_TRtuaLYBvUOkzC5FyU3T8
Challenge failed for domain lab.pins.atomized.org
dns-01 challenge for lab.pins.atomized.org
Notifying user: 
Certbot failed to authenticate some domains (authenticator: dns-joker). The Certificate Authority reported these problems:
  Domain: lab.pins.atomized.org
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for _acme-challenge.lab.pins.atomized.org - the domain's nameservers may be malfunctioning

Hint: The Certificate Authority failed to verify the DNS TXT records created by --dns-joker. Ensure the above domains are hosted by this DNS provider, or try increasing --dns-joker-propagation-seconds (currently 120 seconds).


Certbot failed to authenticate some domains (authenticator: dns-joker). The Certificate Authority reported these problems:
  Domain: lab.pins.atomized.org
  Type:   dns
  Detail: DNS problem: SERVFAIL looking up TXT for _acme-challenge.lab.pins.atomized.org - the domain's nameservers may be malfunctioning

Hint: The Certificate Authority failed to verify the DNS TXT records created by --dns-joker. Ensure the above domains are hosted by this DNS provider, or try increasing --dns-joker-propagation-seconds (currently 120 seconds).

Encountered exception:
Traceback (most recent call last):
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.

Calling registered functions
Cleaning up challenges
Starting new HTTPS connection (1): svc.joker.com:443
https://svc.joker.com:443 "POST /nic/replace HTTP/1.1" 200 None
Exiting abnormally:
Traceback (most recent call last):
  File "/usr/local/bin/certbot", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.9/dist-packages/certbot/main.py", line 19, in main
    return internal_main.main(cli_args)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/main.py", line 1862, in main
    return config.func(config, plugins)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/main.py", line 1595, in certonly
    lineage = _get_and_save_cert(le_client, config, domains, certname, lineage)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/main.py", line 140, in _get_and_save_cert
    lineage = le_client.obtain_and_enroll_certificate(domains, certname)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/client.py", line 516, in obtain_and_enroll_certificate
    cert, chain, key, _ = self.obtain_certificate(domains)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/client.py", line 428, in obtain_certificate
    orderr = self._get_order_and_authorizations(csr.data, self.config.allow_subset_of_names)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/client.py", line 496, in _get_order_and_authorizations
    authzr = self.auth_handler.handle_authorizations(orderr, self.config, best_effort)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/auth_handler.py", line 108, in handle_authorizations
    self._poll_authorizations(authzrs, max_retries, max_time_mins, best_effort)
  File "/usr/local/lib/python3.9/dist-packages/certbot/_internal/auth_handler.py", line 212, in _poll_authorizations
    raise errors.AuthorizationError('Some challenges have failed.')
certbot.errors.AuthorizationError: Some challenges have failed.
Some challenges have failed.
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.

My web server is (include version): n/a

The operating system my web server runs on is (include version): Debian Bullseye

My hosting provider, if applicable, is: Joker

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): 2.3.0

danb35 · February 25, 2023, 11:09pm

Not exactly; the nameserver is responding, but responding with an error:

I'm getting the same thing when I query your nameserver:

dan@Dan-MBP-2019  ~  dig @a.ns.joker.com txt _acme-challenge.lab.pins.atomized.org

; <<>> DiG 9.10.6 <<>> @a.ns.joker.com txt _acme-challenge.lab.pins.atomized.org
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: SERVFAIL, id: 16085
;; flags: qr rd ra; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;_acme-challenge.lab.pins.atomized.org. IN TXT

;; Query time: 4471 msec
;; SERVER: 23.88.49.189#53(23.88.49.189)
;; WHEN: Sat Feb 25 18:09:42 EST 2023
;; MSG SIZE  rcvd: 66

There's something wrong with joker.com's nameservers or their configuration.

MikeMcQ · February 25, 2023, 11:16pm

Agree with Dan in this thread too

See the dnsviz report for some, um, interesting error messages (link here)

_az · February 25, 2023, 11:16pm

It looks like that the pins zone is subdelegated:

pins.atomized.org.      86400   IN      NS      top-int.atomized.org.

But that nameserver is in a private IP space:

top-int.atomized.org.   86400   IN      A       192.168.128.1

so no public resolver would not be able to resolve the domain.

I am not sure what the DNS specifications say about putting an _acme-challenge.pins.atomized.org. record in the parent nameserver at the same time as it is being subdelegated. But my bet would be that this is the cause of your troubles.

ieure · February 25, 2023, 11:17pm

Thanks for the suggestions! I agree, that erroneous NS record is the source of my issues. I'll delete it and try again.

system · March 27, 2023, 11:18pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.