I use DNS-01 auth for certbot renewal. I can't use the other methods requiring FTP service, as I don't wish to set it up on the GCP server.
From what I have read, the cert created with "--manual" cannot auto-renew b/c; certbot issues a new challenge for each renewal, then expects to find that challenge in the TXT record of the (sub) domain. The only solution I have found is to run the subdomain NS (Name Server) on my localhost where TXT records can be created and purged via script. This of course requires separate NS assignment for the acme subdomain.
MY QUESTION:
Is my above conclusion correct, or are there other/better ways to handle DNS-01 challenges? Should I have not created the initial certs with "--manual", perhaps I wold not run into this issue but I wanted a wildcard cert so this is probably the only way?
Thank you for your input on this question.
Regards
THE BASICS:
My domain: .aerv.us, aerv.us
Cert Creation Command: # certbot certonly --manual --preferred-challenges dns-01 --server https://acme-v02.api.letsencrypt.org/directory -d ".aerv.us" -d aerv.us
I ran this command: # certbot renew
It produced this output: ... unexpected error: Failed authorization procedure. aerv.us (dns-01): urn:ietf:params:acme:error:unauthorized :: The client lacks sufficient authorization :: No TXT record found at _acme-challenge.aerv.us, aerv.us (dns-01)...
web server: Lighttpd
OS: Debian Buster 4.19.208-1
Hosting: GCP
root shell: YES
control panel: NO
certbot --version: 0.31.0
Yes, you pretty much shouldn't ever use --manual except when testing something.
Ideally, there's a plugin for certbot for your DNS provider. This has certbot use your DNS provider's API to create the TXT record.
If your DNS provider doesn't have an API, or your security policy doesn't want to allow for your server to have DNS API credentials (as depending on the DNS provider, you may only be able to have API credentials with full access to edit the DNS zone rather than just the TXT record), then you probably want to look at a tool like acme-dns or agnos, which act as a special-purpose DNS server just for responding with the right TXT record.
It's possible to do what you want with your existing Geoscaling provider, using smart subdomains.
Add a "Smart subdomain" for the _acme-challenge. subdomain. (I'll use _acme-challenge.geoscaling.letsdebug.net. in the example here). Set it up like this, making sure to have the "extra info" checkbox selected:
if (!isset($extra_info))
{
$output[] = array("TXT", "No ACME challenge records were configured.");
return;
}
$records = explode(" ", $extra_info);
foreach ($records as $record)
{
$output[] = array("TXT", $record);
}
Next, you need the Certbot hook which will send the ACME challenges to the smart subdomains functionality. Here it is as a Python 3 script (make sure to change credentials):
#!/usr/bin/env python3
import json
import sys
import xmlrpc.client
from os import getenv
from time import sleep
GEOSCALING_USERNAME = "USER_HERE"
GEOSCALING_PASSWORD = "PASSWORD_HERE"
#######
STATE_FILE = "/tmp/geoscaling-acme-challenges.json"
if len(sys.argv) != 2:
print('Usage: update_acme.py [add/delete]')
exit(1)
act = sys.argv[1]
payload = getenv("CERTBOT_VALIDATION")
# Since we can't readily query the existing records from the API, we'll just
# keep the list of records we're managing in a local temporary file.
all_records = []
try:
with open(STATE_FILE, 'r') as f:
all_records = set(json.loads(f.read()))
except:
# File might not exist or be garbage. Whatever, ignore.
pass
# Make the modifications we need
if act == "add":
all_records.add(payload)
elif act == "delete":
all_records.remove(payload)
# Save the records to the temporary file again.
with open(STATE_FILE, 'w+') as f:
f.write(json.dumps(list(all_records)))
# Update the geoscaling extra info with the full list of records to render
with xmlrpc.client.ServerProxy("http://api.geoscaling.com/dns2/xml-rpc/") as proxy:
assert proxy.geoscaling.extra_info(GEOSCALING_USERNAME,
GEOSCALING_PASSWORD,
" ".join(all_records)) == 1
if getenv("CERTBOT_REMAINING_CHALLENGES") == "0" and act == "add":
sleep(15)
Now, try use it with Certbot:
$ sudo certbot certonly -d "geoscaling.letsdebug.net" -d "*.geoscaling.letsdebug.net" \
--manual --manual-auth-hook "/etc/letsencrypt/geoscaling.py add" \
--manual-cleanup-hook "/etc/letsencrypt/geoscaling.py delete" \
--preferred-challenges dns --dry-run
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Simulating a certificate request for geoscaling.letsdebug.net and *.geoscaling.letsdebug.net
The dry run was successful.
Thank you for that great answer. If you don't mind, could you clarify a few points for me?
#1
you probably want to look at a tool like acme-dns or agnos
These solutions require the real NS to point the subdomain (_acme-challenge.example.com) records to localhost, do they not? If that's the case, my idea of using a local NS (Unbound NSD) for the subdomain would be on par?
#2
you pretty much shouldn't ever use --manual except when testing something
I most likely missed it, but is there any other way to issue wildcard cert with DNS-01?
#3
Ideally, there's a plugin for certbot for your DNS provider.
Turns out, Geoscaling does indeed have DNS API, but as you stated I would prefer not to expose the entire DNS account management to a internet exposed server.