I concur with regard to the use of dns_cloudflare_api_key and dns_cloudflare_email, but I don't understand where the earlier mentioned dns_cloudflare_api_token comes from then..?
With regard to debugging: if everything else fails, I'd personally resort to sniffing the entire HTTPS stream between Certbot and Cloudflare, which includes the actual contents somehow. Even if this would require a MitM-attack I set up myself. (Browsers can export the ephemeral TLS keys so WireShark can decode the stream, but I'm not sure if that's possible with Python HTTP(S) clients..)
I've run that test already and unfortunately it hasn't made any difference. Also I'm not sure what you mean by "timing"...? The script will execute in order and the certbot command will not be run until the .ini file as been written.
Either way it's unlikely to be the issue as the same code works when using the global API key instead.
"written" can be subject to interpretation.
Caching can play tricks with that.
And why write something down [that is already in your memory] only to then read that back?
Step 1: Honey, I'm going to the store to get some milk.
Step 2: Let me write that down.
Step 3: OK, let me see/read what I need to get from the store.
Unless your memory is failing you... and then how would you ever really know if it were?
Why take two those extra steps?
I mean, no it can't? At least not in this context. I don't know what to tell you, writing out to a file then processing that file is extremely common practice and while I have seen issues with it before they are typically when working with very large datasets or on remote datastores. Even in those exceptions the problem is considered to be the data/storage and is either fixed on that end or worked around.
Regardless, not something that applies in this scenario at all.
The dns plugin requires a ini file, it's not optional and cannot have the token passed in directly to the command (nor should it). That means the ini file needs to exist. Because this is a container I can either pass the file in directly and have it reference the secret variable there, or I can create it programmatically and eliminate a manual step from the process which is always preferable.
So the secret has to be passed in securely and it has to be written to a file before it can be used, which is what I'm doing. Far as I'm aware this is both standard and best practice, but if you know of a better method I would be very grateful to learn (no, that isn't sarcasm at all, if I'm doing it wrong I want people to tell me how to do it right).
Anyway, all of that aside this is one of the first things I tested and I've provided multiple tests showing that this is not the problem.
I've manually created the file to eliminate environment variables, which also eliminates the creation of the file and any potential caching issues.
It works just fine with the global API key, so the issue is 100% the way in which the token is being processed.
But just to put this to bed entirely here is another test I've run (and yes the token was immediately deleted after the test):
#!/bin/sh
# Create the authentication .ini
echo "Writing the ini file..."
echo "dns_cloudflare_api_token = $CLOUDFLARE_TOKEN" > /cloudflare.ini
chmod 600 /cloudflare.ini
# Make extremely sure the file has been correctly created
echo "Giving the ini file 10 seconds to chill out..."
sleep 10
echo "The ini file currently reads:" && cat /cloudflare.ini
# Run certbot
certbot certonly \
--dns-cloudflare \
--dns-cloudflare-credentials /cloudflare.ini \
--dns-cloudflare-propagation-seconds 60 \
--agree-tos \
--no-eff-email \
--staging \
-n \
-m $CLOUDFLARE_EMAIL \
-d $CLOUDFLARE_DOMAIN_LIST
And the output:
/opt/certbot # /init-certbot.sh
Writing the ini file...
Giving the ini file 10 seconds to chill out...
The ini file currently reads:
dns_cloudflare_api_token = LFxVfgurhKP-BeCsovHDPSeCTQGBgsIQN7-skBSr
Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for testtest.snkdev.au
Error determining zone_id: 6003 Invalid request headers. Please confirm that you have supplied valid Cloudflare API credentials. (Did you copy your entire API token/key? To use Cloudflare tokens, you'll need the python package cloudflare>=2.3.1. This certbot is running cloudflare 2.11.1)
Ask for help or search for solutions at https://community.letsencrypt.org. See the logfile /var/log/letsencrypt/letsencrypt.log or re-run Certbot with -v for more details.
So I definitely appreciate the input but that particular issue has been tested and confirmed to not be the problem .
.