I’m definitely not the most experienced here with the setup you’ve described, but I’ve been doing some research and have a couple things to offer that might be a start at the moment.
The certbot version packaged with your software is really outdated (0.40.0 vs 1.7.0):
Ubuntu would need to upgrade their python3-cloudflare package to 2.3.1 or newer, when support for API Tokens was added. Currently packaged version is 2.1.0-0. OS packages typically take quite a long time to receive updates, so if you’re really dead set on using API tokens, consider an alternative installation method.
(Ubuntu 20.10 will have support when it comes out, but I wouldn’t hold my breath for 20.04 receiving that update).
As @griffin mentions, if you install Certbot and its DNS plugins from snaps, you will get the latest and greatest, including support for Cloudflare API tokens.
You can also look at other ACME clients which support Cloudflare’s API tokens, acme.sh for instance.
I see, fair enough. I think it's indeed the case that snaps probably rely on systemd right now.
Could you share which systemd-less flavor of Ubuntu you are using? Or if you have just replaced PID1 on your own? That could be useful info for the Certbot team.
Ah, I somehow entirely missed that you are using containers. Yes, snaps won't work there.
Since it is a container where everything is pinned, you could consider just using a Python virtualenv and installing Certbot and its DNS plugins into it using pip.
If the container does nothing but Certbot, there's also Docker.