I managed to get it working.
On the IPv6 host, check to see if the file exists, if not reverse-proxy to the IPv4 host.
On the IPv4 host use certbot certonly --webroot and make sure your webserver serves up /.well-known/acme-challenge from the --webroot-path you give certbot.
You cannot use the nginx plugin, because it creates a temporary config that will listen on the IPv6 certbot wants to check.