Anything above does not help to resolve this issue and it took too much time.
As expected, I had to disable firewalls and SSL everywhere, reissue certificates, reconfigure firewalls, SSL back every where. Now it works. But issue still requires answer, because it will occur in next 3 month when these certs will expire.
How to make it auto renew?
DNS as you can see CloudFlare.
Maybe edit cron to run before certs expire?
As it tries to challenge site with HTTP when there is only HTTPS.
Or just install plugin?
I do not want disable SSL for specific links eg for acme challenge.
Cloudflare still reports a 520 Origin Error for the insecure version of your website, which is where visitors are going to land by default, unless they manually type in https://. They won't get autoredirected to the secure version, as your browser probably does due to your HSTS setup.
This is also the same cause underlying your renewal failure, since Let's Encrypt also connects insecurely to do validation.
I don't think you have re-issued your certificates either - nothing has been logged in CT logs for your real domain.
Maybe with Cloudflare Argo Tunnel? But normal Cloudflare won't proxy ALPNs other than HTTP/2.
You need port 80: Best Practice - Keep Port 80 Open - Let's Encrypt . Especially if you want your preload HSTS directive to work - they require a redirect to be established to be included on the list.
If you've intentionally not responding on port 80, then have Cloudflare establish a page rule to do an automatic redirect to HTTPS.
Easier still, make a simple port 80 virtualhost which performs the required redirect to HTTPS.
You will still need to use the webroot approach - none of this will fix the Certbot/nginx bug preventing --nginx from succeeding.