How can I authenticate the ownership of the server without opening port 80 in letsencrypt using nginx?

You can't.
nginx is a web server - that can only serve HTTP/HTTPS request.
DNS servers serve DNS requests.

4 Likes