Unless your server responds to multiple IP addresses, setting the IP won't make much of a difference.
More likely, this may be the only virtual host with :443, so it becomes the "default" and any 443 request on your IP will be handled by this virtual host - despite any servername/serveralias mismatch.