The browser would have to request the IP address. If you request https://google.com but get served a certificate for the IP address, you will receive an error because google.com is not a listed SAN
If software doesn't properly check SANs I would consider that a catastrophic vulnerability