There should be 5 connections: 1 from the primary and 4 from secondary (remote) validation points from all over the world (I believe 3 US and 2 somewhere else entirely I believe). See:
If some validation points work, but not others, this usually is a sign there is some kind of geoblocking in play. Although you say that cannot be the case.. Maybe network issues then? I dunno.. LE uses AWS currently for their remote validations.