You should be seeing requests from at least 3 different IPs, and the requests should all be in random case. If they're not in random case, then Let's Encrypt's Unbound server has already tried a few times and not received a response and is trying to fall back to not using case randomization.
Your problem reminds me of this thread from a year ago, where there was some sort of packet loss between Let's Encrypt's servers and the authoritative DNS servers, leading to unbound retrying but enough packets were lost to sometimes cause a SERVFAIL.