The E5 is an intermediate ECDSA cert. Let's Encrypt will randomly issue your ECDSA leaf using E5 or E6. What you are describing is what is called "pinning" of an intermediate and that is not good practice.
The chains are described here: Chains of Trust - Let's Encrypt
Are you willing to post the domain name you are having problems with? We can probably give a more precise explanation.