You should never add the intermediates to your root trust store. The trust anchors are the ISRG and DST roots, and the intermediates are signed/cross-signed by them.
The DST root should already be present in any Debian distribution you are using, and therefore you shouldn’t run into trust issues. The intermediates are cross-signed by the DST root for the precise reason of ensuring compatibility.
I would guess that the hosts you are experiencing issues with are not properly bundling the intermediates along with their certificates.
Are you able to successfully fetch the following from the servers where you experience this issue?
$ curl https://helloworld.letsencrypt.org