Oof, finally solved the problem.
If you export the ISRG Root X1 certificate from a working Windows 10 computer and import it from a non-working computer, the import won't work.
I downloaded the self-signed ISRG Root X1 .der file from Chain of Trust - Let's Encrypt and imported it and voila, it was able to access all sites with letsencrypt certificate without errors.
So my solution for Windows client machines would be to delete DST Root CA X3 certificate, download and import ISRG Root X1 certificate.
I still don't understand why this particular client wasn't served the new certificate from the server though.
3 Likes