Should I add both DST Root CA X3 and ISRG Root X1 intermediates?

Did you investigate more that error? It means that the client can't check if the certificate have been revoked. To check it for a let's encrypt certificate, the client need to be able to fetch OCSP. If your client failed to fetch it, it may decide it's a fatal error. You can force your server to send OCSP information to avoid that using OCSP stapling.