I totally agree, the reputation that Let's Encrypt has created itself is proof that users are capable of learning to trust names like, well, Let's Encrypt's.
The way I see it, the people who obsessively inspect SSL certificates should know by now that a trusted name is a trusted name — regardless of all the other details. Think of it this way — Google has their own Intermediate Certificate, which is used to secure all of their websites, and even the most obsessive visitor will see that it is cross-signed by a trusted root authority, which means that a trusted company had to go through the trouble of vetting and doing the paperwork.
If someone ever says this to you, just pull up any website that makes use of a Let's Encrypt certificate and show them the lock in the address bar — this should be all they need as proof, and if it's not you can educate them.