I’m not entirely sure what the actual concern is here. You don’t give anyone access to your server. All you’re doing is running software that automatically requests new certificates when they are needed.
There is some automation to install those new certificates when they are received, but that shouldn’t be any ADDITIONAL concern. By definition you have to trust all trusted certificate authorities. Any single one of them can make certificates for any domain. Now that the intermediate certificates are cross signed, this already goes for LE as well. There is lots wrong with a system that inherently trusts a large list of CA’s and LE is not going to fix that. But it will make it much easier to get domain validation and encryption running on all servers.
“Nothing has to be validated” seems like a very generic claim. This is not true. When a certificate is requested for a certain domain, LE asks the requester to put a file on that domain for verification of control of the domain. This is done by the software running on your server. That way LE knows you are requesting a cert for a domain you have control over and the new cert is made and issued. The software on your server than updates the certs on your server with the new one.
In short, all it does is automate the steps you would have done manually.
Maybe the concern is that a perfectly good cert is automatically replaced with one that somehow doesn’t work? I don’t really know. And this seems highly unlikely.
And if you don’t trust LE… well, tough, your browser now does. So unless you dig in there and remove CA’s from that trusted list. Any certificate issued for any domain by LE will be trusted by your browser and everyone elses.