Osiris has it exactly right regarding the certificate signing requests (CSRs), which is what I mentioned above that the ACME website client that I authored myself uses to get you a certificate. The whole idea behind PKI is that you don't and shouldn't trust any third-party entity (including an ACME client) with the integrity of your certificates. From reading your response, I'm doing my best to conservatively gauge your experience, so I apologize if some of what I'm about to say might be known already to you.
When generating a CSR, it is customary to generate a new public-private key pair on the server where the certificate is intended to be used. This secures the private key by minimizing operations performed on it and thus minimizes its exposure. The public key is included in the CSR along with several pieces of information related to the entity to which the public key belongs. Once the readable content of the CSR is prepared, it is signed (encrypted) using the private key with that signature being appended to the bottom. Now, anyone possessing the public key (which is available in the CSR itself) can use it to decrypt the signature and compare the result to the content to ensure that the content has not been tampered with. The CSR contains no private information and is safe to pass to the world, including a potentially-wayward ACME client.
When the Let's Encrypt uses a CSR to generate a certificate, it results in updating various transparency logs (which Osiris and I have mentioned or alluded-to many times). Some services (e.g. crt.sh) provide a means to search those logs AND download PEM versions of the certificates. Let's Encrypt provides both your certificate and its intermediate certificate that was used to sign your certificate. That intermediate certificate can also be downloaded directly from the Let's Encrypt website. Thus, as long as Let's Encrypt has been provided adequate proof that the controller of the domain(s) identified on the CSR wishes to have a certificate issued using that CSR, Let's Encrypt will issue that certificate. From that point, the website ACME client can just be closed and the other means mentioned can be your (more humanly-trusted) sources to acquire what you need.
Q: What if the website ACME client deviously generates a new CSR for me (including generating a new public-private key pair)?
A: Unfortunately, Let's Encrypt has no way of filtering this and thus a certificate will be issued. However, if you attempt to install this dubious certificate for usage on your website(s), your authentic private key will (most likely with great probability) not match the dubious private key used to generate the dubious CSR. This will mean that the dubious certificate and authentic private key will be incompatible and you will know.
Fun fact: the RSA public exponent (e) used for almost every website on the internet is the same (65537). It is the modulus (n) that varies based on both the public exponent (e) and the private exponent (d).
Osiris is very good at playing devil's advocate and presenting additional/conflicting information to check what I've presented. It's one of the things I've learned to respect about him. So don't take our banter as anything other than healthy discussion. We're all comrades here.