Let’s Encrypt definitely does not use passphrases on any certificates it issues. Certificates are public documents after all.
Let’ Encrypt does not have access to your private key either. This means that this is either an issue with the method you use to generate the files you upload to Heroku, or an issue on Heroku’s end. If you could share more details about the software and/or commands you use to generate those files, we could probably figure out which of the two it is.