I don't have a Windows system right now to double-check but I think you should have a privkey.pem file. Which I think is what you are looking for.
If that does not help, what files are in the folder with the .pem files?
Here is certbot's doc about the files created. It doesn't say anything specific about Windows but I assume it's the same as Linux (except for folder names)
Thanks for the responses but the application i need to upload the private key to is expecting a .key file. Do I need to change the format of the privkey.pem file?