According to https://gethttpsforfree.com, when registering you need to provide an email and public key - I’ve got the email, and my server has tons of keys on it in /etc/letsencrypt, but I’m uncertain which pair I need to back up in order to not lose access to the account, and to use tools such as this.
You could look through the community here. The private_key.json is basically an expanded version of the account private key data in JSON format, but not directly usable in things like OpenSSL without converting to PEM.