Account Key rotation best practice?

Seeing the discussion about the challenges of rotating certificate private keys in the Certbot support for ECDSA certificates thread brought to mind a question I've been meaning to ask:

What's the best practice for rotating account keys?

I don't see a lot of guidance around it, and I don't think many clients handle doing it. The main thing that concerns me is this paragraph in the ACME RFC 8885:

Compromise of the private key of an account key pair has more serious consequences than compromise of a private key corresponding to a certificate. While the compromise of a certificate key pair allows the attacker to impersonate the entities named in the certificate for the lifetime of the certificate, the compromise of an account key pair allows the attacker to take full control of the victim's ACME account and take any action that the legitimate account holder could take within the scope of ACME:

I'm not yet completely convinced of the first sentence there, but I know that I know just enough about crypto to be dangerous so I want to take the experts who wrote this at their word on it.

But if the account private key is at least as important as the certificate private key, as the RFC claims, should I be rotating it every 2 months along with my certificates? Or at least once a year or so? Or should I just be leaving my account key alone forever unless I have reason to believe it might be compromised?

1 Like

Compromise of the private key of an account key pair has more serious consequences than compromise of a private key corresponding to a certificate.

I completely disagree with that first sentence.

I, for one, would much prefer that someone be able to revoke my certificates (thus temporarily shutting down my websites) and issue false certificates (thus temporarily being able to spoof my websites) than be able to decrypt my traffic (given the right circumstances). Getting a new ACME account is very easy. Recovering from a security breach, not so much.

Otherwise, why would website ACME clients, like mine, be allowed to operate at all?

1 Like

Well, it's good to know that it's not just me that found that sentence suspect. The RFC goes on to list 4 specific problems with an account key compromise:

  1. Issuing certificates using existing authorizations

Sure, that's a problem, and lets you compromise all domains validated recently instead of just whichever certificate had a private key compromise. But if you only have a couple domains under the account, it seems about the same. I can see how this might be an issue for some large hosting provider that gets all their customers under one account key, though. (And this is in theory quickly detectable through Certificate Transparency, right?)

  1. Revoking existing certificates

So, you can revoke all certificates under the account rather than just the compromised cert. And sure, revoking a server's certificate and DoS'ing it would be bad, but I don't see how it'd be worse than compromising the certificate key directly. (Again, for a large hosting provider that has a bunch of certificates under their account key, I could see it being a big hassle. But still I'm not seeing how it'd be "more serious consequences" than compromising the cert keys.)

  1. Accessing and changing account information (e.g., contacts)

So you can redirect the expiration emails elsewhere? That's all it'd do for Let's Encrypt, right? Maybe this is a bigger deal for CAs other than Let's Encrypt, if you could use the ACME protocol to take over a whole "real" account with payments and such?

  1. Changing the account key pair for the account, locking out the legitimate account holder

Who could just make a new account with new authorizations/orders/certificates if all else fails, right? Annoying and problematic, sure, but I don't think it's "more serious".


I think that the guidance in the RFC may be trying to cover all the bases, especially if CAs start using the account for more or for accounts that have a lot of certificates. But again, I'm no expert cryptographer, and there might be more problems I'm not seeing.

But maybe the "best practice" here depends on if you have just a handful of certs (for yourself) or a ton (for your customer), and on what exactly the CA you're using does with the account contact info?

1 Like

You pose excellent questions. Typically the contact email is only used for notification of issues or expiration notices. I could see how denying access to that information (via changing the contact) could pose some risks. Locking out the legitimate account holder could be annoying, but it takes seconds to generate a new account. For a particularly large and broad deployment, I could see how the downtime and headaches involved could be an issue. Compared to a breach though...

Handling a compromised account key:

  1. Warn people of potential downtime.
  2. Fix the leak.
  3. Create a new ACME account.
  4. Reissue all involved certificates.
  5. Warn people to be on the lookout for fake websites.

Handling a compromised certificate key:

  1. Panic.
  2. Take any related devices offline.
  3. Revoke said certificate.
  4. Reissue said certificate.
  5. Advise anyone involved to change their credentials.
  6. Hold a press conference announcing the breach.
  7. Prepare for the fallout.
  8. Deal with months (or even years) of fallout.

Yes, PFS should save you from the latter and possibly avoid 6 through 8 for the morally bankrupt.

Very good question indeed.

Don't forget step 0.: unregister the account in question!

1 Like

I would assume that the first thing a knowledgeable attacker would do upon compromising an account key would be rotating it to a new one that they solely control. If you still have the account key, rather than deactivating the account you could just rotate the key to a new one instead. (Is there anything one can do with a no-longer-used account key once a key rotation happens?)

I guess your actions could depend on whether compromise was something like "hacker logged into server and downloaded /etc/letsencrypt" or something like "the key was accidentally posted publicly somewhere for a moment", and if you could tell what actions had been taken by an attacker on the ACME account.

1 Like

The account key rotation I'm referring to is the "Key Change" command, section 7.3.5 of the RFC. Works fine on Let's Encrypt when I was playing around with it. I just don't know if there's some recommendation on how often one should do it, or if it's just "safer" in some sense to never rotate unless there's a key compromise.

As most clients don't seem to have implemented it, I'm guessing it's not that important, but I find it odd when the standard advice seems to be to rotate certificate keys and the RFC says that account keys are even more sensitive (though as we're discussing that may be subjective).

2 Likes

Hmm, didn't know ACME had an account key rotation. I assumed this thread was about getting a whole new account. But I see ACME 7.3.5 deals with account key rollover. Haven't read that before to be honest.

Anyway, I'm wondering what "power" your old account key has when the key has been rolled over? I assume it has none.. That's the whole point of the account key rollover.. So if an attacker has rolled over the account key, you won't be able to do anything, not even unregister?

I think the attack vector is smaller? What are the odds a nation-state attacker comes across your ACME account public key? It's not advertised. Your certificates public key is.

1 Like

As best I could tell when I was experimenting with this, once an account key is rotated the old key isn't anywhere in the system (at least for Let's Encrypt). I think you could even make a certificate using that old key, since it's not on the can't-make-certificate-keys-matching-account-keys list anymore. (Unlike deactivating an account, where it's still "in the system" but just deactivated.) So maybe deactivating a compromised account key is the right thing to do, to prevent that key from being used again?

1 Like

This is starting to sound like this short topic:


So it seems marking an AccountKey as compromised must still be done by alerting the LetsEncrypt service, though a rollover or deactivation would have the same effect of blacklisting it - though it wouldn’t necessarily revoke any outstanding certificates. (I am assuming that my client will not necessarily know of every action tied to a given AccountKey that may require revocation, cleanup, etc).
- @jvanasco

2 Likes

And this topic (that you sent me for grammar lessons @rg305 :slightly_smiling_face:):

1 Like

That should not be possible. See the RFC, section 11.1, last paragraph https://tools.ietf.org/html/rfc8555#section-11.1

Iirc, as explained to me previously by LE staff, Account Keys are blacklisted on rollover and deactivation.

2 Likes

Well, I just tried it on the staging server. I think my testing before was just with account keys (making a new account with a key that used to be an account key but is no longer), not with certificates. So I just tried some tests and it still looks to me that once an account key is changed to a new one that their servers don't keep the old key on the certificate-key-blacklist.

  1. Created a new account, with key 1
  2. Made a certificate on it, with a new key 2, just to make sure my script was working, which worked.
  3. Tried making a certificate with key 1, which of course failed since it was the same as the account key.
  4. Rotated the account key to a new key 3.
  5. Tried making a certificate with key 1, which worked this time because it was no longer an account key.

I haven't tried it on production yet, I must admit, but I suppose I could if you think it might be different there. I don't think that this violates the RFC though, since once a key is no longer used for an account, it's no longer a "known account key pair". I'm guessing that if they needed to blacklist any key which had ever been used on an account, there might need to be a rate limit on how often one could change the account key or some other mitigation to keep me from filling up the blacklist with a bunch of keys.

The account key does remain unusable for certificates if the account is deactivated.

But, none of this interesting digression really answers my original question though, on what procedures I should be following as a "regular" user with a handful of certificates on how often or whether to rotate my account key.

1 Like

I take that back, I think I did just manage to create a certificate with the key of a deactivated account. I think once it's deactivated you can't use the key for another account but you can use it for a certificate. At least in staging. Unless I screwed something up in my testing.

1 Like

Can you explain, so we all understand, exactly how you did that step?

1 Like

I'm assuming he used the keyChange endpoint.

Sure. Here's the key rotation script I was testing with. It's in Node.js, with the only dependency being first running npm install acme to be able to use ACME.js and related libraries.

There's probably a better way to do it, but I've had fun learning how the protocol actually works. It reads the account id and key from files, and then writes the new key to a new file.

[I also have a version of this in AWS Lambda that reads and writes from the AWS Systems Manager Parameter Store instead of files (storing the key in the same place my hobby AWS Lambda cert renewal pulls it from), so I could just schedule a key rotation in AWS for any interval if I knew what rotation interval I "should" use. Hence my initial question here.]

I dedicate this code, such as it is, to the public domain, on the off chance that someone else finds it useful.

"use strict";
const ACME = require('acme');
const Keypairs = require('@root/keypairs');
const fs = require("fs").promises;

const maintainerEmail = "pete-acme-lambda-renewal@cooperjr.name";
const directoryUrl = 'https://acme-staging-v02.api.letsencrypt.org/directory';
const packageAgent = "acme-lambda-renewal/1.0.0";

const accountId = require("./acct-key-id.json");
const oldAccountPrivKey = require("./acct-key-priv-1.json");

const newAccountKeyFilename = "acct-key-priv-2.json";

function notify(ev, msg) {
	console.log("acme notify", ev, msg.altname || '', msg.status || '', msg.message || '');
}

async function getNewNonce(acme) {
	const newNonceUrl = acme._directoryUrls.newNonce;
	const response = await acme.request({
		method: 'HEAD',
		url: newNonceUrl
	});
	return response.headers['replay-nonce'];
}

async function rotateKey() {

	const acme = ACME.create({ maintainerEmail, packageAgent, notify });

	await acme.init(directoryUrl);

	const keyChangeUrl = acme._directoryUrls.keyChange;
	//console.log("keyChangeUrl", keyChangeUrl);

	const noncePromise = getNewNonce(acme);


	const newAccountKeypair = await Keypairs.generate({ kty: 'EC', format: 'jwk' });
	const newAccountPrivKey = newAccountKeypair.private;
	const newAccountPubKey = newAccountKeypair.public;

	const oldAccountPubKey = Keypairs.neuter({jwk: oldAccountPrivKey});

	const innerRequest = await Keypairs.signJws({
		jwk: newAccountPrivKey,
		protected: {
			url: keyChangeUrl,
			jwk: newAccountPubKey,
			kid: false
		},
		payload: {
			account: accountId,
			oldKey: oldAccountPubKey
		}
	});

	const outerRequest = await Keypairs.signJws({
		jwk: oldAccountPrivKey,
		protected: {
			"kid": accountId,
			"nonce": await noncePromise,
			"url": keyChangeUrl
		},
		payload: innerRequest
	});

	try {
		const keyChangeResponse = await acme.request({url: keyChangeUrl, json: outerRequest});
		const statusCode = keyChangeResponse.statusCode;
		console.log("statusCode", statusCode);
		console.log("headers", JSON.stringify(keyChangeResponse.headers));
		console.log("body", keyChangeResponse.body);
		if (statusCode == 200) {
			await fs.writeFile(newAccountKeyFilename, JSON.stringify(newAccountPrivKey));
		}
	} catch (err) {
		console.log("error", err);
		throw err;
	}

}

rotateKey().then(console.log).catch(console.log);
2 Likes