JWS verification error on account deactivation preceded by key rollover

stewe · April 9, 2022, 9:23pm

Recently I stumbled upon an JWS verification error (urn:ietf:params:acme:error:malformed) error when performing the following steps:

register account with key A
account key rollover to key B
deactivate account

In step 3. the request should be of course signed with the new key B, but if steps 2. and 3. are performed in close succession the account deactivation sometimes fails with the error mentioned above. I assume it is some kind of race condition because when performing step 3. with some time delay the error does not appear. To test this therory I just signed the request to deactivate the account with the old key A. Surprisingly it sometimes works, here is the log:

For comparison here is the log of the expected behaviour (when lucky or using a delay between step 2. and 3.):

Osiris · April 9, 2022, 10:00pm

I'm sorry, but I don't understand anything of what you're saying..

What is it exactly you're trying to do here? Develop your own ACME client? Could you please elaborate about the background of your thread, because right now, it's rather difficult to understand..

stewe · April 9, 2022, 10:13pm

I just stumbled upon this problem when running some tests for the next release of my ACME client ACMECert.
It is not a big problem since it is pretty much an edge case. Just wanted to let you know that there is a potential bug in boulder.
Maybe this is the wrong place to post this and it is better if I open a github issue on boulder?

They key point is that boulder accepts (a short period of time) requests signed with the old key after account key rollover to a new key.

_az · April 9, 2022, 10:35pm

Hmm. Boulder has an in-memory cache for account registrations, sitting in front of the database.

I wonder if there is potentially a missing invalidation in the key rollover path. I can't see one in the code, but I didn't look super closely. Going to tag @lestaff just in case.

jvanasco · April 12, 2022, 3:18pm

Are you seeing this on a self-hosted version of Boulder? Do you see this on Pebble? If you can reproduce this on self-hosted versions of either, I would post reports there and cross-reference.

If you can't recreate there, but can on LetsEncrypt's production and/or staging, it may be a dev/ops issue with caching or something in that environment and posting here would be better.

stewe · April 12, 2022, 4:34pm

So far I only tested it on Let's Encrypt live and staging. Here is the log of the live system (I modified the script to sleep 1 second after receiving the JWS verification error and trying the same request again, which then succeeds):

I'm going to try to test it using a self-hosted boulder/pebble next.

stewe · April 13, 2022, 7:20pm

Just tested it with pebble, there I can't reproduce the problem.

So it seems to be a caching/missing invalidation problem..

_az · April 13, 2022, 10:37pm

@jsha sorry for the second tag, I'm not sure whether you came across this thread. I think @stewe is right about something being wrong here.

My own reproduction using config-next:

package main

import (
	"crypto/ecdsa"
	"crypto/elliptic"
	"crypto/rand"
	"log"

	"github.com/eggsampler/acme/v3"
)

// docker-compose -f docker-compose.yml -f docker-compose.next.yml up
// go run main.go

func main() {
	cl, err := acme.NewClient("http://localhost:4001/directory")
	if err != nil {
		log.Fatalf("Couldn't create a client: %v", err)
	}

	// Register an account
	ak1, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
	acc, err := cl.NewAccount(ak1, false, true)
	if err != nil {
		log.Fatalf("Couldn't register an account: %v", err)
	}

	// Change the account's key.
	ak2, _ := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
	updatedAcc, err := cl.AccountKeyChange(acc, ak2)
	if err != nil {
		log.Fatalf("Couldn't change the account key: %v", err)
	}

	// Here's where things seem to go wrong:
	// 1. Deactivating the account with the new key should succeed, but doesn't
	if _, err := cl.DeactivateAccount(updatedAcc); err != nil {
		log.Printf("Could not deactivate account using new key: %v", err)
	}

	// 2. Deactivating the account with the old key should fail, but doesn't.
	if _, err := cl.DeactivateAccount(acc); err != nil {
		log.Printf("Could not deactivate account using old key: %v", err)
	} else {
		log.Println("Managed to deactivate the account using the old key. :(")
	}
}

$ go run ./cmd/repro-key-rollover-race/main.go
2022/04/14 08:33:34 Could not deactivate account using new key: acme: error code 400 "urn:ietf:params:acme:error:malformed": JWS verification error
2022/04/14 08:33:34 Managed to deactivate the account using the old key. :(

Running the same program on config works just fine, presumably because the account cache is not enabled there.

jsha · April 13, 2022, 11:21pm

Thanks for the second tag. I had seen this but it had fallen back off my radar. I think you're right that there's a cache invalidation issue here. We'll have to think about how best to solve it.

system · May 13, 2022, 11:22pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
JWS verification error - EC account keys not working Client dev	9	4327	March 9, 2016
Account Deactivation with ‘kid’ Field per ACME 8555 Client dev	7	95	April 29, 2025
'JWS verification error' trying to revoke a certificate Client dev	23	4091	December 14, 2021
Account Key Roll-over : Parse error reading JWS Client dev	10	3133	November 13, 2017
Help creating new account/JWS errors Client dev	7	2585	February 13, 2019

JWS verification error on account deactivation preceded by key rollover

Related topics