Running boulder test instance -> replace Issuer certificate

[haja0011]

Hi,
we installed a boulder test instance in our company. We can get certificates and all is up and working.

Now we want to replace the issuer certificates: ‘cackling cryptographer fake ROOT’ and the intermediate certificates with our own QA certificates.

In directory: '/usr/share/go/src/github.com/letsencrypt/boulder/test/'
I found these certificates and tried to replace them with our own root and intermediate certificates and keys.
test-ca2.pem test-ca.pem test-root.pem
test-ca2.key test-ca.key test-root.key

In this directory are also some configuration files for the keys:
test-root.key-pkcs11.json …
I don’t know, if and what, I have to do with them.
Our keys are not password protected.

But this simple approach doesn’t work. Boulder doesn’t start anymore.
I get following Error message:
E141454 boulder-ca [AUDIT] Couldn’t load private key: Issuer key did not match issuer cert test/test-ca2.pem

What do I have to do to use our own Issuer Certificates?

Regards
Jan

[schoen]

@jsha, do you think you could help with this?

[jsha]

You’ll also need to update test-ca.key.der and test-root.key.der with the DER format of your keys. That’s what gets loaded into SoftHSM via test/entrypoint.sh and used via PKCS#11 for signing. You can use openssl rsa to transform your PEM-formatted keys into DER format.

[haja0011]

I converted the following key files to the DER Format:
(I tried it also without the test-ca2.key.der, same result)

openssl rsa -inform PEM -in <filename.key> -outform DER -out <out_filename.key.der>
test-root.key -> test-root.key.der
test-ca.key -> test-ca.key.der
test-ca2.key -> test-ca2.key.der

I started boulder using:

/usr/bin/docker-compose run --service-ports boulder ./start.py

Now I get the error:

E135812 boulder-ca [AUDIT] Couldn't load private key: Issuer key did not match issuer cert test/test-ca2.pem

---

---

I double checked all my keys with openssl:

openssl x509 -in test-root.pem -noout -modulus |md5sum
openssl rsa -in test-root.key -noout -modulus |md5sum
openssl rsa -inform DER -in test-root.key.der -noout -modulus |md5sum
71b1a1ed8e7b85ab9b9e47010d35c983
71b1a1ed8e7b85ab9b9e47010d35c983
71b1a1ed8e7b85ab9b9e47010d35c983

openssl x509 -in test-ca2.pem -noout -modulus |md5sum
openssl rsa -in test-ca2.key -noout -modulus |md5sum
openssl rsa -inform DER -in test-ca2.key.der -noout -modulus |md5sum
711be643141a5e953fc5a1596a5613ac
711be643141a5e953fc5a1596a5613ac
711be643141a5e953fc5a1596a5613ac

openssl rsa -in test-ca.key -noout -modulus |md5sum
openssl x509 -in test-ca.pem -noout -modulus |md5sum
openssl rsa -inform DER -in test-ca.key.der -noout -modulus |md5sum
edd07c907adbce7b7ace1eaf53de6dba
edd07c907adbce7b7ace1eaf53de6dba
edd07c907adbce7b7ace1eaf53de6dba

I verified the certificate chain:

openssl verify -CAfile test-root.pem -trusted test-root.pem test-ca2.pem test-ca.pem
test-ca2.pem: OK
test-ca.pem: OK

I checked all keys using:

openssl rsa -inform DER -in <filename.key.der> -check
openssl rsa -in <filename.key> -check

Do you have another idea what can be wrong?

[jsha]

It’s possible that entrypoint.sh is failing to load the new key into the HSM. Do you see any error messages from that part of the script? Once boulder and the bhsm container are up and running, you should be able to set PKCS11_PROXY_SOCKET=tcp://boulder-hsm:5657 in a shell on your host and use the same pkcs11-tool commands to communicate with the HSM directly to load keys manually and see if that helps. Note that you’d need to compile and build pkcs11-proxy locally.

[haja0011]

There is following log output which could come from this place, but it is good output:

Created private key:
Private Key Object; RSA
label: intermediate_key
Usage: decrypt, sign, unwrap
Created private key:
Private Key Object; RSA
label: root_key
Usage: decrypt, sign, unwrap

started exec listenbuddy -listen :19091 -speak publisher.boulder:9091 with pid 65
started exec listenbuddy -listen :19092 -speak va.boulder:9092 with pid 66
started exec listenbuddy -listen :19093 -speak ca.boulder:9093 with pid 67
started exec listenbuddy -listen :19094 -speak ra.boulder:9094 with pid 68
started exec listenbuddy -listen :19095 -speak sa.boulder:9095 with pid 81
started exec listenbuddy -listen :19096 -speak ca.boulder:9096 with pid 86
started exec listenbuddy -listen :19097 -speak va.boulder:9097 with pid 91
started exec listenbuddy -listen :19098 -speak va.boulder:9098 with pid 97
export GO15VENDOREXPERIMENT=1
GOBIN=/go/src/github.com/letsencrypt/boulder/bin go install ./...
gsb-test-srv: 2018/02/02 16:21:50 Starting GSB Test Server on ":6000"

Boulder itself has not been started:

root@jha-utm-02:/usr/share/go/src/github.com/letsencrypt/boulder# docker-compose ps
Name Command State Ports
--------------------------------------------------------------------
boulder_bhsm_1 /usr/local/bin/pkcs11-daem ... Up 5657/tcp
boulder_bmysql_1 docker-entrypoint.sh mysql ... Up 3306/tcp

After the error message it exits:

E162150 boulder-ca [AUDIT] Couldn't load private key: Issuer key did not match issuer cert test/test-ca2.pem
These processes exited early (check above for their output):
'exec ./bin/boulder-ca --config test/config/ca.json' with pid 1080 exited 1

I successfully compiled locally pkcs11-proxy by following the steps in (...)boulder-tools/build.sh and I tried to import the key manually, but the connection refused.:

export PKCS11_PROXY_SOCKET=tcp://boulder-hsm:5657
pkcs11-tool --module=/usr/local/lib/libpkcs11-proxy.so --type privkey --pin 5678 --login --so-pin 1234 addkey --token-label root --write-object test/test-root.key.der --label root_key

I think the connection refused cause boulder was not started. I also tried it:

export PKCS11_PROXY_SOCKET=tcp://127.0.0.1:5657
export PKCS11_PROXY_SOCKET=tcp://boulder_bhsm_1:5657

without success. Perhaps I must configure docker to allow it...

[jsha]

Ah, you can start bhsm separately with docker-compose up bhsm

[haja0011]

Hi,

thank you for your help. I have it running now.
I didn’t notice that test-ca2 uses the same key as test-ca.

Regards
Jan