PKCS12 Certificate conversion stopped working


#1

I’m running an IBM Spectrum Protect Operations Centre server using a wildcard certificate. The Spectrum Protect server is actually running WebSphere Liberty server under the hood. WebSphere Liberty requires a pkcs12 certificate, which I’ve been using without problem for six months or so. That is: I’ve converted Let’s Encrypt wildcard certificates twice and successfully imported them into Liberty. Until the last update I did yesterday, where the procedure I have been using generates an error.

I have let’s encrypt installed on a Linux box and performed the update as usual using:
/usr/local/bin/certbot -d *.domain.net --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns certonly

The webserver on the Linux box is working fine with the newly updated cert.

To convert for pkcs12 to be imported into the WebSphere liberty server, I issue the following command (which is the same command I’ve used twice before with no problem):
openssl pkcs12 -export -out keystore.p12 -inkey privkey.pem -in fullchain.pem -name "default" -password pass:password

This generates a keystore.p12 file, which I copy over to the websphere server and import using the java keystore, using the following process:

Open “ikeyman” (IBM key Manager java gui)
Open the “gui-truststore.jks” file - the websphere liberty servers key database
Delete the old “default” cert
Import the new keystore.p12 file as “default” cert

The new “default” certificate is highlited in yellow, when I click on “validate” it says “warning: validation failed: Missing intermediate or root certificate”

I have updated Spectrum Protect (and consequently WebSphere Liberty) however this was with a previous version of the certificate made with the same process and everything continued working just fine, until this update.

From searching around the internet, it appears that I’m using the correct method to convert the let’s encrypt supplied files into a pkcs12, but TBH encryption is one of the few areas in IT where I’m at a loss knowledge wise, so I’m just not sure if I’m asking the right questions.

Does anyone have any ideas?


#2

Hi @fraser73

can you share a screenshot? Perhaps the ISRG Root Certificate

is missing.

Looks not like an import problem.


#3

Happy to share screenshots, but not entirely sure what’s going to be helpful… Here’s the key info:

As for the ISRG Root Certificate - I’m not 100% I understand what this is, but what I think I know is that it’s the certificate used to sign the certificates that Let’s encrypt send to me? (I’m painfully aware that I don’t really know much about encryption…) So it forms part of the chain of trust? Does this need to be included into my pkcs12 file for some reason? If so, I don’t understand what it has worked before and not now?

Thanks for your help, and sorry for my lack of understanding…


#4

I need something like this:

Root-Certificate

If your import routine would be wrong, you wouldn’t see such a screenshot with these data. So your import routine is correct.


#5

The keytool doesn’t have a certificate hierarchy view, it’s got the following:
image

I can’t find any mention of “ISRG Root X1” anywhere in the list. Should I have been including this in the openssl command to create the pkcs12 in the first place?


#6

As new users can only put one image into a post, here is the common name:
Here is the issuer’s common name:


#7

There are root certificates - often preinstalled.

The root cerficiates validates intermediate certificates - “Let’s Encrypt Authority X3” is one. These validates user generated certificates.

You need to install the intermediate certificate. The correct root certificate should be installed.

But your error

says: One of these certificates is missing - the intermediate, the root or both.


#8

Ok, I think I understand - What the error is saying is that the PKCS12 file should include root and intermediate certs, but one of these is missing, so it’s not valid.

That much seems obvious, but thanks for taking the time to explain it.

So, the question is - why has this just become the case, when the same procedure has worked before?
I can see two possible problems:

Has the websphere server stopped trusting something and this has caused the problem?
Does the pkcs12 file I’ve created no longer contain everything it needs?


#9

Not exactly. Normally, root certificates are never included in pkcs12 files.

Isn’t it possible that you open your pkcs12 in Windows? Then you should see it.

Or upload your file fullchain.pem - then it’s possible to test that. Informations there are public, not sensitive. Public Certificate Transparency logs have the same informations.


#10

Here is my fullchain;pem - however as I’m a new user, I can’t upload files, so it’s plaintext in the message:

-----BEGIN CERTIFICATE-----
MIIGCTCCBPGgAwIBAgISA0BNyEbHij6vkKPxnH1elDNCMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0xODA5MDMxNDM4MDZaFw0x
ODEyMDIxNDM4MDZaMBoxGDAWBgNVBAMMDyouZnJpbmtpYWM3Lm5ldDCCASIwDQYJ
KoZIhvcNAQEBBQADggEPADCCAQoCggEBAKsT22o/kzkXAjBq2RYqiuVWaG03J77S
5YBpx9c/3TjNcNff2ILic8XO3HlkZ5ffGFVDNOp/GE9tOqZYZx+maBF+wVRR07Iw
ZMguV3Rn2RJZlajPp8qxbTS4KmN7RoBdXKD0w8B+9wu60dqhv16xjp33WM0m/RWw
EJVbOiZWz/PZXNKcRJhOxuDd2a94bXfeQu2YaDPfubY0NxbCgJNWUSWpu3+0CVuf
GKXR+sTdaqFgysDowkvfP3yIKWQ8rEE/hOhn3zEhNmSbxJws6y6KWO3Z9WI2MqD1
TsjMHkzWwKusGOTzr6Sq1mV7ewvCTY0G9ZCYip7Fq5gOUdpQtCiAvJkCAwEAAaOC
AxcwggMTMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYB
BQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUrTfC2/NOtkG4DZO3/KPpuGnK
H1YwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEwbwYIKwYBBQUHAQEE
YzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMubGV0c2VuY3J5cHQu
b3JnLzAaBgNVHREEEzARgg8qLmZyaW5raWFjNy5uZXQwgf4GA1UdIASB9jCB8zAI
BgZngQwBAgEwgeYGCysGAQQBgt8TAQEBMIHWMCYGCCsGAQUFBwIBFhpodHRwOi8v
Y3BzLmxldHNlbmNyeXB0Lm9yZzCBqwYIKwYBBQUHAgIwgZ4MgZtUaGlzIENlcnRp
ZmljYXRlIG1heSBvbmx5IGJlIHJlbGllZCB1cG9uIGJ5IFJlbHlpbmcgUGFydGll
cyBhbmQgb25seSBpbiBhY2NvcmRhbmNlIHdpdGggdGhlIENlcnRpZmljYXRlIFBv
bGljeSBmb3VuZCBhdCBodHRwczovL2xldHNlbmNyeXB0Lm9yZy9yZXBvc2l0b3J5
LzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB1ACk8UZZUyDlluqpQ/FgH1Ldvv1h6
KXLcpMMM9OVFR/R4AAABZaAVensAAAQDAEYwRAIgNYPWSD7MgztGT1I2tw6IFGNP
THtQRF53q7cfCYrQp/gCIA2cc3cm9Vazz1EYYhQzC9GZpmm+wBQuvoWPNshb2+05
AHcAVYHUwhaQNgFK6gubVzxT8MDkOHhwJQgXL6OqHQcT0wwAAAFloBV7JwAABAMA
SDBGAiEAzm9vmAhUGfyX7gZc0kcnIn/6ohZexKVNs4+5arrTGmECIQCcsD+6KLfJ
kVFb48OegyrQpvWSGiAjz3AcdN+5wmhGZDANBgkqhkiG9w0BAQsFAAOCAQEAJGTy
bAEP4KDv8SWF0bm1PbaiOdA86nGxGEVXAJvMRSeRCnwWywZo8NDKUQOnPb+s524E
M09Cs6HMuOAmXqD/lw9pL7G7tNVyCmf9Q1pYHNRS1j8Ftxg8kWsxiFt170fzOa/e
KdPVuthlEHjbO7BjdjV4Ab2B86ta7bvZumix91EvNPP1+hrA7TGz0gseryXeFIVn
ELuo278rwnSfN/g6mdUV7RShcXRAWDwhEq27bXzxKL/4IflWQqOUQKQmsPdftr4+
KcqfgNqaRYPrCa3NHUyBi4Q8VOMFS6jtcFDvT4EK+ewdNdOocRvMN1AG2kiq+ZYm
GjNklIFaAIDHvDeVPw==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----

#11

This is simple - these are two certificates. Your domain certificate and the intermediate Let’s Encrypt X3 certificate. First is yours (save it as windows txt file with the file extension .crt), second is Let’s Encrypt X3.

I don’t use OpenSSL, but normally the intermediate certificate should be included in the pkcs12 file.

But I see the DST Root CA X3 as Root-Certificate. The Let’s Encrypt X3 is cross signed with that root certificate.

Isn’t there a certificate store or something else you can check?


#12

I’ve installed the certificate onto my windows workstation and it’s reporting the certification path of:

DST Root CA X3
- Let's Encrypt Authority X3
-- default

and a certificate status of “This certificate is ok”.

I’m beginning to think that the problem may be with WebSphere Liberty.


#13

I just installed the certificate onto my Exchange server web access and it’s working fine. I therefor think this is a problem with WebSphere Liberty Server.

Would this likely be that WebSphere doesn’t trust Let’s Encrypt? I’m not sure how this may have happened, as it has done before though.


#14

I have managed to solve the problem with help gratefully received from @JurgenAuer and a contact at IBM - without both of whom I could barely have started working this out.

As noted above, the PKCS12 file was valid.

In order to resolve the problem, I needed to download both the “Let’s Encrypt Authority X3 (IdenTrust Cross Signed)” and the IdenTrust “TrustID X3” root certificate and install them into the key manager as “Signer Certificates”. Then restart WebSphere Liberty Server.

I am at a total loss to explain why it worked for the last 9 months, but am investigating with my contact at IBM.


#15

Thanks, good to know. So this “key manager” is your certificate store.


#16

Hi… To clarify, key manager is a java app which allows management of the certificate store database files.


#17

FYI, manually importing root and especially intermediate certificates isn’t a perfect long-term plan. The protocol is intended that CAs can change intermediates at any time without warning, though it doesn’t happen often; Let’s Encrypt will change to a different default root around 2021.

If you’re automating this, maybe you can script it to check if /etc/letsencrypt/live/example.com/chain.pem has changed and is in the store. (That’s the intermediate.)

If you’re doing it manually, maybe just add it to your runbook.


#18

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.