I’m running an IBM Spectrum Protect Operations Centre server using a wildcard certificate. The Spectrum Protect server is actually running WebSphere Liberty server under the hood. WebSphere Liberty requires a pkcs12 certificate, which I’ve been using without problem for six months or so. That is: I’ve converted Let’s Encrypt wildcard certificates twice and successfully imported them into Liberty. Until the last update I did yesterday, where the procedure I have been using generates an error.
I have let’s encrypt installed on a Linux box and performed the update as usual using: /usr/local/bin/certbot -d *.domain.net --server https://acme-v02.api.letsencrypt.org/directory --manual --preferred-challenges dns certonly
The webserver on the Linux box is working fine with the newly updated cert.
To convert for pkcs12 to be imported into the WebSphere liberty server, I issue the following command (which is the same command I’ve used twice before with no problem): openssl pkcs12 -export -out keystore.p12 -inkey privkey.pem -in fullchain.pem -name "default" -password pass:password
This generates a keystore.p12 file, which I copy over to the websphere server and import using the java keystore, using the following process:
Open “ikeyman” (IBM key Manager java gui)
Open the “gui-truststore.jks” file - the websphere liberty servers key database
Delete the old “default” cert
Import the new keystore.p12 file as “default” cert
The new “default” certificate is highlited in yellow, when I click on “validate” it says “warning: validation failed: Missing intermediate or root certificate”
I have updated Spectrum Protect (and consequently WebSphere Liberty) however this was with a previous version of the certificate made with the same process and everything continued working just fine, until this update.
From searching around the internet, it appears that I’m using the correct method to convert the let’s encrypt supplied files into a pkcs12, but TBH encryption is one of the few areas in IT where I’m at a loss knowledge wise, so I’m just not sure if I’m asking the right questions.
As for the ISRG Root Certificate - I’m not 100% I understand what this is, but what I think I know is that it’s the certificate used to sign the certificates that Let’s encrypt send to me? (I’m painfully aware that I don’t really know much about encryption…) So it forms part of the chain of trust? Does this need to be included into my pkcs12 file for some reason? If so, I don’t understand what it has worked before and not now?
Thanks for your help, and sorry for my lack of understanding…
The keytool doesn’t have a certificate hierarchy view, it’s got the following:
I can’t find any mention of “ISRG Root X1” anywhere in the list. Should I have been including this in the openssl command to create the pkcs12 in the first place?
Ok, I think I understand - What the error is saying is that the PKCS12 file should include root and intermediate certs, but one of these is missing, so it’s not valid.
That much seems obvious, but thanks for taking the time to explain it.
So, the question is - why has this just become the case, when the same procedure has worked before?
I can see two possible problems:
Has the websphere server stopped trusting something and this has caused the problem?
Does the pkcs12 file I’ve created no longer contain everything it needs?
Not exactly. Normally, root certificates are never included in pkcs12 files.
Isn't it possible that you open your pkcs12 in Windows? Then you should see it.
Or upload your file fullchain.pem - then it's possible to test that. Informations there are public, not sensitive. Public Certificate Transparency logs have the same informations.
This is simple - these are two certificates. Your domain certificate and the intermediate Let's Encrypt X3 certificate. First is yours (save it as windows txt file with the file extension .crt), second is Let's Encrypt X3.
I don't use OpenSSL, but normally the intermediate certificate should be included in the pkcs12 file.
But I see the DST Root CA X3 as Root-Certificate. The Let's Encrypt X3 is cross signed with that root certificate.
Isn't there a certificate store or something else you can check?
I just installed the certificate onto my Exchange server web access and it’s working fine. I therefor think this is a problem with WebSphere Liberty Server.
Would this likely be that WebSphere doesn’t trust Let’s Encrypt? I’m not sure how this may have happened, as it has done before though.
I have managed to solve the problem with help gratefully received from @JurgenAuer and a contact at IBM - without both of whom I could barely have started working this out.
As noted above, the PKCS12 file was valid.
In order to resolve the problem, I needed to download both the “Let’s Encrypt Authority X3 (IdenTrust Cross Signed)” and the IdenTrust “TrustID X3” root certificate and install them into the key manager as “Signer Certificates”. Then restart WebSphere Liberty Server.
I am at a total loss to explain why it worked for the last 9 months, but am investigating with my contact at IBM.
FYI, manually importing root and especially intermediate certificates isn’t a perfect long-term plan. The protocol is intended that CAs can change intermediates at any time without warning, though it doesn’t happen often; Let’s Encrypt will change to a different default root around 2021.
If you’re automating this, maybe you can script it to check if /etc/letsencrypt/live/example.com/chain.pem has changed and is in the store. (That’s the intermediate.)
If you’re doing it manually, maybe just add it to your runbook.
