Openssl verification fails for Letsencrypt issued certificate

This sure feels like a simple problem… and yet, I cannot figure out why my openssl won't verify this cert and chain . . .

versions

This chain and cert come from a certbot v0.40.0. (Which is the latest package available for Ubuntu 20.04.)

OpenSSL 1.1.1f My Openssl should have the feature (or "bug fix" if you prefer) that the trusted-first option is enabled by default. And this is confirmed:

17:57UTC:root@bard:/etc# openssl verify -help 2>&1| grep trusted_first
 -trusted_first        search trust store first (default)

The certificates and chain (below) work fine installed in a web server. Both a local curl from the command line, and my desktop [different than this system] web browser are happy with the cert and chain files (below).

The problem I'm trying to solve here is that I cannot openssl verify … this chain and certificate file using openssl from the command line.

ISRG Root X1 installed?

Yes, and ca-certificates is happy.

17:57UTC:root@bard:/etc# ls /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
-rw-r--r-- 1 root root 1939 Sep 22 11:46 /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
17:59UTC:root@bard:/etc# grep ISRG /etc/ca-certificates.conf
mozilla/ISRG_Root_X1.crt
17:59UTC:root@bard:/etc# update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.

the cert and chain

Cert is one file, and there's a cert chain file containing two certs.

Here's the cert…

-----BEGIN CERTIFICATE-----
MIIFKDCCBBCgAwIBAgISA1G6nqla/B5CEPZI6YO4Y7eIMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MTcwNjE1MjBaFw0yMTEyMTYwNjE1MTlaMBwxGjAYBgNVBAMT
EXdpbmRzb3IuYmxrYnguY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEA2vM7VD207G7s04jA5QSG9jmmYCO+xi+4B/nnID66BIA5YJ05ZU6iahuIWDHn
hqfHNW/E+DlouC6MqInuAPOuOMhkQ8sOJMXOwvMXwKbuH5TmkBzieMHcp1X2igAm
/3tGFXzJ4kCLvmm3OOttwAC+fS38F7+pLKQ/yEnNtV/nZbYlHWAJjee1pMc64oHz
jaG7bdXNBvFNFTAk3p9Td61IU0GiF5IG9dDFAFTdV3lYchH9mpKRcuIlupKHchXE
1ZS2MkFX6W4oRdiCT5/lDRACSpLS58LFUTpL/dCOxYAH5xuVHg0teLubiEmygPbT
/k/7JcQBnznBfwCBANJJG15PWQIDAQABo4ICTDCCAkgwDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBQTeSF5n3kz5+2Jk8YIS3WMf6EBOTAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghF3aW5kc29yLmJsa2J4LmNvbTBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3AESUZS6w
7s6vxEAH2Kj+KMDa5oK+2MsxtT/TM5a1toGoAAABe/KcFs0AAAQDAEgwRgIhANJ1
8OZD4y0iwpF7tWBRI+Mzo0m4jAr66TeG+zPFeGhKAiEA6YEv9lTAZNY9V8kmBWs7
FUWddQnd7Defgl+C0AQa+uEAdQB9PvL4j/+IVWgkwsDKnlKJeSvFDngJfy5ql2iZ
fiLw1wAAAXvynBbuAAAEAwBGMEQCIC3LXfXhust7RRcK+491g2UrVZ92m1BfbA19
71tzuTk0AiAhsz2A1NKEq7cjeDEb9i+XAopxh6A/rXvQrzLf0IjNLjANBgkqhkiG
9w0BAQsFAAOCAQEAb6FnAvJwOSWWFILY2E9lUJ/3fo/xDG7gtXfQfNkZfzrQ5uXa
tcc2YCV0SJve7OqRF3xAK/TYbIFpLjsBfetDoE+Tz2Vqa7sAYwH4J5IIVXt4LQef
PoHapxOmUO4+gfDqkVwKlXYXyKTCoIH0JUEfuwuA4hABBnbD6WDh0Zt4RO+ZYd9C
8Lh14YixuupKu3/Xm+yjT7OksVU9zX65gJ9Pades/qjHMb58SCli5ouljueLLNrJ
9/jhxM/5bcom4AgMvIybU2UnXaTDKkw5PhYRE9d6O8hSy1AZ4txKDyYEy9rESeR0
cDn5dB90nx74SJXHQfK2ScC2oVNvz4MXAPrYEg==
-----END CERTIFICATE-----

The chain file has two intermediate certificates. The first ("MIIFFj…") is signed by the newer "ISRG Root X1" CA certificate, and the second one ("MIIFYD…") is signed by the old/now-expired "DST whateveritwas X3".

Here's the chain file…

-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----

what's wrong?

If I simply ask openssl to verify, it is unhappy:

18:09UTC:root@bard:~/TEMP# openssl verify -purpose sslserver -CAfile ./chain.pem ./cert.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error ./cert.pem: verification failed

…does that mean the "ISRG Root X1" CA Certificate isn't installed actually installed in local trust store?

muddying the waters

What if I simplify the chain and just include the intermediate cert signed by the newer "ISRG Root X1". So, just the "MIIFFj…" in the chain file…

18:21UTC:root@bard:~/TEMP# openssl verify -purpose sslserver -CAfile ./chainISRGX1.pem ./cert.pem
./cert.pem: OK
18:22UTC:root@bard:~/TEMP# echo $?
0

Happy!

Which leads me to believe that it's the presence of the second, certificate in the chain file that causes the problem.

but my Openssl has the trusted_first option enabled.

1 Like

-CAfile option doesn't do what you expect:

   -CAfile file
       A file of trusted certificates.  The file should contain one or
       more certificates in PEM format.

it override the ISRG root X1 from trust store to ISRG-signed_by-DST because it's same identity: but non self signed certificate can't used as root in openssl verify unless:

   -partial_chain
       Allow verification to succeed even if a complete chain cannot be
       built to a self-signed trust-anchor, provided it is possible to
       construct a chain to a trusted certificate that might not be self-
       signed.

option is set

6 Likes

The verify used to work fine, until that "DST Root X3" cert expired. So it used to verify because the "DST Root X3" in my chain file was-and-still-is self-signed, but is now expired?

Then I really should have been using the -partial_chain option all along in my verify command?

(Just trying to be sure I understand what's going on / how this all works. :slight_smile:

no: for exemple:
openssl verify -partial_chain -CAfile testin1.pem exemple.invalid.crt
will succide to verify if exemple.invalid.crt did signed by testin1.pem (as this CA is now in trust for this command)

2 Likes

Btw I have to warn you using unfiltered -CAfile to verify random is dangerous:
for exemple
openssl verify somesite.pem -CAfile evilselfsigned will pass the verification, which I pretty sure you don't want
use -untrusted option to do that (this still does not fix old root overriding though)

3 Likes

Well, now I'm confused and my original question remains: I have a cert and a chain obtained from LE via the certbot. How can I openssl verify them?

Why can't I simply use the -partial_chain option in my situation where there's no danger of a bad actor signing the cert, nor injecting a bad cert into my chain file?

For openssl verify, my understanding is that you need to put just the root that you're verifying against as -CAfile (unless you're trying to validate against your system root), and other intermediates as argument to -untrusted. I don't think there's a command line parameter that takes the whole chain file, you need to sort out separately which is the cert you're testing, what are the additional intermediates that were sent with it, and what is the trust root.

4 Likes

openssl verify -purpose sslserver -untrusted chain.pem cert.pem

or

openssl verify -purpose sslserver -untrusted chain_1.pem -untrusted chain_0.pem cert.pem

in which you split the chain file into chain_0 and chain_1, then pass them into verify in reverse order.

(both verified to work on your cert+chain, even on older installs)

6 Likes

Thanks all! I think I have my brain wrapped around this enough now to proceed. :+1:

4 Likes

And on a Friday!
Congrats to you - I lose mine by Wednesdays - LOL

1 Like

That's what we've all said. You'll be back. [evil laugh]

Chain Verification is probably the most user-antagonistic component of OpenSSL. Don't feel bad if you get tripped up on this again.

4 Likes

You can ommit partial_chain if you retrieve all the certs needed for full chain and following command would do it for you.

wget -O isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem && wget -O isrg-root-x1-cross-signed.pem https://letsencrypt.org/certs/isrg-root-x1-cross-signed.pem && wget -O lets-encrypt-r3.pem https://letsencrypt.org/certs/lets-encrypt-r3.pem && wget -O lets-encrypt-r3-cross-signed.pem https://letsencrypt.org/certs/lets-encrypt-r3-cross-signed.pem && cat isrgrootx1.pem isrg-root-x1-cross-signed.pem lets-encrypt-r3.pem lets-encrypt-r3-cross-signed.pem > combined_chain1.pem && dos2unix combined_chain1.pem && rm -f lets-encrypt-r3*.* && rm -f isrg*.*

and use combined_chain1.pem for -CAfile, note the order of files concatention seems important, root, signed root, intermediate cert root and then intermediate signed cert root.

openssl verify -CAfile combined_chain1.pem cert1.pem

Should return

cert1.pem: OK
1 Like

What I'm using uses openssl to verify the cert and chain files the web servers are using. So fiddling with the files, or making extra files with chains, defeats the point of the thing I'm using.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.