This sure feels like a simple problem… and yet, I cannot figure out why my openssl won't verify this cert and chain . . .
versions
This chain and cert come from a certbot v0.40.0. (Which is the latest package available for Ubuntu 20.04.)
OpenSSL 1.1.1f My Openssl should have the feature (or "bug fix" if you prefer) that the trusted-first option is enabled by default. And this is confirmed:
17:57UTC:root@bard:/etc# openssl verify -help 2>&1| grep trusted_first
-trusted_first search trust store first (default)
The certificates and chain (below) work fine installed in a web server. Both a local curl
from the command line, and my desktop [different than this system] web browser are happy with the cert and chain files (below).
The problem I'm trying to solve here is that I cannot openssl verify …
this chain and certificate file using openssl from the command line.
ISRG Root X1 installed?
Yes, and ca-certificates is happy.
17:57UTC:root@bard:/etc# ls /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
-rw-r--r-- 1 root root 1939 Sep 22 11:46 /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
17:59UTC:root@bard:/etc# grep ISRG /etc/ca-certificates.conf
mozilla/ISRG_Root_X1.crt
17:59UTC:root@bard:/etc# update-ca-certificates
Updating certificates in /etc/ssl/certs...
0 added, 0 removed; done.
Running hooks in /etc/ca-certificates/update.d...
done.
the cert and chain
Cert is one file, and there's a cert chain file containing two certs.
Here's the cert…
-----BEGIN CERTIFICATE-----
MIIFKDCCBBCgAwIBAgISA1G6nqla/B5CEPZI6YO4Y7eIMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA5MTcwNjE1MjBaFw0yMTEyMTYwNjE1MTlaMBwxGjAYBgNVBAMT
EXdpbmRzb3IuYmxrYnguY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEA2vM7VD207G7s04jA5QSG9jmmYCO+xi+4B/nnID66BIA5YJ05ZU6iahuIWDHn
hqfHNW/E+DlouC6MqInuAPOuOMhkQ8sOJMXOwvMXwKbuH5TmkBzieMHcp1X2igAm
/3tGFXzJ4kCLvmm3OOttwAC+fS38F7+pLKQ/yEnNtV/nZbYlHWAJjee1pMc64oHz
jaG7bdXNBvFNFTAk3p9Td61IU0GiF5IG9dDFAFTdV3lYchH9mpKRcuIlupKHchXE
1ZS2MkFX6W4oRdiCT5/lDRACSpLS58LFUTpL/dCOxYAH5xuVHg0teLubiEmygPbT
/k/7JcQBnznBfwCBANJJG15PWQIDAQABo4ICTDCCAkgwDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBQTeSF5n3kz5+2Jk8YIS3WMf6EBOTAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghF3aW5kc29yLmJsa2J4LmNvbTBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQQGCisGAQQB1nkCBAIEgfUEgfIA8AB3AESUZS6w
7s6vxEAH2Kj+KMDa5oK+2MsxtT/TM5a1toGoAAABe/KcFs0AAAQDAEgwRgIhANJ1
8OZD4y0iwpF7tWBRI+Mzo0m4jAr66TeG+zPFeGhKAiEA6YEv9lTAZNY9V8kmBWs7
FUWddQnd7Defgl+C0AQa+uEAdQB9PvL4j/+IVWgkwsDKnlKJeSvFDngJfy5ql2iZ
fiLw1wAAAXvynBbuAAAEAwBGMEQCIC3LXfXhust7RRcK+491g2UrVZ92m1BfbA19
71tzuTk0AiAhsz2A1NKEq7cjeDEb9i+XAopxh6A/rXvQrzLf0IjNLjANBgkqhkiG
9w0BAQsFAAOCAQEAb6FnAvJwOSWWFILY2E9lUJ/3fo/xDG7gtXfQfNkZfzrQ5uXa
tcc2YCV0SJve7OqRF3xAK/TYbIFpLjsBfetDoE+Tz2Vqa7sAYwH4J5IIVXt4LQef
PoHapxOmUO4+gfDqkVwKlXYXyKTCoIH0JUEfuwuA4hABBnbD6WDh0Zt4RO+ZYd9C
8Lh14YixuupKu3/Xm+yjT7OksVU9zX65gJ9Pades/qjHMb58SCli5ouljueLLNrJ
9/jhxM/5bcom4AgMvIybU2UnXaTDKkw5PhYRE9d6O8hSy1AZ4txKDyYEy9rESeR0
cDn5dB90nx74SJXHQfK2ScC2oVNvz4MXAPrYEg==
-----END CERTIFICATE-----
The chain file has two intermediate certificates. The first ("MIIFFj…") is signed by the newer "ISRG Root X1" CA certificate, and the second one ("MIIFYD…") is signed by the old/now-expired "DST whateveritwas X3".
Here's the chain file…
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFYDCCBEigAwIBAgIQQAF3ITfU6UK47naqPGQKtzANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTIxMDEyMDE5MTQwM1oXDTI0MDkzMDE4MTQwM1ow
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwggIiMA0GCSqGSIb3DQEB
AQUAA4ICDwAwggIKAoICAQCt6CRz9BQ385ueK1coHIe+3LffOJCMbjzmV6B493XC
ov71am72AE8o295ohmxEk7axY/0UEmu/H9LqMZshftEzPLpI9d1537O4/xLxIZpL
wYqGcWlKZmZsj348cL+tKSIG8+TA5oCu4kuPt5l+lAOf00eXfJlII1PoOK5PCm+D
LtFJV4yAdLbaL9A4jXsDcCEbdfIwPPqPrt3aY6vrFk/CjhFLfs8L6P+1dy70sntK
4EwSJQxwjQMpoOFTJOwT2e4ZvxCzSow/iaNhUd6shweU9GNx7C7ib1uYgeGJXDR5
bHbvO5BieebbpJovJsXQEOEO3tkQjhb7t/eo98flAgeYjzYIlefiN5YNNnWe+w5y
sR2bvAP5SQXYgd0FtCrWQemsAXaVCg/Y39W9Eh81LygXbNKYwagJZHduRze6zqxZ
Xmidf3LWicUGQSk+WT7dJvUkyRGnWqNMQB9GoZm1pzpRboY7nn1ypxIFeFntPlF4
FQsDj43QLwWyPntKHEtzBRL8xurgUBN8Q5N0s8p0544fAQjQMNRbcTa0B7rBMDBc
SLeCO5imfWCKoqMpgsy6vYMEG6KDA0Gh1gXxG8K28Kh8hjtGqEgqiNx2mna/H2ql
PRmP6zjzZN7IKw0KKP/32+IVQtQi0Cdd4Xn+GOdwiK1O5tmLOsbdJ1Fu/7xk9TND
TwIDAQABo4IBRjCCAUIwDwYDVR0TAQH/BAUwAwEB/zAOBgNVHQ8BAf8EBAMCAQYw
SwYIKwYBBQUHAQEEPzA9MDsGCCsGAQUFBzAChi9odHRwOi8vYXBwcy5pZGVudHJ1
c3QuY29tL3Jvb3RzL2RzdHJvb3RjYXgzLnA3YzAfBgNVHSMEGDAWgBTEp7Gkeyxx
+tvhS5B1/8QVYIWJEDBUBgNVHSAETTBLMAgGBmeBDAECATA/BgsrBgEEAYLfEwEB
ATAwMC4GCCsGAQUFBwIBFiJodHRwOi8vY3BzLnJvb3QteDEubGV0c2VuY3J5cHQu
b3JnMDwGA1UdHwQ1MDMwMaAvoC2GK2h0dHA6Ly9jcmwuaWRlbnRydXN0LmNvbS9E
U1RST09UQ0FYM0NSTC5jcmwwHQYDVR0OBBYEFHm0WeZ7tuXkAXOACIjIGlj26Ztu
MA0GCSqGSIb3DQEBCwUAA4IBAQAKcwBslm7/DlLQrt2M51oGrS+o44+/yQoDFVDC
5WxCu2+b9LRPwkSICHXM6webFGJueN7sJ7o5XPWioW5WlHAQU7G75K/QosMrAdSW
9MUgNTP52GE24HGNtLi1qoJFlcDyqSMo59ahy2cI2qBDLKobkx/J3vWraV0T9VuG
WCLKTVXkcGdtwlfFRjlBz4pYg1htmf5X6DYO8A4jqv2Il9DjXA6USbW1FzXSLr9O
he8Y4IWS6wY7bCkjCWDcRQJMEhg76fsO3txE+FiYruq9RUWhiF1myv4Q6W+CyBFC
Dfvp7OOGAN6dEOM4+qR9sdjoSYKEBpsr6GtPAQw4dy753ec5
-----END CERTIFICATE-----
what's wrong?
If I simply ask openssl to verify, it is unhappy:
18:09UTC:root@bard:~/TEMP# openssl verify -purpose sslserver -CAfile ./chain.pem ./cert.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error ./cert.pem: verification failed
…does that mean the "ISRG Root X1" CA Certificate isn't installed actually installed in local trust store?
muddying the waters
What if I simplify the chain and just include the intermediate cert signed by the newer "ISRG Root X1". So, just the "MIIFFj…" in the chain file…
18:21UTC:root@bard:~/TEMP# openssl verify -purpose sslserver -CAfile ./chainISRGX1.pem ./cert.pem
./cert.pem: OK
18:22UTC:root@bard:~/TEMP# echo $?
0
Happy!
Which leads me to believe that it's the presence of the second, certificate in the chain file that causes the problem.
…but my Openssl has the trusted_first option enabled.