Openssl is unable to get local issuer certificate ever since DST Root X3 expired

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: tmp.heconomics.org

I ran this command: openssl verify -CAfile /etc/ssl/certs/ca-bundle.crt /etc/letsencrypt/live/tmp.heconomics.org/chain.pem

It produced this output:
CONNECTED(00000004)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = tmp.heconomics.org
verify return:1

Certificate chain
0 s:CN = tmp.heconomics.org
i:C = US, O = Let's Encrypt, CN = R3
1 s:C = US, O = Let's Encrypt, CN = R3
i:C = US, O = Internet Security Research Group, CN = ISRG Root X1

Server certificate
-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISA37pqweAcZnUxldB5tOBqUXCMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEyMjMyMTQ1MTNaFw0yMjAzMjMyMTQ1MTJaMB0xGzAZBgNVBAMT
EnRtcC5oZWNvbm9taWNzLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKWWhuDxvNW07g4fpDR7xZLKdjuh+MV82YX4aGXaQ3wR7k8EIlEMoBJXvUh1
WLMviqZD47tRCKgk8ek6tITelniL0gBVK/uxxPk4g4JqIQ8f5hg6dqWFxKJ0ii0w
SWDFcIVu5Nk2Qj5MQKAOnwZi2/GyePBUFHYBNkEqOYu2smZBq6Zvsxr7/chuBBTj
0xA2VAKm/4DNnFiKCRuwSAklkOfN2mmvioNb/7ljvTCzwTgYy+Wrp3wNNHc4u5D7
7ICOjpD1h0y8BR2Jdl20fuGE6jtY0xjEwGfIZuF7L4hTjKKMuiKM1EZwC1rQaGoK
StGe/geiMsJgx4QLlhMIT9hnowUCAwEAAaOCAkwwggJIMA4GA1UdDwEB/wQEAwIF
oDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAd
BgNVHQ4EFgQUWREKc/9phYBRX7a+ym42rCW1ngswHwYDVR0jBBgwFoAUFC6zF7dY
VsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRw
Oi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNy
Lm9yZy8wHQYDVR0RBBYwFIISdG1wLmhlY29ub21pY3Mub3JnMEwGA1UdIARFMEMw
CAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9j
cHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUARqVV
63X6kSAwtaKJafTzfREsQXS+/Um4havy/HD+bUcAAAF96XhGjQAABAMARjBEAiBE
o8FuoTq4Teyhz6zfcoHf73wuj00AoHOv3M94sJtnBgIgKLAu0/N5GiXnUXNf3GH0
lRKEqHTS2rnp+uKcmAVxHlQAdgBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkGjbII
mjfZEwAAAX3peEfmAAAEAwBHMEUCIQCE43ansUL5/mH1i4joINHpGb1ZP2UMcpAf
8K0fy0TUWAIgCZUKWqiX9FuCFPfjMpkOASweiEJPiyMuKigeXZYNwYowDQYJKoZI
hvcNAQELBQADggEBAFIzGmCMWef58Jo/kYXKIf9oLckFxGnm7n9gitRG1yMMzBch
hOXHtoy/Nd79ao8BSDveD1NUEDXgEkv/RcB8LhKD0/Dk7tqT7wyYtdc+OXAOLP5r
6htM9BRKUJaLQcjrzPoUzvCzenugAp71F2XNCJBsLZ13orSgcLePUz8A6lV44OpC
CA+5/DP3FQe78IWxAJRxChs58cDWPJrg3IjWMd2JHKtM0/zhuVibR7u5/iLPD8JG
IYZkwDSxGd1TB9h1foc9m2Vg/ghSKSGYbDa+wwDELU9ApTkBc8qSbRM/21kM+v9r
7qrgXivrJ0dwmIcUJNM1mG+mhtkdVE+usYbEqN8=
-----END CERTIFICATE-----
subject=CN = tmp.heconomics.org

issuer=C = US, O = Let's Encrypt, CN = R3

No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits

SSL handshake has read 3340 bytes and written 446 bytes
Verification error: unable to get local issuer certificate

New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol : TLSv1.2
Cipher : ECDHE-RSA-AES256-GCM-SHA384
Session-ID: B9876524022F49A174751F5A83200C324311B7E92A755530B63BB4DD95E540FD
Session-ID-ctx:
Master-Key: 2B925A2DA40C9C3B24A8F24ED52656AC1ED3E606EC3AC53A8AB1F606B1FA99EF40C6784905216B212E22A9323E0D0474
PSK identity: None
PSK identity hint: None
SRP username: None
TLS session ticket lifetime hint: 300 (seconds)
TLS session ticket:
0000 - 25 79 88 1b 8f a6 4e b0-64 d7 63 a0 d6 b5 6c 79 %y....N.d.c...ly
0010 - e9 dc 59 89 50 81 41 8f-83 e5 23 30 98 10 60 0e ..Y.P.A...#0... 0020 - d6 42 e6 ab 3f 78 79 db-e0 83 06 96 43 ff f7 73 .B..?xy.....C..s 0030 - 2d 38 3c fb 21 23 50 68-d3 42 a4 13 20 07 a3 9d -8<.!#Ph.B.. ... 0040 - d3 85 96 21 5f 70 e5 61-9c e4 2e 34 34 c6 1b 6c ...!_p.a...44..l 0050 - 07 2d 8f e4 be ca 12 7d-03 da e8 7d fa 61 38 3c .-.....}...}.a8< 0060 - 33 32 81 90 ef 98 c5 a7-26 9c 39 30 6d 8f fc fd 32......&.90m... 0070 - df 06 41 0d 1d 63 e5 2b-69 c2 0a 7e cb 9d 71 26 ..A..c.+i..~..q& 0080 - 2d 3a 04 1a e8 b7 df 6e-78 14 ae ef fe a9 cb b4 -:.....nx....... 0090 - d0 55 64 2e a9 9f 47 de-3b d1 e2 20 d9 f1 9b 69 .Ud...G.;.. ...i 00a0 - 00 6d b9 e4 ca 94 57 cb-01 ae d3 d0 f8 e1 bb a5 .m....W......... 00b0 - 02 e4 aa aa 47 fa 9d 42-b9 fa 96 bd 07 b1 14 ff ....G..B........ 00c0 - 5a 0e 76 da d0 88 7e 4e-b5 bf 32 c7 60 16 a9 3b Z.v...~N..2...;

Start Time: 1640304973
Timeout   : 7200 (sec)
Verify return code: 20 (unable to get local issuer certificate)
Extended master secret: no

My web server is (include version):
Server version: Apache/2.4.6 (CentOS)
Server built: Nov 10 2021 14:26:31

The operating system my web server runs on is (include version): CentOS Linux release 7.9.2009 (Core)

My hosting provider, if applicable, is: Brownrice

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 1.22.0

1 Like
2

Sorry - I hadn't completed my topic - new to this forum.

We typically create keystore files by using the openssl verify command. However, since October 1, this command has failed. We upgraded to Centos7, installed openssl 1.1.1k in /usr/local/ssl, installed certbot via pip, installed ca-certificates, downloaded chain of trust pem files from let's encrypt, issued update-ca-trust extract command, tried applying --preferred-chain-option to certbot, etc.

But, the end result is always the same -- 'unable to get local issuer certificate'

I reset the list of trusted CA certificates according to these instructions:

I am not a sys admin person, so I don't know where else to look. I have scrubbed all posts here and followed directions that worked for other folks, but sadly not for me. HELP MR. WIZARD!

1 Like
3

Please show:
ls -l /etc/ssl/certs/ca-bundle.crt
ls -l /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

and also the entire file:
/etc/letsencrypt/live/tmp.heconomics.org/chain.pem

1 Like
4

Thank you for the quick response --

[root@www anchors]# ls -l /etc/ssl/certs/ca-bundle.crt
lrwxrwxrwx 1 root root 49 Dec 23 19:57 /etc/ssl/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

[root@www anchors]# ls -l /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-r--r--r-- 1 root root 203131 Dec 24 00:14 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

/etc/letsencrypt/live/tmp.heconomics.org/chain.pem:
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

1 Like
5

Well, maybe I am missing something obvious but why would an openssl verify of a local file show an error saying connected? The error message seems more suited to an s_client request. Can you double-check the command and error syntax?

I do not see a similar error format with either 1.0.1 or 1.1.1 versions of openssl

2 Likes
6

Yes, cut/paste error. Thank you for spotting!

FWIW: I last ran certbot as standalone with the --preferred-chain 'ISRG Root X1'
Now there are only 2 certificates in the chain. On production, there's 3 certificates.

[root@www admin]# openssl verify -CAfile /etc/ssl/certs/ca-bundle.crt /etc/letsencrypt/live/tmp.heconomics.org/chain.pem
C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 1 depth lookup: unable to get issuer certificate
error /etc/letsencrypt/live/tmp.heconomics.org/chain.pem: verification failed

[root@www admin]# openssl s_client -connect tmp.heconomics.org:443
CONNECTED(00000004)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = tmp.heconomics.org
verify return:1
---
Certificate chain
 0 s:CN = tmp.heconomics.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISA37pqweAcZnUxldB5tOBqUXCMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEyMjMyMTQ1MTNaFw0yMjAzMjMyMTQ1MTJaMB0xGzAZBgNVBAMT
EnRtcC5oZWNvbm9taWNzLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKWWhuDxvNW07g4fpDR7xZLKdjuh+MV82YX4aGXaQ3wR7k8EIlEMoBJXvUh1
WLMviqZD47tRCKgk8ek6tITelniL0gBVK/uxxPk4g4JqIQ8f5hg6dqWFxKJ0ii0w
SWDFcIVu5Nk2Qj5MQKAOnwZi2/GyePBUFHYBNkEqOYu2smZBq6Zvsxr7/chuBBTj
0xA2VAKm/4DNnFiKCRuwSAklkOfN2mmvioNb/7ljvTCzwTgYy+Wrp3wNNHc4u5D7
7ICOjpD1h0y8BR2Jdl20fuGE6jtY0xjEwGfIZuF7L4hTjKKMuiKM1EZwC1rQaGoK
StGe/geiMsJgx4QLlhMIT9hnowUCAwEAAaOCAkwwggJIMA4GA1UdDwEB/wQEAwIF
oDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAd
BgNVHQ4EFgQUWREKc/9phYBRX7a+ym42rCW1ngswHwYDVR0jBBgwFoAUFC6zF7dY
VsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRw
Oi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNy
Lm9yZy8wHQYDVR0RBBYwFIISdG1wLmhlY29ub21pY3Mub3JnMEwGA1UdIARFMEMw
CAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9j
cHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUARqVV
63X6kSAwtaKJafTzfREsQXS+/Um4havy/HD+bUcAAAF96XhGjQAABAMARjBEAiBE
o8FuoTq4Teyhz6zfcoHf73wuj00AoHOv3M94sJtnBgIgKLAu0/N5GiXnUXNf3GH0
lRKEqHTS2rnp+uKcmAVxHlQAdgBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkGjbII
mjfZEwAAAX3peEfmAAAEAwBHMEUCIQCE43ansUL5/mH1i4joINHpGb1ZP2UMcpAf
8K0fy0TUWAIgCZUKWqiX9FuCFPfjMpkOASweiEJPiyMuKigeXZYNwYowDQYJKoZI
hvcNAQELBQADggEBAFIzGmCMWef58Jo/kYXKIf9oLckFxGnm7n9gitRG1yMMzBch
hOXHtoy/Nd79ao8BSDveD1NUEDXgEkv/RcB8LhKD0/Dk7tqT7wyYtdc+OXAOLP5r
6htM9BRKUJaLQcjrzPoUzvCzenugAp71F2XNCJBsLZ13orSgcLePUz8A6lV44OpC
CA+5/DP3FQe78IWxAJRxChs58cDWPJrg3IjWMd2JHKtM0/zhuVibR7u5/iLPD8JG
IYZkwDSxGd1TB9h1foc9m2Vg/ghSKSGYbDa+wwDELU9ApTkBc8qSbRM/21kM+v9r
7qrgXivrJ0dwmIcUJNM1mG+mhtkdVE+usYbEqN8=
-----END CERTIFICATE-----
subject=CN = tmp.heconomics.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3340 bytes and written 446 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 0B265EC11E00D299E399C71A1C892524B52C45AB78BE2456B3019F27EFE26626
    Session-ID-ctx: 
    Master-Key: 0A7034C4BC3759854DE69DD67824A836DBD4EBFFD8E569CEA01B54E75CD2BEE2953B2183722B872C9AB14D2C250FA184
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 25 79 88 1b 8f a6 4e b0-64 d7 63 a0 d6 b5 6c 79   %y....N.d.c...ly
    0010 - 0b 4c 38 9c e1 f0 95 9e-cf fa 6c 97 17 8e 55 24   .L8.......l...U$
    0020 - ce 53 c8 fd d3 b6 ac 22-e7 d4 8a 1b a8 9e 6a 15   .S....."......j.
    0030 - 9a 4a 68 c4 f3 bc bd 18-38 6d 72 4b 2f e9 5e 31   .Jh.....8mrK/.^1
    0040 - 12 a3 51 06 88 54 37 e0-09 7f f9 67 2b 57 2b 00   ..Q..T7....g+W+.
    0050 - cb 11 e0 1e 69 d3 a7 a1-72 cb 0f 77 88 0e 64 f6   ....i...r..w..d.
    0060 - fd 74 b9 ee 4a ac 94 ea-41 a2 25 eb 24 3e dc ed   .t..J...A.%.$>..
    0070 - 36 2b 4d c0 c6 0c 05 2b-1e a0 68 7d 93 c1 61 0e   6+M....+..h}..a.
    0080 - 3b 64 5a 82 0d 33 29 ae-d4 36 92 28 d4 77 9f 41   ;dZ..3)..6.(.w.A
    0090 - 7c 13 64 ba 6e 25 10 6c-83 c7 07 0b 9c 4d 83 b2   |.d.n%.l.....M..
    00a0 - 08 6b 9b ba 95 b1 07 01-35 78 8d 8d ad fd f9 b7   .k......5x......
    00b0 - f6 4e 9b b4 ca 27 98 22-9d fd 7b 06 fd 9f d1 83   .N...'."..{.....
    00c0 - 51 ed 5f 43 60 75 d4 32-a9 84 3b 82 94 26 23 6c   Q._C`u.2..;..&#l

    Start Time: 1640315186
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
---
closed
2 Likes
7

Ok, your chain looks fine from my server and also a 3rd party checker like this:
https://decoder.link/sslchecker/tmp.heconomics.org/443

So, why does it fail to verify on your server. Almost always this is because your CA store is out of date. By the date of the ls command it looks fine but can we check:

grep -Ei 'ISRG Root|DST Root|R3' /etc/ssl/certs/ca-bundle.crt | grep '#'
1 Like
8 
[root@www admin]# grep -Ei 'ISRG Root|DST Root|R3' /etc/ssl/certs/ca-bundle.crt | grep '#'
# ISRG Root X1
# R3
# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1
1 Like
9

Hello, in my efforts to try to fix this, I also downloaded these files and placed them in the anchors folder, followed by update-ca-trust extract

[root@www anchors]# pwd
/etc/pki/ca-trust/source/anchors
[root@www anchors]# ls -la
total 16
drwxr-xr-x 2 root root 4096 Dec 24 00:14 .
drwxr-xr-x 4 root root 4096 Dec 23 19:57 ..
-rw-r--r-- 1 root root    0 Dec 24 00:14 DST-Root-CA-X3.pem
-rw-r--r-- 1 root root 1380 Dec 23 23:58 isrg-root-x1-cross-signed.der
-rw-r--r-- 1 root root 1826 Dec 24 00:11 lets-encrypt-r3.pem
1 Like
11

That explains the date::

and the double "ISRG" entries:

I show:

ls -l /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-r--r--r--. 1 root root 199360 Oct  1 13:24 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

grep -Ei 'ISRG Root|DST Root|R3' /etc/ssl/certs/ca-bundle.crt | grep '#'
# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1

This is a zero byte file:

This is an intermediate cert and should never be added to any trust system:

2 Likes
12

Ah, I see Rudy just posted most of what I was just going to post. And, I am checking out for the night so he can help you get it working.

I would add that, yes, you need an ISRG Root X1 but I don't think the one you added is appropriate. It is easy enough to obtain again so I would remove all 3 of these and rebuild your CA store. That should give you, I think, the latest store based on updates for Centos 7. I thought they had packaged updates for all this so should be fine (I think).

2 Likes
13

Thank you MikeMcQ and Rudy --

So many attempts, I think I put those pem & der files into the anchors folder out of desperation!

I removed them from the anchors folder, followed by 'update-ca-trust extract' (was this needed?)
Here goes:

[root@www anchors]# grep -Ei 'ISRG Root|DST Root|R3' /etc/ssl/certs/ca-bundle.crt | grep '#'
# GTS Root R3
# GlobalSign Root CA - R3
# ISRG Root X1

[root@www anchors]# openssl verify -CAfile /etc/ssl/certs/ca-bundle.crt /etc/letsencrypt/live/tmp.heconomics.org/chain.pem
/etc/letsencrypt/live/tmp.heconomics.org/chain.pem: OK

To create the keystore, I need to verify the server certificate. Not sure why this fails but chain.pem didn't:

[root@www anchors]# openssl verify -CAfile /etc/ssl/certs/ca-bundle.crt /etc/letsencrypt/live/tmp.heconomics.org/cert.pem
CN = tmp.heconomics.org
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/letsencrypt/live/tmp.heconomics.org/cert.pem: verification failed

Next, I restarted apache, service httpd restart, but this also gives me the error:

[root@www anchors]# openssl s_client -connect tmp.heconomics.org:443
CONNECTED(00000004)
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 CN = tmp.heconomics.org
verify return:1
---
Certificate chain
 0 s:CN = tmp.heconomics.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIFKTCCBBGgAwIBAgISA37pqweAcZnUxldB5tOBqUXCMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTEyMjMyMTQ1MTNaFw0yMjAzMjMyMTQ1MTJaMB0xGzAZBgNVBAMT
EnRtcC5oZWNvbm9taWNzLm9yZzCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoC
ggEBAKWWhuDxvNW07g4fpDR7xZLKdjuh+MV82YX4aGXaQ3wR7k8EIlEMoBJXvUh1
WLMviqZD47tRCKgk8ek6tITelniL0gBVK/uxxPk4g4JqIQ8f5hg6dqWFxKJ0ii0w
SWDFcIVu5Nk2Qj5MQKAOnwZi2/GyePBUFHYBNkEqOYu2smZBq6Zvsxr7/chuBBTj
0xA2VAKm/4DNnFiKCRuwSAklkOfN2mmvioNb/7ljvTCzwTgYy+Wrp3wNNHc4u5D7
7ICOjpD1h0y8BR2Jdl20fuGE6jtY0xjEwGfIZuF7L4hTjKKMuiKM1EZwC1rQaGoK
StGe/geiMsJgx4QLlhMIT9hnowUCAwEAAaOCAkwwggJIMA4GA1UdDwEB/wQEAwIF
oDAdBgNVHSUEFjAUBggrBgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAd
BgNVHQ4EFgQUWREKc/9phYBRX7a+ym42rCW1ngswHwYDVR0jBBgwFoAUFC6zF7dY
VsuuUAlA5h+vnYsUwsYwVQYIKwYBBQUHAQEESTBHMCEGCCsGAQUFBzABhhVodHRw
Oi8vcjMuby5sZW5jci5vcmcwIgYIKwYBBQUHMAKGFmh0dHA6Ly9yMy5pLmxlbmNy
Lm9yZy8wHQYDVR0RBBYwFIISdG1wLmhlY29ub21pY3Mub3JnMEwGA1UdIARFMEMw
CAYGZ4EMAQIBMDcGCysGAQQBgt8TAQEBMCgwJgYIKwYBBQUHAgEWGmh0dHA6Ly9j
cHMubGV0c2VuY3J5cHQub3JnMIIBAwYKKwYBBAHWeQIEAgSB9ASB8QDvAHUARqVV
63X6kSAwtaKJafTzfREsQXS+/Um4havy/HD+bUcAAAF96XhGjQAABAMARjBEAiBE
o8FuoTq4Teyhz6zfcoHf73wuj00AoHOv3M94sJtnBgIgKLAu0/N5GiXnUXNf3GH0
lRKEqHTS2rnp+uKcmAVxHlQAdgBvU3asMfAxGdiZAKRRFf93FRwR2QLBACkGjbII
mjfZEwAAAX3peEfmAAAEAwBHMEUCIQCE43ansUL5/mH1i4joINHpGb1ZP2UMcpAf
8K0fy0TUWAIgCZUKWqiX9FuCFPfjMpkOASweiEJPiyMuKigeXZYNwYowDQYJKoZI
hvcNAQELBQADggEBAFIzGmCMWef58Jo/kYXKIf9oLckFxGnm7n9gitRG1yMMzBch
hOXHtoy/Nd79ao8BSDveD1NUEDXgEkv/RcB8LhKD0/Dk7tqT7wyYtdc+OXAOLP5r
6htM9BRKUJaLQcjrzPoUzvCzenugAp71F2XNCJBsLZ13orSgcLePUz8A6lV44OpC
CA+5/DP3FQe78IWxAJRxChs58cDWPJrg3IjWMd2JHKtM0/zhuVibR7u5/iLPD8JG
IYZkwDSxGd1TB9h1foc9m2Vg/ghSKSGYbDa+wwDELU9ApTkBc8qSbRM/21kM+v9r
7qrgXivrJ0dwmIcUJNM1mG+mhtkdVE+usYbEqN8=
-----END CERTIFICATE-----
subject=CN = tmp.heconomics.org

issuer=C = US, O = Let's Encrypt, CN = R3

---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: RSA
Server Temp Key: ECDH, P-256, 256 bits
---
SSL handshake has read 3340 bytes and written 446 bytes
Verification error: unable to get local issuer certificate
---
New, TLSv1.2, Cipher is ECDHE-RSA-AES256-GCM-SHA384
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES256-GCM-SHA384
    Session-ID: 6AF3FE810813F295FB8062DBEBFE88E02A8D506501F3C33834D63B6AB5131E7A
    Session-ID-ctx: 
    Master-Key: DC53D4D2A5E38C15CCCAAED19479D4E50098990327BFCE5FDAE90BEAD9B7E155E41D3EA1EB901F28FE290EAD1997C58C
    PSK identity: None
    PSK identity hint: None
    SRP username: None
    TLS session ticket lifetime hint: 300 (seconds)
    TLS session ticket:
    0000 - 25 79 88 1b 8f a6 4e b0-64 d7 63 a0 d6 b5 6c 79   %y....N.d.c...ly
    0010 - c9 ee 78 9e 44 1a db 9b-08 c9 29 e6 92 a5 b5 f8   ..x.D.....).....
    0020 - 2a f3 15 4e ee 23 b0 ee-13 e5 50 18 50 4f 85 be   *..N.#....P.PO..
    0030 - 22 cc 29 aa 40 b4 45 56-8f 7d e0 d5 4b ea 36 a3   ".).@.EV.}..K.6.
    0040 - 36 87 78 5a 2c c5 c6 57-d7 3d 8a 5f 21 ad 7a e8   6.xZ,..W.=._!.z.
    0050 - 0f d3 45 a9 3c 11 a8 df-b0 3f f3 1a 4e 97 3a ac   ..E.<....?..N.:.
    0060 - 27 3c ac ac c0 86 ba de-ae 99 b9 ab b5 e3 aa 54   '<.............T
    0070 - a1 3a 4d 72 f9 d6 5d 4e-9d 47 b0 17 5a 4c 9d 8b   .:Mr..]N.G..ZL..
    0080 - 28 48 fb 96 a1 f6 97 2d-6e 6a 98 14 2b bc e2 2b   (H.....-nj..+..+
    0090 - f8 55 fd 36 4e 50 ac 18-17 19 56 ed 2b 61 76 00   .U.6NP....V.+av.
    00a0 - 0d 0d 0e 00 01 57 aa 02-5f 6f 6a 6b 7a 28 90 1e   .....W.._ojkz(..
    00b0 - 88 64 60 34 c0 83 f1 96-89 7f b8 6f 48 ec d6 d4   .d`4.......oH...
    00c0 - af 2f 5f e3 a4 07 36 b6-d5 c2 6b 22 f6 56 f4 f1   ./_...6...k".V..

    Start Time: 1640322787
    Timeout   : 7200 (sec)
    Verify return code: 20 (unable to get local issuer certificate)
    Extended master secret: no
---
closed
1 Like
14

out of completeness:

[root@www anchors]# ls -l /etc/ssl/certs/ca-bundle.crt
lrwxrwxrwx 1 root root 49 Dec 23 19:57 /etc/ssl/certs/ca-bundle.crt -> /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

[root@www anchors]# ls -l /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem
-r--r--r-- 1 root root 199360 Dec 24 05:11 /etc/pki/ca-trust/extracted/pem/tls-ca-bundle.pem

please let me know if you would like to see the contents of chain.pem again

1 Like
16

Try:

echo | openssl s_client \
 -connect tmp.heconomics.org:443 \
 -CAfile /etc/ssl/certs/ca-bundle.crt | head

Then save this cert: https://letsencrypt.org/certs/isrgrootx1.pem
Like:
wget -O /var/tmp/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem
Then try:

echo | openssl s_client \
 -connect tmp.heconomics.org:443 \
 -CAfile /var/tmp/isrgrootx1.pem | head
2 Likes
17

And just for kicks:
grep 'MIIDSjCCAjKgAwIBAgIQRK' /etc/ssl/certs/ca-bundle.crt
[which should NOT find anything]

2 Likes
18

Can we also see results of this?

curl --version
openssl version
2 Likes
19 
[root@www admin]# echo | openssl s_client \
>  -connect tmp.heconomics.org:443 \
>  -CAfile /etc/ssl/certs/ca-bundle.crt | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = tmp.heconomics.org
verify return:1
CONNECTED(00000004)
---
Certificate chain
 0 s:CN = tmp.heconomics.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----

[root@www admin]# wget -O /var/tmp/isrgrootx1.pem https://letsencrypt.org/certs/isrgrootx1.pem
--2021-12-24 15:01:48--  https://letsencrypt.org/certs/isrgrootx1.pem
Resolving letsencrypt.org (letsencrypt.org)... 54.241.246.27, 54.151.57.158, 2600:1f1c:471:9d00:64a9:5908:2245:64e0, ...
Connecting to letsencrypt.org (letsencrypt.org)|54.241.246.27|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1939 (1.9K) [application/x-pem-file]
Saving to: ‘/var/tmp/isrgrootx1.pem’

100%[=================================================================================================================>] 1,939       --.-K/s   in 0s      

2021-12-24 15:01:48 (39.5 MB/s) - ‘/var/tmp/isrgrootx1.pem’ saved [1939/1939]

[root@www admin]# echo | openssl s_client \
>  -connect tmp.heconomics.org:443 \
>  -CAfile /var/tmp/isrgrootx1.pem | head
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = tmp.heconomics.org
verify return:1
CONNECTED(00000004)
---
Certificate chain
 0 s:CN = tmp.heconomics.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
1 Like
20 
[root@www admin]# grep 'MIIDSjCCAjKgAwIBAgIQRK' /etc/ssl/certs/ca-bundle.crt
1 Like
21 
[root@www admin]# curl --version
curl 7.29.0 (x86_64-redhat-linux-gnu) libcurl/7.29.0 NSS/3.53.1 zlib/1.2.7 libidn/1.28 libssh2/1.8.0
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp scp sftp smtp smtps telnet tftp 
Features: AsynchDNS GSS-Negotiate IDN IPv6 Largefile NTLM NTLM_WB SSL libz unix-sockets 
[root@www admin]# openssl version
OpenSSL 1.1.1k  25 Mar 2021
1 Like
22

Good Morning, after reviewing these results, all looks really good!

So, I tried (again) to issue the 'openssl verify' from my Keystore script, using your method, and got the following:

[root@www admin]# openssl verify -CAfile /usr/tmp/isrgrootx1.pem /etc/letsencrypt/live/tmp.heconomics.org/cert.pem
CN = tmp.heconomics.org
error 20 at 0 depth lookup: unable to get local issuer certificate
error /etc/letsencrypt/live/tmp.heconomics.org/cert.pem: verification failed

Is there something wrong with cert.perm? Did I not create it correctly with certbot?

1 Like