Certificate still showing X3 as root with openssl 1.1.01

Help
21

OK, so following up on

check the scripts that manipulate/install the cert into your system (after LE has issued them to your server - via certbot ).

I try looking at what I think is the exact information cert-bot has put onto my system,

# openssl verify -verbose -CAfile /etc/letsencrypt/live/ldap.ligo.uwm.edu/chain.pem /etc/letsencrypt/live/ldap.ligo.uwm.edu/cert.pem 
O = Digital Signature Trust Co., CN = DST Root CA X3
error 10 at 3 depth lookup: certificate has expired
error /etc/letsencrypt/live/ldap.ligo.uwm.edu/cert.pem: verification failed

Is the root cert included in chain.pem?

I should probably confess that I inherited this system, which is configured via puppet, from someone who left our organization and did not leave documentation. I have a very limited understanding of puppet and a very limited understanding of Lets Encrypt and ACME. I am not a computer scientist or a trained system administrator, I am an astrophysicist. So I can learn complex things, but I don't have the time (unfortunately) to learn all the things I need to know about all the systems my colleague left behind as well as doing my day job, so any guidance I can get is much appreciated, and I really thank everyone who has contributed to helping me so far.

22

The CSR is technically signed by it's own keypair, but that's not really relevant for this process. What matters is that the CA creates a certificate based on information supplied in the CSR, which the CA then signs.

For signing end-entity certificates ("leafs"), CA's like Let's Encrypt use so-called intermediate certificates. R3 is one of those intermediates. They're called intermediates, because they're in the middle of the chain; R3 is not a root certificate, and R3 is not included in system trust stores. Instead, R3 is a certificate signed by a root certificate (which are included in system trust stores).

There are two versions of R3:

  • R3, signed by DST Root CA X3 (a root certificate, now expired)
  • R3, signed by ISRG Root X1 (also a root certificate, not expired)

R3 is not pre-installed on your system. Rather it is usually downloaded from the ACME server by your ACME client and put into the file chain.pem.

No, because it is already present in system trust stores. Sending it over the wire would just waste bandwidth. R3 is included in that file though.

Can you show us the contents of these files, just so that we're all on the same page? (Just to ensure that the files generated by certbot are all up-to-date and well)

What I believe to be the issue is, that you have some non-certbot script that copies the certificates in an incorrect way. Specifically:

As you have shown, your certbot certificates and keys are @ /etc/letsencrypt/live/ldap.ligo.uwm.edu/*.pem. However, looking at your original LDAP output:

olcTLSCertificateFile:: L2V0Yy9sZGFwL3g1MDktY2VydHMvbGRhcC5saWdvLnV3bS5lZHVf
olcTLSCertificateKeyFile:: L2V0Yy9sZGFwL3g1MDktY2VydHMvbGRhcC5saWdvLnV3bS5lZ

This base64-decodes to

olcTLSCertificateFile:: /etc/ldap/x509-certs/ldap.ligo.uwm.edu_
olcTLSCertificateKeyFile:: /etc/ldap/x509-certs/ldap.ligo.uwm.e

(The filenames are weird looking, maybe the base64 was cut-off early?)

Anyway, looking at this, there must be some script which copies the certificates from /etc/letsencrypt/live/ to /etc/ldap/x509-certs. This script may be the culprit of your chain issues*. Can you additonally check/show us the files in /etc/ldap/x509-certs?

*I have a suspicion, that the olcTLSCertificateFile is a copy of /etc/letsencrypt/live/ldap.ligo.uwm.edu/cert.pem. However, based on my openldap experience, it should instead be a copy of /etc/letsencrypt/live/ldap.ligo.uwm.edu/fullchain.pem. If you show us the files, we can check if this is the case.

2 Likes
23

OK, I hope this is what you mean by "show".

For cert.pem:

# openssl x509 -in /etc/letsencrypt/live/ldap.ligo.uwm.edu/cert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:8f:be:4a:16:91:6b:d1:02:e0:04:aa:53:a9:6e:89:bf:ab
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Aug 10 12:00:16 2021 GMT
            Not After : Nov  8 12:00:14 2021 GMT
        Subject: CN = ldap.ligo.uwm.edu
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bb:c9:67:ee:cb:8e:fc:9b:d2:1a:d5:e1:b9:70:
                    51:48:70:50:b1:ee:f4:78:e8:9c:78:48:5e:b8:2c:
                    d2:e1:b8:64:11:41:a5:86:c0:16:4f:5f:fb:d7:8b:
                    ce:77:2d:5e:66:9c:e8:6f:da:ff:73:6b:73:69:46:
                    c5:9e:1c:b0:3a:31:6e:6c:c8:81:0a:e0:d4:b4:d3:
                    ee:c9:e9:24:fc:32:82:5f:01:62:4a:e6:c9:71:13:
                    ac:40:db:b4:52:1f:3a:e9:d0:a6:76:62:38:e2:ff:
                    ac:1f:49:c5:23:86:38:44:bc:c8:c7:e4:d9:af:4a:
                    3a:a5:38:d8:5c:d2:1c:49:bb:ba:3c:79:d7:a7:be:
                    99:6d:cb:16:71:5e:92:9f:89:92:6f:66:8c:2f:6b:
                    e7:88:07:34:3b:2d:f7:11:3c:b0:0d:40:f3:1d:e5:
                    de:a0:1b:3c:aa:d8:eb:a3:ce:20:39:c3:e4:cd:98:
                    e1:7d:28:76:ca:f7:bc:14:97:14:2a:47:99:d4:e8:
                    19:74:08:d8:f1:30:0d:cc:9b:39:00:f4:56:91:74:
                    92:94:ce:c2:f5:e0:54:e1:14:c5:7e:94:06:c2:3b:
                    74:0e:c0:79:b2:27:fa:0e:ce:44:27:5c:68:1c:b6:
                    3a:ac:fb:3a:d4:0a:5c:6b:08:01:89:67:59:ca:9e:
                    7e:8d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                F3:1A:67:79:8D:55:06:EC:B5:76:71:C8:F1:3F:D6:68:CB:4C:C6:10
            X509v3 Authority Key Identifier: 
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

            Authority Information Access: 
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

            X509v3 Subject Alternative Name: 
                DNS:ldap.ligo.uwm.edu
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 94:20:BC:1E:8E:D5:8D:6C:88:73:1F:82:8B:22:2C:0D:
                                D1:DA:4D:5E:6C:4F:94:3D:61:DB:4E:2F:58:4D:A2:C2
                    Timestamp : Aug 10 13:00:16.123 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:CD:E4:22:4C:59:B5:60:C1:31:1C:86:
                                FA:3A:17:D0:7F:BB:27:19:DC:67:69:6F:EA:D7:E9:C7:
                                F0:E8:F4:E7:40:02:21:00:9A:C1:19:0E:91:2A:0E:A5:
                                E3:39:01:7F:09:7B:37:A3:92:96:99:A5:3C:A1:A8:D6:
                                6C:C4:16:B1:56:0F:BB:18
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : F6:5C:94:2F:D1:77:30:22:14:54:18:08:30:94:56:8E:
                                E3:4D:13:19:33:BF:DF:0C:2F:20:0B:CC:4E:F1:64:E3
                    Timestamp : Aug 10 13:00:16.107 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:FC:A4:E9:FA:5B:49:88:99:EC:96:2B:
                                B9:3C:49:80:4C:70:93:6D:7F:10:C2:1F:AF:43:6A:63:
                                FE:F3:C9:23:34:02:21:00:E0:AA:99:EA:85:78:3A:23:
                                C6:6B:8A:4D:AF:A3:63:27:DD:09:E8:F1:51:44:5F:D7:
                                C0:09:BE:29:78:2B:48:E6
    Signature Algorithm: sha256WithRSAEncryption
         62:16:6b:5e:2b:e2:01:6f:df:d1:c9:62:13:3d:e9:0a:26:f3:
         67:87:0c:29:e8:89:46:93:4a:13:40:7c:2b:1a:6d:0d:62:64:
         ff:03:53:8e:9b:47:b4:b8:d6:af:ac:34:52:79:ce:17:a9:89:
         59:60:62:76:e4:44:99:28:4e:69:db:d5:01:d5:95:9e:60:66:
         5c:37:2f:73:4b:70:c8:b0:55:d1:a5:b8:9d:bb:31:85:87:d2:
         4d:e7:58:82:48:d1:c8:1c:4a:d8:79:7d:53:e9:e6:e2:e9:07:
         bd:61:72:6b:6e:ee:4d:85:0c:a5:70:d2:7b:93:1a:eb:ae:7a:
         95:99:fd:ec:a8:ef:ae:b6:92:21:84:ab:a8:fd:1d:d1:9d:35:
         1c:93:55:27:eb:7f:60:6b:7a:61:a1:4e:9b:8e:51:06:07:74:
         e6:db:03:6b:03:34:d4:2a:03:37:31:ad:68:ab:2a:ae:1d:af:
         78:4c:09:6f:24:75:fe:7d:07:85:c7:1e:eb:3f:0e:86:58:84:
         dc:d0:8b:52:5c:bc:c5:b2:45:75:93:16:1e:cc:73:3b:1c:01:
         4a:6b:dd:4f:df:fd:b4:b9:94:93:86:16:8a:09:21:5d:fe:93:
         8a:3d:d8:fe:8d:31:06:14:d2:5f:a6:72:9d:23:d4:53:f0:c4:
         56:a1:d3:34
-----BEGIN CERTIFICATE-----
MIIFKjCCBBKgAwIBAgISBI++ShaRa9EC4ASqU6luib+rMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA4MTAxMjAwMTZaFw0yMTExMDgxMjAwMTRaMBwxGjAYBgNVBAMT
EWxkYXAubGlnby51d20uZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAu8ln7suO/JvSGtXhuXBRSHBQse70eOiceEheuCzS4bhkEUGlhsAWT1/714vO
dy1eZpzob9r/c2tzaUbFnhywOjFubMiBCuDUtNPuyekk/DKCXwFiSubJcROsQNu0
Uh866dCmdmI44v+sH0nFI4Y4RLzIx+TZr0o6pTjYXNIcSbu6PHnXp76ZbcsWcV6S
n4mSb2aML2vniAc0Oy33ETywDUDzHeXeoBs8qtjro84gOcPkzZjhfSh2yve8FJcU
KkeZ1OgZdAjY8TANzJs5APRWkXSSlM7C9eBU4RTFfpQGwjt0DsB5sif6Ds5EJ1xo
HLY6rPs61ApcawgBiWdZyp5+jQIDAQABo4ICTjCCAkowDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBTzGmd5jVUG7LV2ccjxP9Zoy0zGEDAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghFsZGFwLmxpZ28udXdtLmVkdTBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQYGCisGAQQB1nkCBAIEgfcEgfQA8gB3AJQgvB6O
1Y1siHMfgosiLA3R2k1ebE+UPWHbTi9YTaLCAAABezAmN3sAAAQDAEgwRgIhAM3k
IkxZtWDBMRyG+joX0H+7JxncZ2lv6tfpx/Do9OdAAiEAmsEZDpEqDqXjOQF/CXs3
o5KWmaU8oajWbMQWsVYPuxgAdwD2XJQv0XcwIhRUGAgwlFaO400TGTO/3wwvIAvM
TvFk4wAAAXswJjdrAAAEAwBIMEYCIQD8pOn6W0mImeyWK7k8SYBMcJNtfxDCH69D
amP+88kjNAIhAOCqmeqFeDojxmuKTa+jYyfdCejxUURf18AJvil4K0jmMA0GCSqG
SIb3DQEBCwUAA4IBAQBiFmteK+IBb9/RyWITPekKJvNnhwwp6IlGk0oTQHwrGm0N
YmT/A1OOm0e0uNavrDRSec4XqYlZYGJ25ESZKE5p29UB1ZWeYGZcNy9zS3DIsFXR
pbiduzGFh9JN51iCSNHIHErYeX1T6ebi6Qe9YXJrbu5NhQylcNJ7kxrrrnqVmf3s
qO+utpIhhKuo/R3RnTUck1Un639ga3phoU6bjlEGB3Tm2wNrAzTUKgM3Ma1oqyqu
Ha94TAlvJHX+fQeFxx7rPw6GWITc0ItSXLzFskV1kxYezHM7HAFKa91P3/20uZST
hhaKCSFd/pOKPdj+jTEGFNJfpnKdI9RT8MRWodM0
-----END CERTIFICATE-----

For chain.pem, which does indeed show the correct current chain:

# openssl x509 -in /etc/letsencrypt/live/ldap.ligo.uwm.edu/cert.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            04:8f:be:4a:16:91:6b:d1:02:e0:04:aa:53:a9:6e:89:bf:ab
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Let's Encrypt, CN = R3
        Validity
            Not Before: Aug 10 12:00:16 2021 GMT
            Not After : Nov  8 12:00:14 2021 GMT
        Subject: CN = ldap.ligo.uwm.edu
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bb:c9:67:ee:cb:8e:fc:9b:d2:1a:d5:e1:b9:70:
                    51:48:70:50:b1:ee:f4:78:e8:9c:78:48:5e:b8:2c:
                    d2:e1:b8:64:11:41:a5:86:c0:16:4f:5f:fb:d7:8b:
                    ce:77:2d:5e:66:9c:e8:6f:da:ff:73:6b:73:69:46:
                    c5:9e:1c:b0:3a:31:6e:6c:c8:81:0a:e0:d4:b4:d3:
                    ee:c9:e9:24:fc:32:82:5f:01:62:4a:e6:c9:71:13:
                    ac:40:db:b4:52:1f:3a:e9:d0:a6:76:62:38:e2:ff:
                    ac:1f:49:c5:23:86:38:44:bc:c8:c7:e4:d9:af:4a:
                    3a:a5:38:d8:5c:d2:1c:49:bb:ba:3c:79:d7:a7:be:
                    99:6d:cb:16:71:5e:92:9f:89:92:6f:66:8c:2f:6b:
                    e7:88:07:34:3b:2d:f7:11:3c:b0:0d:40:f3:1d:e5:
                    de:a0:1b:3c:aa:d8:eb:a3:ce:20:39:c3:e4:cd:98:
                    e1:7d:28:76:ca:f7:bc:14:97:14:2a:47:99:d4:e8:
                    19:74:08:d8:f1:30:0d:cc:9b:39:00:f4:56:91:74:
                    92:94:ce:c2:f5:e0:54:e1:14:c5:7e:94:06:c2:3b:
                    74:0e:c0:79:b2:27:fa:0e:ce:44:27:5c:68:1c:b6:
                    3a:ac:fb:3a:d4:0a:5c:6b:08:01:89:67:59:ca:9e:
                    7e:8d
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Key Encipherment
            X509v3 Extended Key Usage: 
                TLS Web Server Authentication, TLS Web Client Authentication
            X509v3 Basic Constraints: critical
                CA:FALSE
            X509v3 Subject Key Identifier: 
                F3:1A:67:79:8D:55:06:EC:B5:76:71:C8:F1:3F:D6:68:CB:4C:C6:10
            X509v3 Authority Key Identifier: 
                keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6

            Authority Information Access: 
                OCSP - URI:http://r3.o.lencr.org
                CA Issuers - URI:http://r3.i.lencr.org/

            X509v3 Subject Alternative Name: 
                DNS:ldap.ligo.uwm.edu
            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1
                  CPS: http://cps.letsencrypt.org

            CT Precertificate SCTs: 
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : 94:20:BC:1E:8E:D5:8D:6C:88:73:1F:82:8B:22:2C:0D:
                                D1:DA:4D:5E:6C:4F:94:3D:61:DB:4E:2F:58:4D:A2:C2
                    Timestamp : Aug 10 13:00:16.123 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:CD:E4:22:4C:59:B5:60:C1:31:1C:86:
                                FA:3A:17:D0:7F:BB:27:19:DC:67:69:6F:EA:D7:E9:C7:
                                F0:E8:F4:E7:40:02:21:00:9A:C1:19:0E:91:2A:0E:A5:
                                E3:39:01:7F:09:7B:37:A3:92:96:99:A5:3C:A1:A8:D6:
                                6C:C4:16:B1:56:0F:BB:18
                Signed Certificate Timestamp:
                    Version   : v1 (0x0)
                    Log ID    : F6:5C:94:2F:D1:77:30:22:14:54:18:08:30:94:56:8E:
                                E3:4D:13:19:33:BF:DF:0C:2F:20:0B:CC:4E:F1:64:E3
                    Timestamp : Aug 10 13:00:16.107 2021 GMT
                    Extensions: none
                    Signature : ecdsa-with-SHA256
                                30:46:02:21:00:FC:A4:E9:FA:5B:49:88:99:EC:96:2B:
                                B9:3C:49:80:4C:70:93:6D:7F:10:C2:1F:AF:43:6A:63:
                                FE:F3:C9:23:34:02:21:00:E0:AA:99:EA:85:78:3A:23:
                                C6:6B:8A:4D:AF:A3:63:27:DD:09:E8:F1:51:44:5F:D7:
                                C0:09:BE:29:78:2B:48:E6
    Signature Algorithm: sha256WithRSAEncryption
         62:16:6b:5e:2b:e2:01:6f:df:d1:c9:62:13:3d:e9:0a:26:f3:
         67:87:0c:29:e8:89:46:93:4a:13:40:7c:2b:1a:6d:0d:62:64:
         ff:03:53:8e:9b:47:b4:b8:d6:af:ac:34:52:79:ce:17:a9:89:
         59:60:62:76:e4:44:99:28:4e:69:db:d5:01:d5:95:9e:60:66:
         5c:37:2f:73:4b:70:c8:b0:55:d1:a5:b8:9d:bb:31:85:87:d2:
         4d:e7:58:82:48:d1:c8:1c:4a:d8:79:7d:53:e9:e6:e2:e9:07:
         bd:61:72:6b:6e:ee:4d:85:0c:a5:70:d2:7b:93:1a:eb:ae:7a:
         95:99:fd:ec:a8:ef:ae:b6:92:21:84:ab:a8:fd:1d:d1:9d:35:
         1c:93:55:27:eb:7f:60:6b:7a:61:a1:4e:9b:8e:51:06:07:74:
         e6:db:03:6b:03:34:d4:2a:03:37:31:ad:68:ab:2a:ae:1d:af:
         78:4c:09:6f:24:75:fe:7d:07:85:c7:1e:eb:3f:0e:86:58:84:
         dc:d0:8b:52:5c:bc:c5:b2:45:75:93:16:1e:cc:73:3b:1c:01:
         4a:6b:dd:4f:df:fd:b4:b9:94:93:86:16:8a:09:21:5d:fe:93:
         8a:3d:d8:fe:8d:31:06:14:d2:5f:a6:72:9d:23:d4:53:f0:c4:
         56:a1:d3:34
-----BEGIN CERTIFICATE-----
MIIFKjCCBBKgAwIBAgISBI++ShaRa9EC4ASqU6luib+rMA0GCSqGSIb3DQEBCwUA
MDIxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MQswCQYDVQQD
EwJSMzAeFw0yMTA4MTAxMjAwMTZaFw0yMTExMDgxMjAwMTRaMBwxGjAYBgNVBAMT
EWxkYXAubGlnby51d20uZWR1MIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKC
AQEAu8ln7suO/JvSGtXhuXBRSHBQse70eOiceEheuCzS4bhkEUGlhsAWT1/714vO
dy1eZpzob9r/c2tzaUbFnhywOjFubMiBCuDUtNPuyekk/DKCXwFiSubJcROsQNu0
Uh866dCmdmI44v+sH0nFI4Y4RLzIx+TZr0o6pTjYXNIcSbu6PHnXp76ZbcsWcV6S
n4mSb2aML2vniAc0Oy33ETywDUDzHeXeoBs8qtjro84gOcPkzZjhfSh2yve8FJcU
KkeZ1OgZdAjY8TANzJs5APRWkXSSlM7C9eBU4RTFfpQGwjt0DsB5sif6Ds5EJ1xo
HLY6rPs61ApcawgBiWdZyp5+jQIDAQABo4ICTjCCAkowDgYDVR0PAQH/BAQDAgWg
MB0GA1UdJQQWMBQGCCsGAQUFBwMBBggrBgEFBQcDAjAMBgNVHRMBAf8EAjAAMB0G
A1UdDgQWBBTzGmd5jVUG7LV2ccjxP9Zoy0zGEDAfBgNVHSMEGDAWgBQULrMXt1hW
y65QCUDmH6+dixTCxjBVBggrBgEFBQcBAQRJMEcwIQYIKwYBBQUHMAGGFWh0dHA6
Ly9yMy5vLmxlbmNyLm9yZzAiBggrBgEFBQcwAoYWaHR0cDovL3IzLmkubGVuY3Iu
b3JnLzAcBgNVHREEFTATghFsZGFwLmxpZ28udXdtLmVkdTBMBgNVHSAERTBDMAgG
BmeBDAECATA3BgsrBgEEAYLfEwEBATAoMCYGCCsGAQUFBwIBFhpodHRwOi8vY3Bz
LmxldHNlbmNyeXB0Lm9yZzCCAQYGCisGAQQB1nkCBAIEgfcEgfQA8gB3AJQgvB6O
1Y1siHMfgosiLA3R2k1ebE+UPWHbTi9YTaLCAAABezAmN3sAAAQDAEgwRgIhAM3k
IkxZtWDBMRyG+joX0H+7JxncZ2lv6tfpx/Do9OdAAiEAmsEZDpEqDqXjOQF/CXs3
o5KWmaU8oajWbMQWsVYPuxgAdwD2XJQv0XcwIhRUGAgwlFaO400TGTO/3wwvIAvM
TvFk4wAAAXswJjdrAAAEAwBIMEYCIQD8pOn6W0mImeyWK7k8SYBMcJNtfxDCH69D
amP+88kjNAIhAOCqmeqFeDojxmuKTa+jYyfdCejxUURf18AJvil4K0jmMA0GCSqG
SIb3DQEBCwUAA4IBAQBiFmteK+IBb9/RyWITPekKJvNnhwwp6IlGk0oTQHwrGm0N
YmT/A1OOm0e0uNavrDRSec4XqYlZYGJ25ESZKE5p29UB1ZWeYGZcNy9zS3DIsFXR
pbiduzGFh9JN51iCSNHIHErYeX1T6ebi6Qe9YXJrbu5NhQylcNJ7kxrrrnqVmf3s
qO+utpIhhKuo/R3RnTUck1Un639ga3phoU6bjlEGB3Tm2wNrAzTUKgM3Ma1oqyqu
Ha94TAlvJHX+fQeFxx7rPw6GWITc0ItSXLzFskV1kxYezHM7HAFKa91P3/20uZST
hhaKCSFd/pOKPdj+jTEGFNJfpnKdI9RT8MRWodM0
-----END CERTIFICATE-----
root@ldap:~# openssl x509 -in /etc/letsencrypt/live/ldap.ligo.uwm.edu/chain.pem -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            91:2b:08:4a:cf:0c:18:a7:53:f6:d6:2e:25:a7:5f:5a
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: C = US, O = Internet Security Research Group, CN = ISRG Root X1
        Validity
            Not Before: Sep  4 00:00:00 2020 GMT
            Not After : Sep 15 16:00:00 2025 GMT
        Subject: C = US, O = Let's Encrypt, CN = R3
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:bb:02:15:28:cc:f6:a0:94:d3:0f:12:ec:8d:55:
                    92:c3:f8:82:f1:99:a6:7a:42:88:a7:5d:26:aa:b5:
                    2b:b9:c5:4c:b1:af:8e:6b:f9:75:c8:a3:d7:0f:47:
                    94:14:55:35:57:8c:9e:a8:a2:39:19:f5:82:3c:42:
                    a9:4e:6e:f5:3b:c3:2e:db:8d:c0:b0:5c:f3:59:38:
                    e7:ed:cf:69:f0:5a:0b:1b:be:c0:94:24:25:87:fa:
                    37:71:b3:13:e7:1c:ac:e1:9b:ef:db:e4:3b:45:52:
                    45:96:a9:c1:53:ce:34:c8:52:ee:b5:ae:ed:8f:de:
                    60:70:e2:a5:54:ab:b6:6d:0e:97:a5:40:34:6b:2b:
                    d3:bc:66:eb:66:34:7c:fa:6b:8b:8f:57:29:99:f8:
                    30:17:5d:ba:72:6f:fb:81:c5:ad:d2:86:58:3d:17:
                    c7:e7:09:bb:f1:2b:f7:86:dc:c1:da:71:5d:d4:46:
                    e3:cc:ad:25:c1:88:bc:60:67:75:66:b3:f1:18:f7:
                    a2:5c:e6:53:ff:3a:88:b6:47:a5:ff:13:18:ea:98:
                    09:77:3f:9d:53:f9:cf:01:e5:f5:a6:70:17:14:af:
                    63:a4:ff:99:b3:93:9d:dc:53:a7:06:fe:48:85:1d:
                    a1:69:ae:25:75:bb:13:cc:52:03:f5:ed:51:a1:8b:
                    db:15
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Key Usage: critical
                Digital Signature, Certificate Sign, CRL Sign
            X509v3 Extended Key Usage: 
                TLS Web Client Authentication, TLS Web Server Authentication
            X509v3 Basic Constraints: critical
                CA:TRUE, pathlen:0
            X509v3 Subject Key Identifier: 
                14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
            X509v3 Authority Key Identifier: 
                keyid:79:B4:59:E6:7B:B6:E5:E4:01:73:80:08:88:C8:1A:58:F6:E9:9B:6E

            Authority Information Access: 
                CA Issuers - URI:http://x1.i.lencr.org/

            X509v3 CRL Distribution Points: 

                Full Name:
                  URI:http://x1.c.lencr.org/

            X509v3 Certificate Policies: 
                Policy: 2.23.140.1.2.1
                Policy: 1.3.6.1.4.1.44947.1.1.1

    Signature Algorithm: sha256WithRSAEncryption
         85:ca:4e:47:3e:a3:f7:85:44:85:bc:d5:67:78:b2:98:63:ad:
         75:4d:1e:96:3d:33:65:72:54:2d:81:a0:ea:c3:ed:f8:20:bf:
         5f:cc:b7:70:00:b7:6e:3b:f6:5e:94:de:e4:20:9f:a6:ef:8b:
         b2:03:e7:a2:b5:16:3c:91:ce:b4:ed:39:02:e7:7c:25:8a:47:
         e6:65:6e:3f:46:f4:d9:f0:ce:94:2b:ee:54:ce:12:bc:8c:27:
         4b:b8:c1:98:2f:a2:af:cd:71:91:4a:08:b7:c8:b8:23:7b:04:
         2d:08:f9:08:57:3e:83:d9:04:33:0a:47:21:78:09:82:27:c3:
         2a:c8:9b:b9:ce:5c:f2:64:c8:c0:be:79:c0:4f:8e:6d:44:0c:
         5e:92:bb:2e:f7:8b:10:e1:e8:1d:44:29:db:59:20:ed:63:b9:
         21:f8:12:26:94:93:57:a0:1d:65:04:c1:0a:22:ae:10:0d:43:
         97:a1:18:1f:7e:e0:e0:86:37:b5:5a:b1:bd:30:bf:87:6e:2b:
         2a:ff:21:4e:1b:05:c3:f5:18:97:f0:5e:ac:c3:a5:b8:6a:f0:
         2e:bc:3b:33:b9:ee:4b:de:cc:fc:e4:af:84:0b:86:3f:c0:55:
         43:36:f6:68:e1:36:17:6a:8e:99:d1:ff:a5:40:a7:34:b7:c0:
         d0:63:39:35:39:75:6e:f2:ba:76:c8:93:02:e9:a9:4b:6c:17:
         ce:0c:02:d9:bd:81:fb:9f:b7:68:d4:06:65:b3:82:3d:77:53:
         f8:8e:79:03:ad:0a:31:07:75:2a:43:d8:55:97:72:c4:29:0e:
         f7:c4:5d:4e:c8:ae:46:84:30:d7:f2:85:5f:18:a1:79:bb:e7:
         5e:70:8b:07:e1:86:93:c3:b9:8f:dc:61:71:25:2a:af:df:ed:
         25:50:52:68:8b:92:dc:e5:d6:b5:e3:da:7d:d0:87:6c:84:21:
         31:ae:82:f5:fb:b9:ab:c8:89:17:3d:e1:4c:e5:38:0e:f6:bd:
         2b:bd:96:81:14:eb:d5:db:3d:20:a7:7e:59:d3:e2:f8:58:f9:
         5b:b8:48:cd:fe:5c:4f:16:29:fe:1e:55:23:af:c8:11:b0:8d:
         ea:7c:93:90:17:2f:fd:ac:a2:09:47:46:3f:f0:e9:b0:b7:ff:
         28:4d:68:32:d6:67:5e:1e:69:a3:93:b8:f5:9d:8b:2f:0b:d2:
         52:43:a6:6f:32:57:65:4d:32:81:df:38:53:85:5d:7e:5d:66:
         29:ea:b8:dd:e4:95:b5:cd:b5:56:12:42:cd:c4:4e:c6:25:38:
         44:50:6d:ec:ce:00:55:18:fe:e9:49:64:d4:4e:ca:97:9c:b4:
         5b:c0:73:a8:ab:b8:47:c2
-----BEGIN CERTIFICATE-----
MIIFFjCCAv6gAwIBAgIRAJErCErPDBinU/bWLiWnX1owDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMjAwOTA0MDAwMDAw
WhcNMjUwOTE1MTYwMDAwWjAyMQswCQYDVQQGEwJVUzEWMBQGA1UEChMNTGV0J3Mg
RW5jcnlwdDELMAkGA1UEAxMCUjMwggEiMA0GCSqGSIb3DQEBAQUAA4IBDwAwggEK
AoIBAQC7AhUozPaglNMPEuyNVZLD+ILxmaZ6QoinXSaqtSu5xUyxr45r+XXIo9cP
R5QUVTVXjJ6oojkZ9YI8QqlObvU7wy7bjcCwXPNZOOftz2nwWgsbvsCUJCWH+jdx
sxPnHKzhm+/b5DtFUkWWqcFTzjTIUu61ru2P3mBw4qVUq7ZtDpelQDRrK9O8Zutm
NHz6a4uPVymZ+DAXXbpyb/uBxa3Shlg9F8fnCbvxK/eG3MHacV3URuPMrSXBiLxg
Z3Vms/EY96Jc5lP/Ooi2R6X/ExjqmAl3P51T+c8B5fWmcBcUr2Ok/5mzk53cU6cG
/kiFHaFpriV1uxPMUgP17VGhi9sVAgMBAAGjggEIMIIBBDAOBgNVHQ8BAf8EBAMC
AYYwHQYDVR0lBBYwFAYIKwYBBQUHAwIGCCsGAQUFBwMBMBIGA1UdEwEB/wQIMAYB
Af8CAQAwHQYDVR0OBBYEFBQusxe3WFbLrlAJQOYfr52LFMLGMB8GA1UdIwQYMBaA
FHm0WeZ7tuXkAXOACIjIGlj26ZtuMDIGCCsGAQUFBwEBBCYwJDAiBggrBgEFBQcw
AoYWaHR0cDovL3gxLmkubGVuY3Iub3JnLzAnBgNVHR8EIDAeMBygGqAYhhZodHRw
Oi8veDEuYy5sZW5jci5vcmcvMCIGA1UdIAQbMBkwCAYGZ4EMAQIBMA0GCysGAQQB
gt8TAQEBMA0GCSqGSIb3DQEBCwUAA4ICAQCFyk5HPqP3hUSFvNVneLKYY611TR6W
PTNlclQtgaDqw+34IL9fzLdwALduO/ZelN7kIJ+m74uyA+eitRY8kc607TkC53wl
ikfmZW4/RvTZ8M6UK+5UzhK8jCdLuMGYL6KvzXGRSgi3yLgjewQtCPkIVz6D2QQz
CkcheAmCJ8MqyJu5zlzyZMjAvnnAT45tRAxekrsu94sQ4egdRCnbWSDtY7kh+BIm
lJNXoB1lBMEKIq4QDUOXoRgffuDghje1WrG9ML+Hbisq/yFOGwXD9RiX8F6sw6W4
avAuvDszue5L3sz85K+EC4Y/wFVDNvZo4TYXao6Z0f+lQKc0t8DQYzk1OXVu8rp2
yJMC6alLbBfODALZvYH7n7do1AZls4I9d1P4jnkDrQoxB3UqQ9hVl3LEKQ73xF1O
yK5GhDDX8oVfGKF5u+decIsH4YaTw7mP3GFxJSqv3+0lUFJoi5Lc5da149p90Ids
hCExroL1+7mryIkXPeFM5TgO9r0rvZaBFOvV2z0gp35Z0+L4WPlbuEjN/lxPFin+
HlUjr8gRsI3qfJOQFy/9rKIJR0Y/8Omwt/8oTWgy1mdeHmmjk7j1nYsvC9JSQ6Zv
MldlTTKB3zhThV1+XWYp6rjd5JW1zbVWEkLNxE7GJThEUG3szgBVGP7pSWTUTsqX
nLRbwHOoq7hHwg==
-----END CERTIFICATE-----

The certs are deployed with a renewal post hook:

# cat /etc/letsencrypt/renewal-hooks/deploy/standard-services 
#!/bin/sh

set -e

if [ -f /etc/os-release ]; then
  . /etc/os-release
  case $ID in
    debian|ubuntu)
      ssl_path=/etc/ssl
      ;;
    rhel|centos|scientific)
      ssl_path=/etc/pki/tls
      ;;
  esac
fi

for domain in ${RENEWED_DOMAINS}; do
  # Make sure the certificate and private key files are
  # never world readable, even just for an instant while
  # we're copying them into daemon_cert_root.
  umask 077

  ssl_cert=${ssl_path}/certs/${domain}_cert.pem
  ssl_chain=${ssl_path}/certs/letsencrypt_chain.pem
  ssl_cert_fullchain=${ssl_path}/certs/${domain}_fullchain.pem
  ssl_key=${ssl_path}/private/${domain}_key.pem

  cp "${RENEWED_LINEAGE}/cert.pem" "${ssl_cert}"
  cp "${RENEWED_LINEAGE}/chain.pem" "${ssl_chain}"
  cp "${RENEWED_LINEAGE}/fullchain.pem" "${ssl_cert_fullchain}"
  cp "${RENEWED_LINEAGE}/privkey.pem" "${ssl_key}"

  if [ "$ID" = "debian" ]; then
    chown root:ssl-cert ${ssl_cert} ${ssl_key} ${ssl_chain} ${ssl_cert_fullchain}
    chmod 0640 ${ssl_key}
  fi

  chmod 0644 ${ssl_cert} ${ssl_chain} ${ssl_cert_fullchain}
done

Checking those, I see the certs and chains copied correctly there:

t# openssl x509 -in /etc/letsencrypt/live/ldap.ligo.uwm.edu/cert.pem > /tmp/live_cert.txt
# openssl x509 -in /etc/ssl/certs/ldap.ligo.uwm.edu_cert.pem > /tmp/ssl_cert.txt
# diff /tmp/{live,ssl}_cert.txt

and

# openssl x509 -in /etc/letsencrypt/live/ldap.ligo.uwm.edu/chain.pem > /tmp/live_chain.txt
# openssl x509 -in /etc/ssl/certs/letsencrypt_chain.pem > /tmp/ssl_chain.txt
# diff /tmp/{live,ssl}_chain.txt
#

I haven't figured out yet how or whence the files are copied into /etc/ldap/x509-certs/ directory, perhaps by puppet, I will make that my next mission. However, the reason you got partial paths with the base64 decode is that the lines were cut off by grep. The full lines are:

olcTLSCertificateFile:: L2V0Yy9sZGFwL3g1MDktY2VydHMvbGRhcC5saWdvLnV3bS5lZHVfY2VydC5wZW0g
olcTLSCertificateKeyFile:: L2V0Yy9sZGFwL3g1MDktY2VydHMvbGRhcC5saWdvLnV3bS5lZHVfa2V5LnBlbSA=

which give the paths:

/etc/ldap/x509-certs/ldap.ligo.uwm.edu_cert.pem
/etc/ldap/x509-certs/ldap.ligo.uwm.edu_key.pem

The cert file there matches the one in /etc/letsencrypt/live/ldap.ligo.uwm.edu/:

# openssl x509 -in /etc/ldap/x509-certs/ldap.ligo.uwm.edu_cert.pem > /tmp/ldap_cert.txt
# diff /tmp/{live,ldap}_cert.txt
#

There is no chain file in /etc/ldap/x509-certs. Just to be on the safe side, I checked the full chain files as well:

# openssl x509 -in /etc/letsencrypt/live/ldap.ligo.uwm.edu/fullchain.pem > /tmp/live_fullchain.txt
# openssl x509 -in /etc/ldap/x509-certs/ldap.ligo.uwm.edu_fullchain.pem > /tmp/ldap_fullchain.txt
# diff /tmp/{live,ldap}_fullchain.txt
#

It appears that my predecessor set the automated copying up correctly.

The change from fullchain to cert and the addition of olcTLSCACertificateFile: /etc/ssl/certs/ca-certificates.crt was my doing last spring when clients stopped being able to authenticate, at the suggestion of this forum if I recall correctly. But it may have been a band-aid on the fundamental pathology that is raising it's head again. I am more than happy to try anything that the forum can suggest as a next step (while I try to figure out how files get copied to /etc/ldap/x509-certs/). Thanks again!

24

Maybe I read too fast...
But that doesn't seem very dynamic.
It seems rigid - It should be able to change.

25

Not sure what you mean. As I read it, after every renewal, /etc/letsencrypt/live/ldap.ligo.uwm.edu/chain.pem is being copied into /etc/ssl/certs/letsencrypt_chain.pem. What more needs to change for the trust chain?

26

OK.
I did read too fast - LOL

hmm...
Using the best trust path?
Testing?
Finally getting some sleep?

27

In addition to cert.pem and chain.pem, certbot provides a file called "fullchain.pem" that contains both cert + chain. This is the file that should be served by LDAP.

So /etc/letsencrypt/live/ldap.ligo.uwm.edu/fullchain.pem should be equal to /etc/ldap/x509-certs/ldap.ligo.uwm.edu_cert.pem.

(Also note that chain.pem often contains more than one certificate. The openssl x509 command however only displays one certificate, even if the input file contains more than one. So when looking at chains, also consider the raw PEM format instead of only openssl parsing)

3 Likes
28

Done, and it did fix the problem. I am slightly (perhaps more than slightly) embarrassed that I wasted so much of everyone's time on such a simple oversight. I am greatly indebted to the help and knowledge that ALL of you gave me in this thread. As I said, I am not skilled at these things and I doubt I would have gotten to the bottom of the problem even as slowly as I did without your help!

2 Likes
29

I believe this will print all certs within the bundle for chain.pem or fullchain.pem too

openssl crl2pkcs7 -nocrl -certfile /path/to/chain.pem | openssl pkcs7 -print_certs
1 Like
30

Oh, nice, thank you!

2 Likes
31

openssl crl2pkcs7 -nocrl -certfile /path/to/chain.pem | openssl pkcs7 -print_certs
It takes a pem input and writes a pem looking output...
That is just slightly fancier than:
cat /path/to/chain.pem

1 Like
32

Yup at least it lists subject/issuer of the cert too so easier to identify :slight_smile:

1 Like
33

@eva2000
Thus the "slightly fancier".

The better "show" is the chain being served.
openssl s_client -connect EXAMPLE.COM:443 -servername EXAMPLE.COM | head
OR
[for more such added detail]
openssl s_client -connect EXAMPLE.COM:443 -servername EXAMPLE.COM -showcerts

[straight from the horse's mouth]

1 Like
34

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.