macOS Catalina 10.15 CN=DST Root CA X3

mshahat_eg · February 16, 2020, 12:22pm

Hello,
thanks for the great work this community is doing.

I’ve successfully generated my first let’s encrypt, configured https on haproxy and all good.
i’m using “kubectl” on macOS 10.15 communicating with the haproxy https interface, this gives an error of
Unable to connect to the server: x509: certificate signed by unknown authority

curl and chrome seem to work ok.

i am doubting this has something to do with Catalina and golang

as i run this test , i can observe the following

crypto/x509: verify-cert rejected CN=Let's Encrypt Authority X3,O=Let's Encrypt,C=US: "Cert Verify Result: Invalid Extended Key Usage for policy"

crypto/x509: verify-cert rejected CN=DST Root CA X3,O=Digital Signature Trust Co.: "Cert Verify Result: CSSMERR_TP_CERT_SUSPENDED"

Any idea what and why is this?

Thanks

My domain is:
hclcnlabs.com (privately hosted so not reachable over the internet)

I ran this command:

It produced this output:

My web server is (include version):
haproxy 2.0 on ubuntu 18.04

I can login to a root shell on my machine (yes or no, or I don’t know):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):
1.1.0 on macOS (generated the certificate manul using dns)

JuergenAuer · February 16, 2020, 2:02pm

Hi @mshahat_eg

it's difficult if it isn't possible to check your domain / certificate.

Only idea (shouldn't happen): You didn't install the pre-certificate?

Letsencrypt uses OCSP to check if a certificate is revoked. Searching that there is a chrome code:

CertStatus CertStatusFromOSStatus(OSStatus status) {
switch (status) {
case noErr:
return 0;

case CSSMERR_TP_INVALID_ANCHOR_CERT:
case CSSMERR_TP_NOT_TRUSTED:
case CSSMERR_TP_INVALID_CERT_AUTHORITY:
  return CERT_STATUS_AUTHORITY_INVALID;

case CSSMERR_TP_CERT_EXPIRED:
case CSSMERR_TP_CERT_NOT_VALID_YET:
  // "Expired" and "not yet valid" collapse into a single status.
  return CERT_STATUS_DATE_INVALID;

case CSSMERR_TP_CERT_REVOKED:
case CSSMERR_TP_CERT_SUSPENDED:
  return CERT_STATUS_REVOKED;

So if your server isn't able to connect Letsencrypt, it's not possible to check the revocation status of the certificate.

Is it possible to use the same code to connect https://letsencrypt.org/ to see if that works?

Looks like there

https://github.com/symfony/cli/issues/146

is the same problem.

mshahat_eg · February 16, 2020, 2:17pm

Thanks, Juergen.

You didn’t install the pre-certificate?
What is the pre-certificate?

Many thanks,

stevenzhu · February 16, 2020, 7:49pm

If your MacOS have OpenSSL installed, run it inside your internal network.

openssl s_client -showcerts -connect hclcnlabs.com:443

Please show the full output. (It should also print out a bunch of certificates)

Thank you

mshahat_eg · February 16, 2020, 8:05pm

thanks, stevenzhu.

here you go the output.

~ ❯❯❯ openssl s_client -showcerts -connect cluster-1.pks.uat.lnd.hclcnlabs.com:8443
CONNECTED(00000005)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.pks.uat.lnd.hclcnlabs.com
verify return:1
---
Certificate chain
 0 s:/CN=*.pks.uat.lnd.hclcnlabs.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFbjCCBFagAwIBAgISA60R/fkm/ATtFOSY4qjQFffRMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDAyMTAwOTAxMDlaFw0y
MDA1MTAwOTAxMDlaMCYxJDAiBgNVBAMMGyoucGtzLnVhdC5sbmQuaGNsY25sYWJz
LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJb7sJvSKYrWht/I
8YaKUbqs82JrKHHJb4EhGc6V5g6Khj6cwgM2ug+PXJ9d9gUc165Dh8/72ayuINF+
e5sRe9DJb+rBFsePj6Q/CtcrGWais+y3JRHvpwaDO4a7ZK5Pllg4FsfZjkwt0nVp
fHAEOxemqe6zeELLJx5qnOQIRaRGk4rmZ6x8XQAahCfGiwN9ROdW65Grm/mUCPrp
qHBJwuIOUQLWEfGpH7vq2XK4RAE7D5v/emmds0RnnhlQpyf94gI0MGZUfuy40rCh
oCav2/VKUgx+BNPolkfDIGvef98hD2Rl9sW4l1xhtuouxFh6/9VdWXMHrD2sz5c+
USKqBEkCAwEAAaOCAnAwggJsMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr
BgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUSKKazE99
6h4jzuXMAghdesSPiYEwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEw
bwYIKwYBBQUHAQEEYzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMu
bGV0c2VuY3J5cHQub3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMu
bGV0c2VuY3J5cHQub3JnLzAmBgNVHREEHzAdghsqLnBrcy51YXQubG5kLmhjbGNu
bGFicy5jb20wTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAm
BggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEE
AdZ5AgQCBIH1BIHyAPAAdgBep3P531bA57U2SH3QSeAyepGaDIShEhKEGHWWgXFF
WAAAAXAui6ohAAAEAwBHMEUCIB/VQfneuhFPGLwLBGbMvInPtBNXL0WxQ5c4gy7A
DdC2AiEAuc3uLUvwJE1kWDYVkRQMN29ey3nU24Eky5o5VlRlHbgAdgCyHgXMi6LN
iiBOh2b5K7mKJSBna9r6cOeySVMt74uQXgAAAXAui6oZAAAEAwBHMEUCIBCJI7cd
op0k/7b+/8mw8w9L6RYUgY34GwRhpZYmBQFYAiEAxWh7BhcGt/k8HK8ZTTWAP/XK
ArxCwUKrVWk3CNcc5kcwDQYJKoZIhvcNAQELBQADggEBAHG9sIa3dLjlHEcacNcW
Qx5w8tkpPNVYB/lLmGYtH76bkKii2G96Lvza6L/Z9bKpHKNchHzXZtEvADKSiNzu
i0osMkA65XVfxS6pKnxmDearO/aJtf1fE75xzKbEiDQxf6/1DmQV4SE0CBjRFGE6
b4XJlzn9OTK4eoZu/k/mxe6BiwA+MCGSrg22mwki6oDCMgxdEdIDWwTYsINNKZue
A0o3ze0ggzvJi28AgP1VdIhHkCemn2lHG0wTwbuXDBo7ZqWQuPLGfA88sqrmBbVo
SUp0hGjS729SsRtkv21JEB5Ps6dfL0VbLypE2kJHdIh8ZQ5eviZW5HpXh34UBsdq
zrQ=
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=*.pks.uat.lnd.hclcnlabs.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 3045 bytes and written 289 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
Protocol  : TLSv1.2
Cipher    : ECDHE-RSA-AES128-GCM-SHA256
Session-ID: B7C86BCA291F2B0786805D5FEFB219D0FBB812C0E68831392BC4CF6E77466769
Session-ID-ctx: 
Master-Key: 4FE9ED0B324AA2DF33F59BEDB8C2C131AA3989602A4DC005DF2E69A3C05990D926251218518A687CE05F47DAEC05B90F
Start Time: 1581883354
Timeout   : 7200 (sec)
Verify return code: 0 (ok)
---

stevenzhu · February 16, 2020, 8:26pm

Hi,

It seems like your Kubernate attempts to use a self-signed certificate (with their own CA).
Your Let’s Encrypt certificate is chained up to valid CA root.

I found the below article that might help you:

Thank you

mshahat_eg · February 16, 2020, 8:42pm

Thanks, Steven.

Could you explain your conclusion a bit more please? and
how did you conclude this from the output i’ve shared in my earlier message?

by the way, the setup is as following

kubectl (cli client on macos) -> https -> :8443 haproxy (https using let's encrypt cert - mode http) -> kubernetes master api-server :8443

Thanks,

stevenzhu · February 16, 2020, 9:51pm

The first error message stated that the certificate is signed by unknown authority, which means the CA (Let's Encrypt) is unknown to the trust store this software used.
When I ask you to run the above command, I can see that the certificate is a leaf certificate, not the pre-certificate (@JuergenAuer's concern). (From the output of certificate code)
While your OpenSSL successfully connects to your website (with s_client), it means your system's CA store contains the root CA.
Then I looked up the certificate issue with Kubernetes, and it seems like if you use a CA other than the CA your software-generated (self-sign), there's this issue. (It's in the other GitHub thread)

Hope this explains.

Thank you
Steven Zhu

mshahat_eg · February 16, 2020, 10:04pm

Thanks, Steven. This helps much.

I’d presume that the kubernetes issue is not relevant, as i’ve setup the loadbalancer to offload TLS. I’m not expecting the TLS handshake to take place with the backend of the loadbalancer.

Does this make sense? i guess the only point that i’ve not acted at all now is the pre-certificate. any guidance on what i need to achieve there?

I can see two entries
https://crt.sh/?id=2436120029
https://crt.sh/?id=2447281798

is that what @JuergenAuer is referring to?

Thanks,

stevenzhu · February 16, 2020, 11:02pm

When i asked for the OpenSSL output, it showed that the certificate you used is not a pre-certificate. (As you can see after decode, their CT field is different)

However, my knowledge stopped at this point... so i'm not sure what would be the issues here.

Based on the other information (error):

I'm getting confused by this since Let's Encrypt's certificate showed a correct EKU, which shouldn't have this error.

(I'll update more when i got home)

mshahat_eg · February 16, 2020, 11:51pm

Thanks @stevenzhu. i will be looking forward to hearing from you and @JuergenAuer.

some finding as well in the meanwhile.
I’m on macOS Catalina 10.15.3

the previous command i run was using openssl that comes with macOS
LibreSSL 2.8.3

i re-ran the command again using openssl i installed using homebrew.
OpenSSL 1.1.1d 10 Sep 2019

~ ❯❯❯ openssl s_client -showcerts -connect cluster-1.pks.uat.lnd.hclcnlabs.com:8443                                                                                                                                                                                        
CONNECTED(00000005)
depth=2 O = Digital Signature Trust Co., CN = DST Root CA X3
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = Let's Encrypt Authority X3
verify return:1
depth=0 CN = *.pks.uat.lnd.hclcnlabs.com
verify return:1
---
Certificate chain
 0 s:/CN=*.pks.uat.lnd.hclcnlabs.com
   i:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
-----BEGIN CERTIFICATE-----
MIIFbjCCBFagAwIBAgISA60R/fkm/ATtFOSY4qjQFffRMA0GCSqGSIb3DQEBCwUA
MEoxCzAJBgNVBAYTAlVTMRYwFAYDVQQKEw1MZXQncyBFbmNyeXB0MSMwIQYDVQQD
ExpMZXQncyBFbmNyeXB0IEF1dGhvcml0eSBYMzAeFw0yMDAyMTAwOTAxMDlaFw0y
MDA1MTAwOTAxMDlaMCYxJDAiBgNVBAMMGyoucGtzLnVhdC5sbmQuaGNsY25sYWJz
LmNvbTCCASIwDQYJKoZIhvcNAQEBBQADggEPADCCAQoCggEBAJb7sJvSKYrWht/I
8YaKUbqs82JrKHHJb4EhGc6V5g6Khj6cwgM2ug+PXJ9d9gUc165Dh8/72ayuINF+
e5sRe9DJb+rBFsePj6Q/CtcrGWais+y3JRHvpwaDO4a7ZK5Pllg4FsfZjkwt0nVp
fHAEOxemqe6zeELLJx5qnOQIRaRGk4rmZ6x8XQAahCfGiwN9ROdW65Grm/mUCPrp
qHBJwuIOUQLWEfGpH7vq2XK4RAE7D5v/emmds0RnnhlQpyf94gI0MGZUfuy40rCh
oCav2/VKUgx+BNPolkfDIGvef98hD2Rl9sW4l1xhtuouxFh6/9VdWXMHrD2sz5c+
USKqBEkCAwEAAaOCAnAwggJsMA4GA1UdDwEB/wQEAwIFoDAdBgNVHSUEFjAUBggr
BgEFBQcDAQYIKwYBBQUHAwIwDAYDVR0TAQH/BAIwADAdBgNVHQ4EFgQUSKKazE99
6h4jzuXMAghdesSPiYEwHwYDVR0jBBgwFoAUqEpqYwR93brm0Tm3pkVl7/Oo7KEw
bwYIKwYBBQUHAQEEYzBhMC4GCCsGAQUFBzABhiJodHRwOi8vb2NzcC5pbnQteDMu
bGV0c2VuY3J5cHQub3JnMC8GCCsGAQUFBzAChiNodHRwOi8vY2VydC5pbnQteDMu
bGV0c2VuY3J5cHQub3JnLzAmBgNVHREEHzAdghsqLnBrcy51YXQubG5kLmhjbGNu
bGFicy5jb20wTAYDVR0gBEUwQzAIBgZngQwBAgEwNwYLKwYBBAGC3xMBAQEwKDAm
BggrBgEFBQcCARYaaHR0cDovL2Nwcy5sZXRzZW5jcnlwdC5vcmcwggEEBgorBgEE
AdZ5AgQCBIH1BIHyAPAAdgBep3P531bA57U2SH3QSeAyepGaDIShEhKEGHWWgXFF
WAAAAXAui6ohAAAEAwBHMEUCIB/VQfneuhFPGLwLBGbMvInPtBNXL0WxQ5c4gy7A
DdC2AiEAuc3uLUvwJE1kWDYVkRQMN29ey3nU24Eky5o5VlRlHbgAdgCyHgXMi6LN
iiBOh2b5K7mKJSBna9r6cOeySVMt74uQXgAAAXAui6oZAAAEAwBHMEUCIBCJI7cd
op0k/7b+/8mw8w9L6RYUgY34GwRhpZYmBQFYAiEAxWh7BhcGt/k8HK8ZTTWAP/XK
ArxCwUKrVWk3CNcc5kcwDQYJKoZIhvcNAQELBQADggEBAHG9sIa3dLjlHEcacNcW
Qx5w8tkpPNVYB/lLmGYtH76bkKii2G96Lvza6L/Z9bKpHKNchHzXZtEvADKSiNzu
i0osMkA65XVfxS6pKnxmDearO/aJtf1fE75xzKbEiDQxf6/1DmQV4SE0CBjRFGE6
b4XJlzn9OTK4eoZu/k/mxe6BiwA+MCGSrg22mwki6oDCMgxdEdIDWwTYsINNKZue
A0o3ze0ggzvJi28AgP1VdIhHkCemn2lHG0wTwbuXDBo7ZqWQuPLGfA88sqrmBbVo
SUp0hGjS729SsRtkv21JEB5Ps6dfL0VbLypE2kJHdIh8ZQ5eviZW5HpXh34UBsdq
zrQ=
-----END CERTIFICATE-----
 1 s:/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
-----BEGIN CERTIFICATE-----
MIIEkjCCA3qgAwIBAgIQCgFBQgAAAVOFc2oLheynCDANBgkqhkiG9w0BAQsFADA/
MSQwIgYDVQQKExtEaWdpdGFsIFNpZ25hdHVyZSBUcnVzdCBDby4xFzAVBgNVBAMT
DkRTVCBSb290IENBIFgzMB4XDTE2MDMxNzE2NDA0NloXDTIxMDMxNzE2NDA0Nlow
SjELMAkGA1UEBhMCVVMxFjAUBgNVBAoTDUxldCdzIEVuY3J5cHQxIzAhBgNVBAMT
GkxldCdzIEVuY3J5cHQgQXV0aG9yaXR5IFgzMIIBIjANBgkqhkiG9w0BAQEFAAOC
AQ8AMIIBCgKCAQEAnNMM8FrlLke3cl03g7NoYzDq1zUmGSXhvb418XCSL7e4S0EF
q6meNQhY7LEqxGiHC6PjdeTm86dicbp5gWAf15Gan/PQeGdxyGkOlZHP/uaZ6WA8
SMx+yk13EiSdRxta67nsHjcAHJyse6cF6s5K671B5TaYucv9bTyWaN8jKkKQDIZ0
Z8h/pZq4UmEUEz9l6YKHy9v6Dlb2honzhT+Xhq+w3Brvaw2VFn3EK6BlspkENnWA
a6xK8xuQSXgvopZPKiAlKQTGdMDQMc2PMTiVFrqoM7hD8bEfwzB/onkxEz0tNvjj
/PIzark5McWvxI0NHWQWM6r6hCm21AvA2H3DkwIDAQABo4IBfTCCAXkwEgYDVR0T
AQH/BAgwBgEB/wIBADAOBgNVHQ8BAf8EBAMCAYYwfwYIKwYBBQUHAQEEczBxMDIG
CCsGAQUFBzABhiZodHRwOi8vaXNyZy50cnVzdGlkLm9jc3AuaWRlbnRydXN0LmNv
bTA7BggrBgEFBQcwAoYvaHR0cDovL2FwcHMuaWRlbnRydXN0LmNvbS9yb290cy9k
c3Ryb290Y2F4My5wN2MwHwYDVR0jBBgwFoAUxKexpHsscfrb4UuQdf/EFWCFiRAw
VAYDVR0gBE0wSzAIBgZngQwBAgEwPwYLKwYBBAGC3xMBAQEwMDAuBggrBgEFBQcC
ARYiaHR0cDovL2Nwcy5yb290LXgxLmxldHNlbmNyeXB0Lm9yZzA8BgNVHR8ENTAz
MDGgL6AthitodHRwOi8vY3JsLmlkZW50cnVzdC5jb20vRFNUUk9PVENBWDNDUkwu
Y3JsMB0GA1UdDgQWBBSoSmpjBH3duubRObemRWXv86jsoTANBgkqhkiG9w0BAQsF
AAOCAQEA3TPXEfNjWDjdGBX7CVW+dla5cEilaUcne8IkCJLxWh9KEik3JHRRHGJo
uM2VcGfl96S8TihRzZvoroed6ti6WqEBmtzw3Wodatg+VyOeph4EYpr/1wXKtx8/
wApIvJSwtmVi4MFU5aMqrSDE6ea73Mj2tcMyo5jMd6jmeWUHK8so/joWUoHOUgwu
X4Po1QYz+3dszkDqMp4fklxBwXRsW10KXzPMTZ+sOPAveyxindmjkW8lGy+QsRlG
PfZ+G6Z6h7mjem0Y+iWlkYcV4PIWL1iwBi8saCbGS5jN2p8M+X+Q7UNKEkROb3N6
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
---
Server certificate
subject=/CN=*.pks.uat.lnd.hclcnlabs.com
issuer=/C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3
---
No client certificate CA names sent
Server Temp Key: ECDH, X25519, 253 bits
---
SSL handshake has read 3045 bytes and written 289 bytes
---
New, TLSv1/SSLv3, Cipher is ECDHE-RSA-AES128-GCM-SHA256
Server public key is 2048 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
No ALPN negotiated
SSL-Session:
    Protocol  : TLSv1.2
    Cipher    : ECDHE-RSA-AES128-GCM-SHA256
    Session-ID: 10DECF3A0ABDDE2130EAA333D6D59E272F1FFF4F4082C70259829DB8F9E4195D
    Session-ID-ctx: 
    Master-Key: 969F52A66C311FC73D17ED17316829F5E689A60C4F8416E3AF7BED99A50CF2F05F4BDA76BFFF446EC758CA5282907EBC
    Start Time: 1581896790
    Timeout   : 7200 (sec)
    Verify return code: 0 (ok)
---

HTTP/1.0 408 Request Time-out
cache-control: no-cache
content-type: text/html

<html><body><h1>408 Request Time-out</h1>
Your browser didn't send a complete request in time.
</body></html>

closed

this is the outoput when i run using the other distro.

Thanks,

stevenzhu · February 17, 2020, 4:35pm

Hi,

When you are running kubectl, what would happen if you append command below:
--kubelet-certificate-authority=path-to-a-copy-of-root-ca (Let's Encrypt CA Chain) --kubelet-client-certificate=path-to-your-let's-encrypt-certificate --kubelet-client-key=path-to-your-let's-encrypt-certificate-key

Thank you

mshahat_eg · February 17, 2020, 8:16pm

hello @stevenzhu,

those flags are not for kubectl, they are flags for the kubernetes api-server that would run on a kubernetes server.

in this case kubectl is just a client on a workstation (the macos one) sending requests to haproxy which is expected to offload TLS traffic then establish another one.

this is the haproxy config for reference

frontend fe-https-all-pks-uat-lnd-hclcnlabs-com
	bind 172.16.5.100:8443 ssl crt /etc/ssl/certs/pks.uat.lnd.hclcnlabs.com.pem
	bind 172.16.5.100:9021 ssl crt /etc/ssl/certs/pks.uat.lnd.hclcnlabs.com.pem
	
	# PKS API Control Plane 
	acl is-api-pks hdr_dom(host) -i api.pks.uat.lnd.hclcnlabs.com
	use_backend be-api-pks-uat-lnd-hclcnlabs.com if is-api-pks

	# Cluster 1 Kubernetes Master node
	acl is-cluster-1-master hdr_dom(host) -i cluster-1.pks.uat.lnd.hclcnlabs.com
	use_backend be-cluster-1-pks-uat-lnd-hclcnlabs-com if is-cluster-1-master


backend be-api-pks-uat-lnd-hclcnlabs.com
	mode http
	server server1 172.16.5.121 ssl verify none

backend be-cluster-1-pks-uat-lnd-hclcnlabs-com
	mode http
	http-request set-header X-Forwarded-Port %[dst_port]
	http-request add-header X-Forwarded-Proto https if { ssl_fc }
	server server2 172.16.5.21:8443 ssl verify none

Thanks,

stevenzhu · February 18, 2020, 4:46am

Hi,

Apologize but that’s all I know about Kubernetes (or anything related to that). I still believe it’s a validation issue between your kubectl client and the server, as you said there’s no issue for curl or other programs to access the same port.
I would suggest you seek help from Kubernetes forums (as I don’t know who in this forum knows Kubernetes, maybe there are hidden gems ) or GitHub.

My last attempt (highly dangerous):
According to this answer: https://stackoverflow.com/a/57701551 and GitHub issue, you can try to connect to your cluster (or proxy) with --insecure-skip-tls-verify=true, which would at least allow you to bypass the CA certificate issue and investigate on what happened (or why it’s saying certificate signed by unknown authority).
(From my basic knowledge, I assume that if you can reach your intended target after bypass the certificate validation, then there’s some issue with the CA cert validation for your client machine.)

Apologize again, but that’s all I know (or can look up from Google).

Thank you

mshahat_eg · February 18, 2020, 5:02am

Many thanks, Steve.
i’m using --insecure-skip-verity for now and will keep the investigation going.

Thanks,

system · March 19, 2020, 5:02am

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.