Issue - Unable to verify certificate chain - CentOS7

MikeMcQ · July 6, 2022, 10:16pm

As an aside, I don't think -purpose has value when checking certs known to be from Let's Encrypt. The openssl docs say

The supplied or "leaf" certificate must have extensions compatible with the supplied purpose and all other certificates must also be valid CA certificates.

We already know LE certs are fine for ssl servers. I'd be more worried of that option showing an error that doesn't exist due purely to an aged or faulty openssl.

AnsRed · July 6, 2022, 11:07pm

Interesting update and good news!

I noticed something, the default fullchain.pem cert from LE doesn't have the actual root certificate that should be there.

Meaning, you can see the root cert is different in the bottom from the next one

and here is another bundle I have created manually on Windows machine, simply copy to file and save and so on, and then combined leaf >> inter >> root into a single file

The manual one worked fine with my web server!

       Valid: Certificate chain is verified.
Current time: 2022-07-06 23:00:18 UTC.
  Not before: 2022-07-06 00:25:27 UTC.
   Not after: 2022-10-04 00:25:26 UTC.
    Validity: 2 months, 29 days.
   Remaining: 2 months, 28 days, 1 hour, 25 minutes, 9 seconds.
Valid: web_service_certificate tests passed

Any idea what is that root they are using in the fullchain.pem?!

MikeMcQ · July 6, 2022, 11:51pm

First, an actual CA root should not be in a chain sent by a server. At best clients will ignore it and it just takes longer to send a longer chain. Maybe some clients will even reject it as invalid chain - I am not sure. That is not how this trust system works.

You could try removing the last cert from fullchain.pem and use that for your server.

It looks like your server is not tolerating the default "long chain" from Let's Encrypt. It contains your leaf and then two intermediates that can link to different roots. The second intermediate, which is the 3rd file in fullchain.pem, is a cross-signed cert issued by DST Root CA X3. It expired last Sept but is kept in place for compatibility with older Android devices. Numerous website servers send this long chain including this website.

It looks like your server is not allowing you to use that. It will limit what clients will trust your website. Whether that is a problem is for you to decide.

First though, make a new file with just the first 2 certs from fullchain.pem and ensure that works in your server. Then we can talk about the implications.

By the way, again, what is the server you are using?

AnsRed · July 7, 2022, 2:06am

Thanks for the explanation about that part with the two intermediates part, hence, it doesn't work until I make the manual tweak to get a complete fullchain with no expired certs.

This setup is for Illumio Microsegmentation PCE

MikeMcQ · July 7, 2022, 2:36am

Can you point me to the config docs for that server? I looked but cannot find them.

That set of certs you created includes the root trust anchor. If your clients actually trust it without confirming in their own CA trust store then you may have been able to use a self-signed cert. Sending the anchor is unnecessary and defeats the purpose of client verifications of trust.

You have a "short chain" and you can read about its compatibility here.

As an overview of the short and long chains there is this topic

With certbot 1.12 you should be able to use the --preferred-chain 'ISRG Root X1' option so you get the short chain returned in fullchain.pem. Then, apparently, you need to add the ISRG Root X1 root trust anchor to satisfy your server. That is probably easier to create than shortening the long chain and adding the root trust anchor.

rg305 · July 7, 2022, 2:40am

The default/long chain uses an expired root.
See:

Read more about that here:

You might want to use the short chain or switch to another free CA chain.

MikeMcQ · July 7, 2022, 2:44am

I found the below docs. Let's Encrypt is a public CA (not your private one). Based on these docs you should not be adding the ISRG root trust anchor. However, you may need to use the short chain.

You should probably ask illumio why the verification fails without the trust anchor. You can see their docs match what I have been describing about the client doing the final step and matching to their own CA store.

See the X509 section in this:

https://docs.illumio.com/core/21.5/Content/Guides/pce-install-upgrade/preparation/requirements-for-pce-installation.htm#tls-ssl-requirements

MikeMcQ · July 7, 2022, 2:45am

Yeah, probably do. But, they are saying they need to add the ISRG root trust anchor to that to satisfy the server's "verification". This is the last hangup and one I just posted about.

Edit: This is the chain they said they need to make this work. I think they should remove the last one but it apparently did not work.

0:
subject= /CN=illumio.consulting
issuer= /C=US/O=Let's Encrypt/CN=R3
notBefore=Jul  6 00:25:27 2022 GMT
notAfter=Oct  4 00:25:26 2022 GMT
1:
subject= /C=US/O=Let's Encrypt/CN=R3
issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
notBefore=Sep  4 00:00:00 2020 GMT
notAfter=Sep 15 16:00:00 2025 GMT
2:
subject= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
issuer= /C=US/O=Internet Security Research Group/CN=ISRG Root X1
notBefore=Jun  4 11:04:38 2015 GMT
notAfter=Jun  4 11:04:38 2035 GMT

AnsRed · July 7, 2022, 3:09am

Everything is working now as expected. The product is just designed to be used with purchased certificates and it has to have a valid chain trust in order to get the product working.

I think it would be better as the person complained on the main announcement post with Let's Encrypt to have both bundles full chain included or simply remove the expired one, there shouldn't be any expired cert in there, even if it's for older android devices. Just thinking.

Anyways, thanks LOADS everything @MikeMcQ @jvanasco and @rg305 for the help here. It was a good long test we did here today

MikeMcQ · July 7, 2022, 3:25am

They do. As I described, the default is the "long chain". You can use the --preferred-chain option of certbot to select the "short chain". It is not practical to return both for each request - you would still need to choose one or the other.

Besides, your biggest problem (now) is that addition of the root trust anchor. That is highly unusual. And, is contrary to the server's own docs so appears to be a bug in their verification routine.

AnsRed · July 7, 2022, 3:55am

Makes sense! Thanks @MikeMcQ !!!!

rg305 · July 7, 2022, 4:10am

The first part must be a misconception/misunderstanding - there is no difference between a paid cert and a free cert [when both come from a valid globally trusted CA].

The second part is a strange requirement when given that it is expected to be included in the bundle provided by the issuing CA [extremely BAD practice].

AnsRed · July 7, 2022, 5:48am

Thanks @rg305!!

jvanasco · July 7, 2022, 3:18pm

A few things here:

The expired certificate is the default option, but not required. The ACME API provides both certificates, it is a deficiency in the client or user if the alternate option is not utilized or offered.
The expired certificate technique is the only way for a large number of devices to utilize the internet.
The expired certificate technique works because:

The Android devices to not care about the expiration date
Nearly every modern client (browsers, openssl & variants, programming libraries, applications) ignores the presented chain and calculates their own path to trusted roots.

The issue that you experienced is due in part to using an old and unsupported version of OpenSSL. 1.02 was released in 2015 and stopped receiving updates in 2019 as part of their LTS (long term support) strategy. Aside from missing the updates and security fixes in the 1.1.0 and 1.1.1 releases, 1.02k is relatively old for that release cycle - which culminated in 1.02u.

The 1.02k release dates back to around Jan 26, 2017 - this is over 5 years ago (see Prepare for 1.0.2k release · openssl/openssl@081314d · GitHub)

CentOS 7 was released in 2014 and only received updates through 2020, when the project was terminated by RedHat. There are some maintenance updates promised through 2024 - but continuing to use the platform is simply not prudent.

This might not be a contradiction. You linked to their active v21.5 docs, but we have no idea what version is running. The 21.5 release looks to be relatively new, but the OS is 8 years old and 2 years out of it's full support window (AFAIK, it's only getting security updates now). I assume the machine is running a much earlier version.

system · August 6, 2022, 3:19pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Openssl verify: unable to get issuer certificate Help	4	7595	April 1, 2020
There are issues with the site's certificate chain (net::ERR_CERT_COMMON_NAME_INVALID Help	4	17526	April 14, 2017
Lost letsecrypt installation CENTOS 7 Help	7	421	January 19, 2023
LetsEncrypt root certifcate not valid? Help	4	1570	January 2, 2022
CERTIFICATE_VERIFY_FAILED self signed certificate in certificate chain Help	15	6560	September 19, 2021

Issue - Unable to verify certificate chain - CentOS7

Related topics