Unable to verify certificate

jeffd · September 27, 2021, 5:28pm

I am having the same issue on 20.04. After reading this thread, i'm going to put in my responses to your questions to andyrue. If I should open another issue, since its a different OS, that is fine. I found this when i was updating ocsp files, and ended up getting it down the first command below.

Before you get to the nitty gritty, thanks in advance!

$> openssl verify -CAfile /etc/letsencrypt/live/ukybonds.com/chain.pem /etc/letsencrypt/live/ukybonds.com/cert.pem Results in:

C = US, O = Internet Security Research Group, CN = ISRG Root X1
error 2 at 2 depth lookup: unable to get issuer certificate
error /etc/letsencrypt/live/ukybonds.com/cert.pem: verification failed

I am able to run wget https://letsencrypt.org/certs/isrgrootx1.pem fine.

$> sudo apt install libgnutls30
Reading package lists... Done
Building dependency tree
Reading state information... Done
libgnutls30 is already the newest version (3.6.13-2ubuntu1.6).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$> sudo apt install libgnutls-openssl27
Reading package lists... Done
Building dependency tree
Reading state information... Done
libgnutls-openssl27 is already the newest version (3.6.13-2ubuntu1.6).
0 upgraded, 0 newly installed, 0 to remove and 0 not upgraded.

$> ls -l /usr/local/share/ca-certificates/
-- Empty directory

$> ls -l /etc/ssl/certs/ca-certificates.crt
-rw-r--r-- 1 root root 199113 Sep 27 17:12 /etc/ssl/certs/ca-certificates.crt

If i use openssl s_client to read the live certs it works fine, and says that each level is valid
$> openssl s_client -connect www.ukybonds.com:443 -showcerts | openssl x509

verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = ukybonds.com
verify return:1
-- certificate omitted for space --

jeffd · September 27, 2021, 5:47pm

Thanks @rg305

rg305 · September 27, 2021, 5:47pm

I think there may be a problem with the way you tried to verify the cert.

I'm able to reach your site with curl and wget without issue.

jeffd · September 27, 2021, 5:53pm

The site works perfectly fine, but i can not generate ocsp files, for HAProxy b/c the local cert itself won't validate with openssl verify using the two files you recommended in the split question. Is there something wrong with my first command?
$> sudo openssl verify -CAfile /etc/letsencrypt/live/ukybonds.com/chain.pem /etc/letsencrypt/live/ukybonds.com/cert.pem?

Oh and this happens with all my LE certs, about 140
Certbot is 1.19.0

rg305 · September 27, 2021, 5:56pm

Please show:

[so others too can troubleshoot this issue]

jeffd · September 27, 2021, 5:57pm

Contents of
/etc/letsencrypt/live/ukybonds.com/chain.pem

rg305 · September 27, 2021, 7:41pm

Not sure what to make of this (from an Ubuntu 14 with OpenSSL 1.0.1f):
[it should have failed?]

curl -I https://www.ukybonds.com/
HTTP/1.1 301 Moved Permanently
server: nginx
date: Mon, 27 Sep 2021 19:36:44 GMT
content-length: 0
keep-alive: timeout=120
location: https://www.ukybonds.com/university-of-kentucky-bonds-ky/i2095
set-cookie: BONDLINK=eyJhbGciOiJIUzI1NiJ9.eyJkYXRhIjp7Im1vZCI6IjE2MzI3NzE0MDQiLCJjc3JmVG9rZW4iOiJkOTlmYzA2NTNlZjUwYzFkNWQwMGU4YmE2YmFiZjQyYmEzNDBkODY3NTA3MWE0NmU3YzE0N2U3YmQwZWE4OTA3LTE2MzI3NzE0MDQ1OTUtZjk5OGQxOGRkNzkyZGI3NTdhYjVmMDg0NDYzNmZkMGQ5NGQ5ZWRhMiJ9LCJleHAiOjE2NjQzMDc0MDQsIm5iZiI6MTYzMjc3MTQwNCwiaWF0IjoxNjMyNzcxNDA0fQ.67vr6FBRfJY5lcafKw-9-NIPnWqjwKwc7BfwY1drsGY; Max-Age=31536000; Expires=Tue, 27 Sep 2022 19:36:44 GMT; SameSite=None; Path=/; Secure; HTTPOnly
server-timing: tst;dur=12;desc="Total Server Time"
p3p: CP="BondLink does not have a P3P Policy"
strict-transport-security: max-age=31536000; includeSubDomains; preload
feature-policy: accelerometer 'none';camera 'none';encrypted-media 'none';geolocation 'none';gyroscope 'none';magnetometer 'none';microphone 'none';midi 'none';usb 'none'

wget https://www.ukybonds.com/
--2021-09-27 15:36:55--  https://www.ukybonds.com/
Resolving www.ukybonds.com (www.ukybonds.com)... 52.55.173.69
Connecting to www.ukybonds.com (www.ukybonds.com)|52.55.173.69|:443... connected.
HTTP request sent, awaiting response... 301 Moved Permanently
Location: https://www.ukybonds.com/university-of-kentucky-bonds-ky/i2095 [following]
--2021-09-27 15:37:00--  https://www.ukybonds.com/university-of-kentucky-bonds-ky/i2095
Reusing existing connection to www.ukybonds.com:443.
HTTP request sent, awaiting response... 200 OK
Length: 94489 (92K) [text/html]
Saving to: ‘index.html’

100%[==============================================================================>] 94,489       291KB/s   in 0.3s

2021-09-27 15:37:01 (291 KB/s) - ‘index.html’ saved [94489/94489]

openssl version
OpenSSL 1.0.1f 6 Jan 2014

openssl s_client -connect ukybonds.com:443 -servername ukybonds.com
CONNECTED(00000003)
depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify error:num=20:unable to get local issuer certificate
verify return:0
---
Certificate chain
 0 s:/CN=ukybonds.com
   i:/C=US/O=Let's Encrypt/CN=R3
 1 s:/C=US/O=Let's Encrypt/CN=R3
   i:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
 2 s:/C=US/O=Internet Security Research Group/CN=ISRG Root X1
   i:/O=Digital Signature Trust Co./CN=DST Root CA X3
---

Osiris · September 27, 2021, 8:06pm

Maybe curl and wget linked against GnuTLS?

rg305 · September 27, 2021, 8:42pm

Ubuntu 14 should have an outdated GnuTLS as well.

jeffd · September 27, 2021, 9:04pm

Just a SO question, but seems to align with exactly what you are seeing, only tls >= 1.1 is allowed to connect to this server. linux - Unable to establish SSL connection upon wget on Ubuntu 14.04 LTS - Stack Overflow

Another variable is that the www.website.com uses TLS 1.0. I don't have an idea how this affects wget. 
But if I wget an image from TLS 1.2 websites I don't get any ssl connection errors from both test cases.

Still fairly confused, but just looking for common issues.

jeffd · September 28, 2021, 9:35pm

A big thing that has been bothering me is this just started, it was noticed during OCSP updates and never impact the running server, in this case haproxy. This only started about two weeks ago (i was on vacation). I ran an openssl verify against certificates created in February, months before the problem, and those certs fail as well. Which makes little sense, as my openssl version OpenSSL 1.1.1f 31 Mar 2020. Yet has not failed until recently.

I would love to hear from anyone that has a recent cert update/creation, on 20.04 and the same openssl version, does it succeed or fail. There are more tests that i want to do, but i'm having a hard time accepting that this is an LE issue.

jeffd · September 28, 2021, 9:47pm

To add a bit more, my haproxy configuration shows the following for openssl

Built with OpenSSL version : OpenSSL 1.1.1  11 Sep 2018
Running on OpenSSL version : OpenSSL 1.1.1f  31 Mar 2020

which should mean the online verify should be using the same as openssl verify. This is production, so i'm not changing anything, but may spin up a new one.

jeffd · October 5, 2021, 12:32am

So i have/had doubts, but since this failed at the same time that the LE deprecated certs hit the market. All of my certs have the correct root (as you can see from both our outputs), but clearly something else is amiss. I'd like to see this kept open. Also spun up a new instance of a webserver and used fresh install of certbot two days ago, and they will not validate for the same reason.
@Osiris @rg305

rg305 · October 5, 2021, 2:57am

@jeffd
Have you tried reissuing a cert with the shorter preferred trust path chain?

MikeMcQ · October 5, 2021, 3:50am

I think that proves that curl does not use the openssl verify function and must do its own validation. The problem with the older openssl verify was just the way openssl walked the chain during verify. Nothing to do with the trust store itself.

I did the same test as you just now and my
curl -I https://community.letsencrypt.org (long chain)
worked without error even though this:

curl -V
**curl 7.76.1** (x86_64-koji-linux-gnu) libcurl/7.76.1 **OpenSSL/1.0.2k-fips** zlib/1.2.7 libidn2/2.3.0 libssh2/1.4.3 nghttp2/1.41.0
Release-Date: 2021-04-14
Protocols: dict file ftp ftps gopher gophers http https imap imaps ldap ldaps mqtt pop3 pop3s rtsp scp sftp smb smbs smtp smtps telnet tftp
Features: alt-svc AsynchDNS GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB SPNEGO SSL UnixSockets

and

openssl version
OpenSSL **1.0.2k-fips**  26 Jan 2017

The openssl verify shows cert expired unless I use the -trusted_first option.

What do you think?

rg305 · October 5, 2021, 3:56am

I think openssl verify does things in a very "all things must be perfect" approach; And we are intentionally inserting an expired root (to serve old Androids), so things are NOT perfect and it fails.
Anything that relies on similar "logic" will also fail.

MikeMcQ · October 5, 2021, 3:57am

Agreed. But your post made it seem like you thought the curl's would fail. But they did not. Perhaps I misunderstood.

rg305 · October 5, 2021, 3:59am

I had expected them to fail - yes - It's Ubuntu 14 ! ! !

jeffd · October 5, 2021, 3:23pm

Thanks for the reply but, I'm not sure exactly what you mean. I've been obtaining certs like this for over 5 years, let me know if i should change something.

sudo certbot certonly --webroot -w /var/www/html -d ${DOMAIN} -d www.${DOMAIN}

There is a wrapper script around this b/c it has to do other things, pushing to haproxy, rsync to the second load balancer (which is why i use certonly). I have 196 live certs from LE in production, and this is how they are obtained. Happy to try something different, but confused, why now?

rg305 · October 5, 2021, 3:31pm

add
--preferred-chain "ISRG Root X1"
OR
edit the fullchain.pem (or chain.pem) file used and remove the last cert.
[note this manual edit method won't survive a cert renewal - but may be faster/simpler to test with]