Ubuntu 18.04 and wget error cannot verify certificate

Help
1

Hi everyone, hoping someone can give me a hand with a Ubuntu 18.04 server getting to web resources using Let's encrypt certificates. This is the problem I am facing:

owcld18:/Installs$ sudo wget https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
--2021-11-14 10:05:50--  https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 167.233.14.167, 2a01:4f8:1c1d:3d1::1
Connecting to download.owncloud.org (download.owncloud.org)|167.233.14.167|:443... connected.
ERROR: cannot verify download.owncloud.org's certificate, issued by ‘CN=R3,O=Let's Encrypt,C=US’:
  unable to get issuer certificate
To connect to download.owncloud.org insecurely, use `--no-check-certificate'.

As you can see I am attempting a wget command to a tarball hosted by ownCloud that must be using a Let's Encrypt cert. My Windows PC appears to download this file just fine, but this Ubuntu server doesn't think the cert is valid. I suspect it has something to do with the Sept certificate chain adjustments made by Let's encrypt, but I can't figure out how to resolve.

I've been all over this thread troubleshooting, nothing suggested here helps.

I am not a wizard when it comes to Linux, but usually I can google enough to figure things out. This one has ben totally stumped, no conversations online so far have helped. I believe this Ubuntu server was upgraded at one time from 16.04 LTS if that helps at all. Thanks for any assistance you can provide!

Server version:

owcld18:/Installs$ sudo lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description:    Ubuntu 18.04.6 LTS
Release:        18.04
Codename:       bionic

ca-certificates info:

owcld18:/Installs$ sudo dpkg -l | grep ca-cert
ii  ca-certificates                        20210119~18.04.2                                                   all          Common CA certificates

openssl info

sudo openssl version
OpenSSL 1.1.1  11 Sep 2018

I've run this
sudo apt-get update && sudo apt-get upgrade

This is what an apt-update looks like now:

sudo apt update
Hit:1 http://mirrors.accretive-networks.net/mariadb/repo/10.5/ubuntu bionic InRelease
Hit:2 http://us.archive.ubuntu.com/ubuntu bionic InRelease
Hit:3 http://ppa.launchpad.net/ondrej/php/ubuntu bionic InRelease
Get:4 http://us.archive.ubuntu.com/ubuntu bionic-updates InRelease [88.7 kB]
Get:5 http://us.archive.ubuntu.com/ubuntu bionic-backports InRelease [74.6 kB]
Get:6 http://us.archive.ubuntu.com/ubuntu bionic-security InRelease [88.7 kB]
Fetched 252 kB in 1s (300 kB/s)
Reading package lists... Done
Building dependency tree
Reading state information... Done

Some lines from ca-certificates.conf file.

# This file lists certificates that you wish to use or to ignore to be
# installed in /etc/ssl/certs.
# update-ca-certificates(8) will update /etc/ssl/certs by reading this file.
#
# This is autogenerated by dpkg-reconfigure ca-certificates.
# Certificates should be installed under /usr/share/ca-certificates
# and files with extension '.crt' is recognized as available certs.
#
# line begins with # is comment.
# line begins with ! is certificate filename to be deselected.
#
mozilla/ACCVRAIZ1.crt
mozilla/AC_RAIZ_FNMT-RCM.crt
mozilla/Actalis_Authentication_Root_CA.crt
!mozilla/AddTrust_External_Root.crt
mozilla/AffirmTrust_Commercial.crt
mozilla/AffirmTrust_Networking.crt
mozilla/AffirmTrust_Premium.crt
mozilla/AffirmTrust_Premium_ECC.crt
mozilla/Amazon_Root_CA_1.crt
mozilla/Amazon_Root_CA_2.crt
mozilla/Amazon_Root_CA_3.crt
mozilla/Amazon_Root_CA_4.crt
mozilla/Atos_TrustedRoot_2011.crt
mozilla/Autoridad_de_Certificacion_Firmaprofesional_CIF_A62634068.crt
mozilla/Baltimore_CyberTrust_Root.crt
mozilla/Buypass_Class_2_Root_CA.crt
mozilla/Buypass_Class_3_Root_CA.crt
mozilla/CA_Disig_Root_R2.crt
mozilla/CFCA_EV_ROOT.crt
mozilla/COMODO_Certification_Authority.crt
mozilla/COMODO_ECC_Certification_Authority.crt
mozilla/COMODO_RSA_Certification_Authority.crt
mozilla/Certigna.crt
!mozilla/Certinomis_-_Root_CA.crt
!mozilla/Certplus_Class_2_Primary_CA.crt
!mozilla/Certplus_Root_CA_G1.crt
!mozilla/Certplus_Root_CA_G2.crt
mozilla/Certum_Trusted_Network_CA.crt
mozilla/Certum_Trusted_Network_CA_2.crt
mozilla/Chambers_of_Commerce_Root_-_2008.crt
mozilla/Comodo_AAA_Services_root.crt
mozilla/Cybertrust_Global_Root.crt
mozilla/D-TRUST_Root_Class_3_CA_2_2009.crt
mozilla/D-TRUST_Root_Class_3_CA_2_EV_2009.crt
!mozilla/DST_Root_CA_X3.crt
!mozilla/Deutsche_Telekom_Root_CA_2.crt
2

Please show the outputs of:

echo | openssl s_client -connect letsencrypt.org:443 | head
echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head

And go through:
dpkg-reconfigure ca-certificates
[and make sure "mozilla/ISRG_Root_X1.crt" is enabled "*"]

2 Likes
3

From:
cat /etc/issue
Ubuntu 18.04.6 LTS \n \l

I get the file (from IPv6 and IPv4):

wget https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
--2021-11-14 18:43:30--  https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:1c1d:3d1::1, 167.233.14.167
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:1c1d:3d1::1|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37541418 (36M) [application/x-bzip2]
Saving to: ‘owncloud-complete-20210721.tar.bz2’

owncloud-complete-20210721.tar.bz2       100%[===============================================================================>]  35.80M  7.60MB/s    in 4.9s

2021-11-14 18:43:35 (7.25 MB/s) - ‘owncloud-complete-20210721.tar.bz2’ saved [37541418/37541418]

wget -4 https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
--2021-11-14 18:45:52--  https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 167.233.14.167
Connecting to download.owncloud.org (download.owncloud.org)|167.233.14.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37541418 (36M) [application/x-bzip2]
Saving to: ‘owncloud-complete-20210721.tar.bz2.1’

owncloud-complete-20210721.tar.bz2.1     100%[===============================================================================>]  35.80M  7.41MB/s    in 5.1s

2021-11-14 18:45:58 (7.03 MB/s) - ‘owncloud-complete-20210721.tar.bz2.1’ saved [37541418/37541418]
2 Likes
4

I looked at the "/etc/ca-certificates.conf" file and the following line did not appear to be commented out. Is that what you mean?

mozilla/ISRG_Root_X1.crt

Output requested, thank you.

owcld18:~$ echo | openssl s_client -connect letsencrypt.org:443 | head
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=2:unable to get issuer certificate
issuer= O = Digital Signature Trust Co., CN = DST Root CA X3
DONE
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = lencr.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
owcld18:~$ echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=2:unable to get issuer certificate
issuer= O = Digital Signature Trust Co., CN = DST Root CA X3
DONE
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = acme-v01.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
5

Can you run that again and show the output? I think there was a mistake when you pasted the results to here. Thanks.

2 Likes
6

Sure, here you go.

owcld18:~$ echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=2:unable to get issuer certificate
issuer= O = Digital Signature Trust Co., CN = DST Root CA X3
CONNECTED(00000005)
DONE
---
Certificate chain
 0 s:CN = acme-v01.api.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
1 Like
7

Same (BAD) output.
hmm...
It shows:

But then (correctly) doesn't show that cert in the chain:

It's like your system has engraved the cross-signed version of "ISRG Root X1" and only uses that one.
I really have no idea where that can be happening...
Have you rebooted the system lately? Can you? LOL

1 Like
8

@rg305 Yeah, I am confused by that too which is why I had to see it twice :slight_smile:

@actyler1001 can you try the below command too? The google chains have nothing to do with Lets Encrypt so if it also shows the DST Root CA X3 error message then it would point to maybe a problem with your cert store or less likely an openssl conf problem.

A reboot as Rudy mentioned is worthwhile too.

echo | openssl s_client -connect google.com:443 | head

Thanks

2 Likes
9

Thanks Both you you. Yes, I've rebooted. Here is the output requested... Scratching my head here....

owcld18:~$ uptime
 18:48:56 up 2 min,  1 user,  load average: 0.15, 0.18, 0.08

owcld18:~$ echo | openssl s_client -connect google.com:443 | head
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = *.google.com
verify return:1
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = *.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
DONE
---
1 Like
10

Argh! I was kind of hoping that would fail too :slight_smile:

Ok, what about this command. It is a more "pure" chain to ISRG Root X1. We need the -servername for this domain

echo | openssl s_client -connect valid-isrgrootx1.letsencrypt.org:443 -servername valid-isrgrootx1.letsencrypt.org | head
2 Likes
11

Here we go..

owcld18:~$ echo | openssl s_client -connect valid-isrgrootx1.letsencrypt.org:443 -servername valid-isrgrootx1.letsencrypt.org | head
depth=1 C = US, O = Let's Encrypt, CN = R3
verify error:num=2:unable to get issuer certificate
issuer= O = Digital Signature Trust Co., CN = DST Root CA X3
CONNECTED(00000005)
---
Certificate chain
 0 s:CN = valid-isrgrootx1.letsencrypt.org
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
---
Server certificate
-----BEGIN CERTIFICATE-----
DONE
1 Like
12

@actyler1001 Thanks for all that. Very strange. You will have to wait for Rudy or other openssl or PKI experts to contribute. I am sure someone will be able to sort it. But, I am out of ideas. Good luck.

3 Likes
13

Please show the output of:
find / -name *ISRG*

And compare the files:

/etc/ssl/certs/ISRG_Root_X1.pem
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

[which should both exist and be identical]

2 Likes
14

Okay, so here is the output of that command..

owcld18:~$ sudo find / -name *ISRG*
/snap/core/11993/etc/ssl/certs/ISRG_Root_X1.pem
/snap/core/11993/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/snap/core/11798/etc/ssl/certs/ISRG_Root_X1.pem
/snap/core/11798/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
/etc/ssl/certs/ISRG_Root_X1.pem
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

And a compare with ls -l

owcld18:~$ ls -l /etc/ssl/certs/ISRG_Root_X1.pem
lrwxrwxrwx 1 root root 51 Aug  5  2019 /etc/ssl/certs/ISRG_Root_X1.pem -> /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

owcld18:~$ ls -l /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
-rw-r--r-- 1 root root 1939 Sep 22 04:46 /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

So, not identical. I am guessing the Aug 2019 one needs to go away? And replaced with.......... ?

Wait.... Looks like the one from Aug is a symlink. Just to the sept 22 version... So, hm...

16

I meant for you to compare their contents.
But seeing as one is just a symbolic links to the same file:
/usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt
they must be identical.

I can't see how this will be wrong, but let's confirm that content.
With:
cat /usr/share/ca-certificates/mozilla/ISRG_Root_X1.crt

Which should be:

-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
TzELMAkGA1UEBhMCVVMxKTAnBgNVBAoTIEludGVybmV0IFNlY3VyaXR5IFJlc2Vh
cmNoIEdyb3VwMRUwEwYDVQQDEwxJU1JHIFJvb3QgWDEwHhcNMTUwNjA0MTEwNDM4
WhcNMzUwNjA0MTEwNDM4WjBPMQswCQYDVQQGEwJVUzEpMCcGA1UEChMgSW50ZXJu
ZXQgU2VjdXJpdHkgUmVzZWFyY2ggR3JvdXAxFTATBgNVBAMTDElTUkcgUm9vdCBY
MTCCAiIwDQYJKoZIhvcNAQEBBQADggIPADCCAgoCggIBAK3oJHP0FDfzm54rVygc
h77ct984kIxuPOZXoHj3dcKi/vVqbvYATyjb3miGbESTtrFj/RQSa78f0uoxmyF+
0TM8ukj13Xnfs7j/EvEhmkvBioZxaUpmZmyPfjxwv60pIgbz5MDmgK7iS4+3mX6U
A5/TR5d8mUgjU+g4rk8Kb4Mu0UlXjIB0ttov0DiNewNwIRt18jA8+o+u3dpjq+sW
T8KOEUt+zwvo/7V3LvSye0rgTBIlDHCNAymg4VMk7BPZ7hm/ELNKjD+Jo2FR3qyH
B5T0Y3HsLuJvW5iB4YlcNHlsdu87kGJ55tukmi8mxdAQ4Q7e2RCOFvu396j3x+UC
B5iPNgiV5+I3lg02dZ77DnKxHZu8A/lJBdiB3QW0KtZB6awBdpUKD9jf1b0SHzUv
KBds0pjBqAlkd25HN7rOrFleaJ1/ctaJxQZBKT5ZPt0m9STJEadao0xAH0ahmbWn
OlFuhjuefXKnEgV4We0+UXgVCwOPjdAvBbI+e0ocS3MFEvzG6uBQE3xDk3SzynTn
jh8BCNAw1FtxNrQHusEwMFxIt4I7mKZ9YIqioymCzLq9gwQbooMDQaHWBfEbwrbw
qHyGO0aoSCqI3Haadr8faqU9GY/rOPNk3sgrDQoo//fb4hVC1CLQJ13hef4Y53CI
rU7m2Ys6xt0nUW7/vGT1M0NPAgMBAAGjQjBAMA4GA1UdDwEB/wQEAwIBBjAPBgNV
HRMBAf8EBTADAQH/MB0GA1UdDgQWBBR5tFnme7bl5AFzgAiIyBpY9umbbjANBgkq
hkiG9w0BAQsFAAOCAgEAVR9YqbyyqFDQDLHYGmkgJykIrGF1XIpu+ILlaS/V9lZL
ubhzEFnTIZd+50xx+7LSYK05qAvqFyFWhfFQDlnrzuBZ6brJFe+GnY+EgPbk6ZGQ
3BebYhtF8GaV0nxvwuo77x/Py9auJ/GpsMiu/X1+mvoiBOv/2X/qkSsisRcOj/KK
NFtY2PwByVS5uCbMiogziUwthDyC3+6WVwW6LLv3xLfHTjuCvjHIInNzktHCgKQ5
ORAzI4JMPJ+GslWYHb4phowim57iaztXOoJwTdwJx4nLCgdNbOhdjsnvzqvHu7Ur
TkXWStAmzOVyyghqpZXjFaH3pO3JLF+l+/+sKAIuvtd7u+Nxe5AW0wdeRlN8NwdC
jNPElpzVmbUq4JUagEiuTDkHzsxHpFKVK7q4+63SM1N95R1NbdWhscdCb+ZAJzVc
oyi3B43njTOQ5yOf+1CceWxG1bQVs5ZufpsMljq4Ui0/1lvh+wjChP4kqKOJ2qxq
4RgqsahDYVvTH9w7jXbyLeiNdd8XM2w9U/t7y0Ff/9yi0GE44Za4rF2LN9d11TPA
mRGunUHBcnWEvgJBQl9nJEiU0Zsnvgc/ubhPgXRR4Xq37Z0j4r7g1SgEEzwxA57d
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----
1 Like
17

And after CAT, quick copy paste into notepad++ to use compare feature. They are exactly the same. WTF Ubuntu server, what is going on!?

1 Like
18

I'm running out of ideas on where the problem may be...
which openssl
I get:
/usr/bin/openssl

ls -l /usr/bin/openssl
-rwxr-xr-x 1 root root 723944 Aug 23 17:02 /usr/bin/openssl
1 Like
19

Please also show output through end of all certs for:
openssl s_client -connect acme-v02.api.letsencrypt.org:443 -showcerts

1 Like
20

@actyler1001 Have you resolved this yet?

I have been following this and another thread with similar results. They are on Ubuntu 20 with the same CA cert store version as you and similar symptoms.

Their core problem was they had earlier (and mistakenly) added the R3 intermediate certificate indirectly to the CA store.

Here is their solution. The post by Nummer378 just prior to that describe their situation well. I could not help connect your problem to theirs so hopefully this helps you too.

3 Likes
21

OMG, this fixed it!

Thank you, thank you! Now why the heck was the intermediate there in the first place I wonder?

These are the commands I ran.

cd /etc/ssl
rsync -av certs/  certs_bak
rm -rf certs
mkdir certs
# Checked that the file /usr/share/ca-certificates/mozilla/DST_Root_CA_X3.crt didn't exist
update-ca-certificates

Working!

owcld18:/Installs$ sudo sudo wget https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
--2021-11-19 11:31:43--  https://download.owncloud.org/community/owncloud-complete-20210721.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 167.233.14.167, 2a01:4f8:1c1d:3d1::1
Connecting to download.owncloud.org (download.owncloud.org)|167.233.14.167|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 37541418 (36M) [application/x-bzip2]
Saving to: ‘owncloud-complete-20210721.tar.bz2’

owncloud-complete-20210721.tar. 100%[=====================================================>]  35.80M  5.78MB/s    in 7.1s

2021-11-19 11:31:51 (5.06 MB/s) - ‘owncloud-complete-20210721.tar.bz2’ saved [37541418/37541418]
3 Likes