Unclear motive for negative server responses

dvo · October 7, 2021, 3:00pm

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:
tuttopepe.saltalafila.online
I ran this command:
https://tuttopepe.saltalafila.online
and
curl 'https://tuttopepe.saltalafila.online/api/v1/articles/array'
It produced unexpected results:
the http request return the warning page (Firefox 78)
Then examining the certificates, the validities are all proper.
10/7/21 => 1/5/2022
9/4/2020 => 9/15/2025 R3
1/20/2021 => 9/30/2024 ISRG ROOT X1

the terminal returns:
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: http://curl.haxx.se/docs/sslcerts.html

Sequence of actions:
Since the auto-renew had not kicked in (I've seen this behaviour a few times now, has something changed?) and I did it manually, without re-directing via lets encrypt (it is already in the sites-enabled/salalafila file) . I did it twice as I missed a sub-domain the first time round.
. I also noticed that
sudo service nginx restart does not kick in and I had to reboot the server to make that command pass (and ensure changes were effectively loaded by nginx) - *this inability to restart nginx without rebooting the server is also recent behaviour..

Then I wondered if the certificates would conflict and therefore re-ran
sudo certbot --nginx -d tuttopepe.saltalafila.online -d artaterme.saltalafila.online -d arta.saltalafila.online -d api.saltalafila.online
this time with option 2 (redirect 80 to 443). and sudo service nginx restart would run.

However... while the browser picks up
https://tuttopepe.saltalafila.online/ as fully encrypted
the curl command
curl 'https://tuttopepe.saltalafila.online/api/v1/articles/array' [...] still returns
(60) SSL certificate problem: Invalid certificate chain

  so this explains the title of post: Unclear motive)

My web server is (include version):
nginx/1.18.0 (Ubuntu)

The operating system my web server runs on is (include version):
Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-88-generic x86_64)

I can login to a root shell on my machine (yes or no, or I don't know):
yes, via mosh and ssh

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 0.40.0

rg305 · October 7, 2021, 6:49pm

What say?:
nginx -t
openssl version

dvo · October 7, 2021, 6:50pm

nginx -t
nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
nginx: configuration file /etc/nginx/nginx.conf test is successful

openssl version
OpenSSL 1.1.1f 31 Mar 2020

rg305 · October 7, 2021, 7:32pm

@dvo
Please show the output of:
curl -Iv https://tuttopepe.saltalafila.online/api/v1/articles/array

dvo · October 7, 2021, 7:34pm

Adding handle: conn: 0x7fba04004400
Adding handle: send: 0
Adding handle: recv: 0
Curl_addHandleToPipeline: length: 1
- Conn 0 (0x7fba04004400) send_pipe: 1, recv_pipe: 0
About to connect() to tuttopepe.saltalafila.online port 443 (#0)
Trying 167.172.183.28...
Connected to tuttopepe.saltalafila.online (167.172.183.28) port 443 (#0)
SSL certificate problem: Invalid certificate chain
Closing connection 0
curl: (60) SSL certificate problem: Invalid certificate chain
More details here: curl - SSL CA Certificates

curl performs SSL certificate verification by default, using a "bundle"
of Certificate Authority (CA) public keys (CA certs). If the default [etc.]

MikeMcQ · October 9, 2021, 3:24pm

@dvo Are you still having problems?

It looks to me like your curl is not using the system CA certificate store. My guess is that the store it is using does not contain ISRG Root X1. Are you using a global .curlrc config which changes the cert store? I am not sure where the .curlrc file is on Ubuntu - you will have to search.

Also, you can override that config by doing something like:

curl -CAcert /etc/ssl/certs/ca-certificates.crt (url)

Can you show the result of: curl -V

The reason I suspect curl is faulty is because you were able to renew your certificates. So, the system CA store seems correct otherwise certbot would fail and it looks like just curl is using wrong store.

For reference, here is the result of my curl to your site - but note the name of the CA cert file is different as I am on Amazon Linux 2 with curl version 7.76.1.

curl -Iv https://tuttopepe.saltalafila.online/api/v1/articles/array
*   Trying 167.172.183.28:443...
* Connected to tuttopepe.saltalafila.online (167.172.183.28) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
*  CAfile: /etc/pki/tls/certs/ca-bundle.crt
*  CApath: none
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
(rest removed)

dvo · October 12, 2021, 5:08am

Hello,

yes this is still behaving in a manner that is unexpected and different than under https calls.

curl 7.30.0 (x86_64-apple-darwin13.0) libcurl/7.30.0 SecureTransport zlib/1.2.5
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtsp smtp smtps telnet tftp
Features: AsynchDNS GSS-Negotiate IPv6 Largefile NTLM NTLM_WB SSL libz

rg305 · October 12, 2021, 5:17am

Try:
apt update
apt install libcurl3-gnutls libcurl4 ca-certificates openssl

dvo · October 12, 2021, 5:21am

I assume this is intended for the server. Having run that, the results is returned as before:

curl: (60) SSL certificate problem: Invalid certificate chain

rg305 · October 12, 2021, 5:22am

Did it update anything?
Please show the output (to see the versions).
To compare with:

ca-certificates is already the newest version (20210119~18.04.2).
libcurl3-gnutls is already the newest version (7.58.0-2ubuntu3.16).
libcurl4 is already the newest version (7.58.0-2ubuntu3.16).
openssl is already the newest version (1.1.1-1ubuntu2.1~18.04.13).

dvo · October 12, 2021, 5:29am

ca-certificates is already the newest version (20210119~20.04.2).
libcurl3-gnutls is already the newest version (7.68.0-1ubuntu2.7).
libcurl3-gnutls set to manually installed.
libcurl4 is already the newest version (7.68.0-1ubuntu2.7).
libcurl4 set to manually installed.
openssl is already the newest version (1.1.1f-1ubuntu2.8).
openssl set to manually installed.
0 upgraded, 0 newly installed, 0 to remove and 4 not upgraded.`

rg305 · October 12, 2021, 5:31am

hmm...

Please show:
curl -Iki https://tuttopepe.saltalafila.online/api/v1/articles/array

dvo · October 12, 2021, 5:37am

[sorry previous post was while entering tunnel]

curl -Iki https://tuttopepe.saltalafila.online/api/v1/articles/array
HTTP/1.1 404 Not Found
Content-Type: text/html; charset=UTF-8
Content-Length: 179054
Connection: keep-alive
Status: 404 Not Found
X-Request-Id: eddc99c4-28e5-4f9e-ac96-3da0979f9adf
X-Runtime: 0.303793
Date: Tue, 12 Oct 2021 05:36:02 GMT
Set-Cookie: __profilin=p%3Dt; path=/; secure; HttpOnly; SameSite=Lax
X-Powered-By: Phusion Passenger(R) 6.0.11
Server: nginx/1.18.0 + Phusion Passenger(R) 6.0.11

rg305 · October 12, 2021, 5:43am

So it can connect...
But it just doesn't trust the chain...

Please show:
grep MIIDSjCCAjKgAwIBAgIQRK /etc/ssl/certs/ca-certificates.crt
grep MIIFazCCA1OgAwIBAgIRAI /etc/ssl/certs/ca-certificates.crt

[The first should fail, the second should be found]

dvo · October 12, 2021, 5:45am

That is correct, the first has an empty response, the latter with:
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw

rg305 · October 12, 2021, 5:52am

And it just keeps getting weirder and weirder...
The first is from "DST Root CA X3" and is expected to have been removed.
The second is from "ISRG Root X1" and is expected to have been included.

And the site uses:

---
Certificate chain
 0 s:CN = tuttopepe.saltalafila.online
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---

So it should have been trusted.

You can get connected without verifying the chain...
You can't trust the chain that is trusted by a very similar system...

hmm...

MITM?

Please show the outputs of:

host tuttopepe.saltalafila.online
echo | openssl s_client -connect tuttopepe.saltalafila.online:443 -servername tuttopepe.saltalafila.online  | head

dvo · October 12, 2021, 6:06am

depth=2 C = US, O = Internet Security Research Group, CN = ISRG Root X1
verify return:1
depth=1 C = US, O = Let's Encrypt, CN = R3
verify return:1
depth=0 CN = tuttopepe.saltalafila.online
verify return:1
CONNECTED(00000003)
---
Certificate chain
 0 s:CN = tuttopepe.saltalafila.online
   i:C = US, O = Let's Encrypt, CN = R3
 1 s:C = US, O = Let's Encrypt, CN = R3
   i:C = US, O = Internet Security Research Group, CN = ISRG Root X1
 2 s:C = US, O = Internet Security Research Group, CN = ISRG Root X1
   i:O = Digital Signature Trust Co., CN = DST Root CA X3
---
DONE

rg305 · October 12, 2021, 6:07am

dvo · October 12, 2021, 6:08am

returning, identically, locally and on server:

tuttopepe.saltalafila.online has address 167.172.183.28

rg305 · October 12, 2021, 6:09am

What? They all verify correctly!
So this is NOT related to OpenSSL...
And there doesn't seem to be any MITM.