Possible maelström of certs confusing matters

I was running the following commands and getting odd results invoking OpenSSL098, when version 1.1.1f is running.

openssl s_client -connect market.saltalafila.online:443 -cert certname
Error opening client certificate private key file certname
49295:error:02001002:system library:fopen:No such file or directory:/SourceCache/OpenSSL098/OpenSSL098-52.8.4/src/crypto/bio/bss_file.c:356:fopen('certname','r')
49295:error:20074002:BIO routines:FILE_CTRL:system lib:/SourceCache/OpenSSL098/OpenSSL098-52.8.4/src/crypto/bio/bss_file.c:358:
unable to load client certificate private key file

$ openssl s_client -connect market.saltalafila..online:443 -certform format
CONNECTED(00000003)
49310:error:1407742E:SSL routines:SSL23_GET_SERVER_HELLO:tlsv1 alert protocol version:/SourceCache/OpenSSL098/OpenSSL098-52.8.4/src/ssl/s23_clnt.c:593:

The domain is running on a Ubuntu 20.04.3 LTS (GNU/Linux 5.4.0-100-generic x86_64) instance.

Then invoking openssl s_client -connect market.saltalafila.online:443 was showing a certificate that was not in the /etc/letsencrypt/live directories with the name: "artaterme.saltalafila.online"

I thus went and combed through the certificates, where yours truly had doubled some up...
sudo certbot delete --cert-name #{unnecessary_cert}
Yet now odder things are happening:

$ openssl s_client -connect market.saltalafila.online:443 -cert certname
Can't open certname for reading, No such file or directory
139806406096192:error:02001002:system library:fopen:No such file or directory:../crypto/bio/bss_file.c:69:fopen('certname','r')
139806406096192:error:2006D080:BIO routines:BIO_new_file:no such file:../crypto/bio/bss_file.c:76:
unable to load client certificate private key file

$ openssl s_client -connect market.saltalafila.online:443 -certform format
s_client: Invalid format "format" for -certform
s_client: Use -help for summary.

One can connect via browser to market.saltalafila.online and the browser reports everything fine. Yet openssl s_client -connect market.saltalafila.online:443 is still pumping out a reference to a cert with the apparent name 'artaterme.saltalafila.online'. No file with that name is found on the server.

can this situation be cleaned up?

Your cert looks fine. You have 5 domain names in the SANs list but that is fine too. See here for easier viewing of what your server sends.

Also, what does this show:

sudo certbot certificates
4 Likes

There are numerous odd things here. We probably need more information about what you're attempting to do.

I noticed the following oddities:

When using openssl s_client together with -cert, you're attempting to use TLS client authentication. TLS client authentication is rarely used in the web domain and the server market.saltalafila.online is also not requesting any client certificate. So any options relating to client authentication ( -certform, -cert) are not valid for what you're attempting to do.


It appears as if this might actually be OpenSSL v0.9.8. OpenSSL 0.9.8 is from 2005 (last patch released 2015) and does not support any modern, secure TLS version such as TLS v1.2. Your server however is configured to only accept TLS v1.2 and higher (a common choice nowadays). That means that it's impossible to connect with an ancient OpenSSL version to that server. That OpenSSL version also has numerous security issues.


A certificate may contain multiple alternative names (SANs). OpenSSL might only print the Common Name (CN), which is usually just one of the SANs. As long as there is no certificate validation error, all is fine.

5 Likes

I hade done that around the cleansing phase.
It shows information for 6 certificates, all of which are expected and distinct

So then the connect does not refere to the cert name but probably the first domain being handle by the cert?

This issue arose as communication with the Telegram API was failing because of this error:

OpenSSL::SSL::SSLError (SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)):

So the digging began. I am going to hit the pause button on this for a while... tackle it with a fresh mind later

openssl will show the Common Name in the cert and shows the SANs list separate.

The name of the file the server used is not visible to any client including openssl

2 Likes

That did clarify one element.

@Nummer378

Although the certificate situation seems ironed out, it may have been a garden path: the situation is still at square 1.

bundle exec rake telegram:bot:set_webhook RAILS_ENV=development
Setting webhook for default...
rake aborted!
OpenSSL::SSL::SSLError: SSL_connect returned=1 errno=0 state=error: certificate verify failed (certificate has expired)

There are 6 certificates on this server, with validities of 84, 76, 76, 89, 34 and 51 days.
So the query must be routing to someplace unexpected and hitting a hidden cert.

From the detailed stack trace there are two inflexion points I deemed worthy of intercepting.

From the gem:
/shared/bundle/ruby/2.6.0/gems/telegram-bot-0.15.4/lib/telegram/bot/client.rb:73:in `request'
is calling method

      def http_request(uri, body)
        client.post(uri, body)
      end

and the uri is
https://market.saltalafila.online//bot#{bot_id}:#{bot_token}/setWebhook
and the final step where the failure occurs:

      if hostname && @ssl_socket.respond_to?(:hostname=)
        @ssl_socket.hostname = hostname
      end
      Rails.logger.info @ssl_socket.hostname
      @ssl_socket.connect

returns market.saltalafila.online

It looks like your software is looking for it. What are you using to host the webhook, and where do you config their certificates?

1 Like

This is a common error usually related to the expiry of DST Root CA X3 last year.

Fixing this most likely requires updating your client system trust store (ca-certificates) to include ISRG Root X1. If you're running OpenSSL < 1.1, additional workarounds must be applied.

2 Likes

Sorry, I slipped in an edit there just before your post.

2 Likes

What are you trying to connect to? (Inside the bot)

Your TLS config looks fine. Error is likely elsewhere. SSL Server Test: market.saltalafila.online (Powered by Qualys SSL Labs)

1 Like

I am uncertain if I have a solid answer to that. Based on the code above and this is the point in the stack trace where the action stops executing and throws the certificate has expired error, the information logging that domain market.saltalafila.online which, yes, everything looks fine for TLS.

I have yet to uncover how/where this all gets routed to.

Check if you're using openssl version <1.1.0 as @Nummer378 said.

1 Like

I opened the post with 'when version 1.1.1f is running'. That is what openssl version returns.

Now did stock Ubuntu have an 0.98 version lurking around? where could I find that out?

I don't know. apt list --installed --all-versions | grep openssl maybe does.

But maybe you can check what version ruby is trying to invoke.

1 Like

does not show anything other than 1.1.1f

I think this thread can be closed, for I have devised a test where another app on an older server is connecting properly to set Telegram webhooks, but this newer version not. It may not be Ruby, but the library that is the culprit. More digging...

For posterity's sake, capturing the arguments of http_request(uri, body) allowed to find a way through the immediate issue and clearly pointing to a problem with the telegram-bot gem.

The uri value is misleading, the body instead is telling how it is connecting to the API:
{:url=>"https://market.saltalafila.online/telegram/#some_uniq_code}", :certificate=>nil, :ip_address=>nil, :drop_pending_updates=>false}
If one connects directly via the API webhooks curl command with just the url parameter (no mention of certificates or ip_addresses, the webhook is set. Then a command to the application hits the controller and is logged as any normal action.

The gem is somehow at fault here.

This is the http client you're talking about, right?

You don't set a certificate in the client, if you don't have a client authentication certificate. (Which Let's Encrypt ones are, but when used as client authentication certificates -- I haven't seen this on the community yet).

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.