Tutorial - Testing Mail Protocols with SSL/TLS

[ahaw021]

Hi All

As we start moving towards a more secure internet (not just HTTP) I have noticed more people starting to ask questions about secure mail protocols and how to test.

This tutorial is about that and hopes it helps out in the future

You will need openssl and python but if you are using certbot you should already have these installed.

• Protocols, Definitions
• STARTTLS and Why It’s Significant
• Testing TLS Connectivity with OpenSSL - GMAIL SMTP
• Testing TLS Connectivity with OpenSSL - GMAIL IMAP GMAIL POP
• How To Talk Protocols - Once TLS Connectivity is established
• Protocol Example - IMAP
• Testing Script (Python)
• Challenges with Email Servers (Examples)
• Challenges with Email Clients (Examples)
• Further Work

Andrei

[ahaw021]

Part A - Protocols and Definitions

PROTOCOL - much like a language. 3 Main Protocols in use are POP, IMAP and SMTP. A comparison can be found here: http://www.contactgenie.info/understanding-email-protocols-pop-imap-mapi-eas/. We will focus on IMAP, POP and SMTP as they are easy to find and work with. MAPI and some of the others are Vendor Specific and require exchange/compatible servers to test.

TRANSPORT - how we connect and communicate with the server. Unlike HTTP which has secure and non secure mail protocols have a 3rd option. Below are the definitions of the 3 ways we can talk to a mail server (using any of the above protocols)

A) IMPLICITLY Secure - like HTTPS - all communications are encrypted from the begining
B) EXPLICITY Secure - the closest comparison would be a redirect. Where we start talking HTTP and then the server tells us to move to HTTPS. (This is a very loose comparison)
C) UNSECURE - HTTP essentially

A good overview is here:

SSL vs TLS vs STARTTLS

https://www.limilabs.com/blog/use-ssl-with-smtp

PORT - something we can connect to. A port can be connected to using multiple TRANSPORT methods and may talk multiple PROTOCOLS. There are recommended port configurations for mail protocols and these can be found here: https://www.siteground.com/tutorials/email/pop3-imap-smtp-ports.htm. Think of this like port 80 and 443 in HTTP and HTTPS. We can listen on other ports like 3000 for NodeJS but usually browsers expect to connect on port 80.

[ahaw021]

STARTTLS and Why It’s Significant

One of the things that will catch out many people coming from the web world is the concept of STARTTLS.

If you are used to things being secure on insecure you may wonder can you have a mix of both and why would you. Both are explained in the fastmail link above so I am not going to worry about the theory.

Let’s do some testing.

Having a look at Googles SMTP service we can see port 25 is open. We know that Google is pretty good about security so we can assume that they won’t let people connect with insecure protocols.

However this doesn’t seem to be the case. As we have a session and we can type things in it.

Now let’s see what happens when we try to use the AUTH LOGIN command (how SMTP logs you in)

The error message says we need to run the STARTTLS command first (which we do)

And after a while you will get disconnected.

The key takeaway - while we can connect insecurely we are limited as to what we can do until we tell the server to start using Secure Connections.

[ahaw021]

STARTTLS and Why It’s Significant

Now let’s do the same thing with OpenSSL and the StartTLS command.

Command:

openssl s_client -connect smtp.gmail.com:25 -starttls smtp

We now get a prompt in to which we can type the same commands

Don’t worry to much what the various commands mean as they are specific to the protocol. (We will cover these later).

If we have a look at a wireshark capture of this transaction we will see the following.

We start talking unsecure SMTP protocol and then upgrade to a secure TLS1.2 procotol. That is how the STARTTLS Command works.

[ahaw021]

Testing TLS Connectivity with OpenSSL - GMAIL SMTP

NOTE: for now as long as we get a command prompt we consider the connection to have passed (we are not doing any PROTOCOL testing. We are just checking TRANSPORT)

Port 465 is designated as STMPS or SMTP over SSL and we should be able to connect without the STARTTLS negotiation.

openssl s_client -connect smtp.gmail.com:465

If we try to connect to port 587 the same way we seem to get an error

openssl s_client -connect smtp.gmail.com:587

That is because there is nothing listening for SSL handshake messages and the handshake is not complete

If we add the -starttls smtp command we will be able to connect as the intial unsecure conversation will start a TLS session (like port 25)

openssl s_client -connect smtp.gmail.com:587 -starttls smtp

[ahaw021]

Testing TLS Connectivity with OpenSSL - GMAIL IMAP GMAIL POP

Most providers like GMAIL, YAHOO, ZOHO only provide IMPLICITLY SECURE POP and IMAP interfaces.

Testing these is easy as all we have to do is specify the correct port.

commands:

openssl s_client -connect imap.gmail.com:993

openssl s_client -connect pop.gmail.com:995

NOTE: for now as long as we get a command prompt we consider the connection to have passed (we are not doing any PROTOCOL testing. We are just checking TRANSPORT)

Screenshots also show the wrong host

[ahaw021]

How To Talk Protocols - Once TLS Connectivity is established

From a TLS testing as long as we get a prompt we consider that TLS is working correctly.

We can go further however and test that we are getting good communications (i.e. the server is responding).

To do this we should know the syntax of the commands that the PROTOCOL expects.

Luckily these have bee outlined in the links below

POP COMMAND REFERENCE: https://blog.yimingliu.com/2009/01/23/testing-a-pop3-server-via-telnet-or-openssl/
SMTP COMMAND REFERENCE: https://www.ndchost.com/wiki/mail/test-smtp-auth-telnet
IMAP COMMAND REFERENCE: http://busylog.net/telnet-imap-commands-note/

[ahaw021]

Protocol Example - IMAP

Using the reference above we should be able to test an IMAP command against a GMAIL server.

Let’s establish connectivity once again by running

openssl s_client -connect imap.gmail.com:993
A1 LOGIN changeme@gmail.com changeme
A1 CAPABILITY

If you refer to the references for IMAP above these commands will be explained to you there.

[ahaw021]

Testing Script (Python)

Because I am lazy and syntax is my biggest weakness I write scripts so I don’t have to remember stuff.

Most of the manual learnings of this guide I have put in to a python script which you can find here: https://github.com/ahaw021/SSL-MAIL-PROTOCOLS-TESTING

Let’s Run the same 3 tests as above and see what the script tells us

SMTP GMAIL on PORT 25:

The first Message is the banner from the server

The Second Message is the response of the STARTTLS command (not all servers have the same response)

The third is the peer certificate returned (this is to test self signed certs)

The fourth is the Domains in the certificate (check for hostname errors etc)

The fifth is the cipher suite (so we can see what cipher the server chose)

The next bits are an attempt to login using the protocols defined methods. NOTE: each protocol varies which is why the Encoded strings etc are printed out.

NOTE ALSO: the script uses changeme@gmail.com and changeme as defaults. If you want to use it change these but DO NOT post the output of the last bits

POP GMAIL on PORT 995:

IMAP GMAIL on PORT 993

As you can see the way these protocols authenticate is very different however the certificates and handshakes are the same.

I have updated the code to include the connection information

[ahaw021]

Challenges with Email Servers (Examples)

One of the reasons why I wrote the test script the way I did is to help people isolate challenges with SERVERS vs PROTOCOLS or TRANSPORT.

Sample case one: Google and Insecure Apps

Valid credentials were provided.

More About this here: https://support.google.com/accounts/answer/6010255?hl=en

Sample Case 2: Yahoo No Longer Responding to Service But Port is Open

[ahaw021]

Challenges with Email Clients (Examples)

Clients Don’t always articulate if they support STARTTLS or just IMPLICITLY SECURE connections.

Confusing Example: PHPMAILER:

Opportunistic Encryption - what is it?

PHPMailer/PHPMailer

Confusing Example: ZOHO Mail

What they mean is STARTTLS however as most people use TLS and SSL the same way I can see how people get confused

Good Example: Python SMTPLIB:

How to use STARTTLS and which methods don’t support is clearly outlined

https://docs.python.org/3/library/smtplib.html

Good Example: GMAIL

STARTTLS is clearly Identified.

[ahaw021]

Further Work

I hope you have read this and learnt something.

I think we should get better at describing Secure Mail Protocols and how they work (hence this bit)

I am planning to finish the testing script and integrate it in to a web service so it can help people test a range of PROTOCOLS and TRANSPORT methods.

Writing this took some effort and looking back I understand how people get confused (lots of standards and confusing terminology)

I hope this helps in some way with your secure mail initiatives

Andrei

[guntbert]

Great idea - and well done!
thx a lot

[ebonsi]

Great post! I was able to test my Google Servers and see their CA’s.

Thank you!

E\