How to meet Gmail's new (2016) email tls requirement - red lock

Server
1

As many of you know in early 2016, Gmail, in an effort to increase email security began displaying a bright red lock on emails that aren't encrypted. While there are many articles about it, practically none of them (including Google's!!) actually address HOW TO MEET THE REQUIREMENT to get rid of that unsightly red lock on unencrypted email transit - specifically on how to properly encrypt emails in transit from private linux email servers.

You'd think if they actually cared about email security they would include that somewhere.

Hmmm :thinking:

The Red Lock:

I would buy an email cert if that was the answer. Or download a free email cert from somewhere if that is the answer.

This post suggests that LetsEncrypt is capable of server to server transit encryption which is what the Gmail red lock is all about. It doesn't matter if the content itself is encrypted, only the transit.

I think LetsEncrypt is capable of this. I have my LetsEncrypt certificate working everywhere perfectly - even on imaps 993 for the server. But its not encrypting the server to server connection from Postfix. Even though its in Postfix cert and key with smtp_tls_security_level = may and smtpd_tls_security_level = may.

Is there any way to debug Postfix to make this work?

One post from that link says:

"First, there must be STARTTLS on the outgoing MX. While the LE certs do say 'web server auth, web client auth', no MX appears to be picky about it, and will accept the cert for SMTP regardless."

Not sure how to get STARTTLS on outgoing messages. It seems like its working when I've tried to debug with telnet and openssl - 250 STARTTLS is there but it certainly isn't working because the TLSv1_0 is missing from the top received headers on every email.

And that red lock on Gmail.

Anyone solve this?

1 Like
2

Let’s Encrypt’s certificates and STARTTLS are not required for outgoing e-mails from your Postfix. In outgoing e-mail, Postfix is just the client. Setting smtp_tls_security_level to “may” should be enough for this. In my case, it results in a header in GMail like:

Received: from example.com (example.com. [2001:xxx:xxx:1::1])
        by mx.google.com with ESMTPS id l190si1955709wmb.49.2017.02.17.06.43.52
        for <example@example.com>
        (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256);
        Fri, 17 Feb 2017 06:43:52 -0800 (PST)

Check the source of an e-mail in GMail to confirm.

Likewise, you should check the source of an e-mail in a reply from GMail to your own Postfix. To debug further, you can test the STARTTLS with OpenSSL:

openssl s_client -connect your.postfix.server:25 -starttls smtp

It should connect without an error.

1 Like
3

My wording on that post might have a bit too brief, targeted at people who already know how their forwarders operate.

When you send mail to gmail, the remote server will offer STARTTLS. In order for your postfix, which as Osiris says is a client in this situation, to be able to send the STARTTLS command, it needs a keypair.

In your postfix config you give it two certificates:

smtpd_tls_cert_file = /etc/postfix/server.pem
smtpd_tls_key_file = $smtpd_tls_cert_file

and

smtp_tls_cert_file = /etc/postfix/client.pem
smtp_tls_key_file = $smtp_tls_cert_file

The first is used in the server role, the other in the client role. Using the letsencrypt cert for both is fine.

4

No offence, but that's just wrong.

GMail doesn't use client certificate authentication, so a SMTP client does not have to configure certificates/private keys! Using Let's Encrypt certificates for a client doesn't make any sense, as Let's Encrypt doesn't offer client certificates.

From the Postfix manual:

Do not configure client certificates unless you must present client TLS certificates to one or more servers. Client certificates are not usually needed, and can cause problems in configurations that work well without them. The recommended setting is to let the defaults stand.

Setting smtp_tls_security_level = may is more than enough to have Postfix send emails with TLS encryption. (Unless you've seriously modified your main.cf for the worse by for example by setting smtp_tls_mandatory_ciphers to a cipher which isn't allowed by GMail, so you get a cipher mismatch. The mail logs of Postfix will log that obviously.

2 Likes
5

Hm, appears you are absolutely right. I just tried removing the client cert from mine, and yes it still negotiates.
I am pretty certain that, long long ago, sendmail wouldn’t send starttls without them, which is why I had them in. Now it works without, I tried. Thank you.

So the OPs problem is something else. And if OP’s mailserver will, as server, send starttls, then the needed libraries are installed on it for it to work as client.

1 Like
6

Can you post your Postfix config, and anything in syslog from Postfix (redacting sensitive info)?

7

Simplified it down to a reinstall of postfix & Ubuntu os to make sure a setting wasn’t conflicting somewhere. Here are my current settings that connect successfully to openssl s_client -connect your.postfix.server:25 -starttls smtp

syslog:

Feb 17 13:04:14 mail postfix/pickup[2751]: C121A3001E2: uid=1000 from=<person>
Feb 17 13:04:14 mail postfix/cleanup[3185]: C121A3001E2: message-id=<20170217180414.C121A3001E2@mail.theserver.com>
Feb 17 13:04:14 mail postfix/qmgr[2752]: C121A3001E2: from=<person@mail.theserver.com>, size=326, nrcpt=1 (queue active)
Feb 17 13:04:17 mail postfix/smtp[3187]: C121A3001E2: to=<email@gmail.com>, relay=gmail-smtp-in.l.google.com[64.233.184.27]:25, delay=2.5, delays=0.01/0.01/0.91/1.6, dsn=2.0.0, status=sent (250 2.0.0 OK 1487354657 v11si2558249wmg.13 - gsmtp)
Feb 17 13:04:17 mail postfix/qmgr[2752]: C121A3001E2: removed

main.cf:

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtp_tls_security_level=may
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = mail.theserver.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = localdomain, localhost, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = ipv4
smtpd_tls_loglevel = 1

master.cf:

smtp      inet  n       -       -       -       -       smtpd
pickup    unix  n       -       -       60      1       pickup
cleanup   unix  n       -       -       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
relay     unix  -       -       -       -       -       smtp
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix  -       n       n       -       2       pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}

Still missing (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); in the top received header when sending from my email server.

Normally I’m on iRedMail and have LetsEncrypt as the postfix cert and key. Tried sending emails through Roundcube, iPhone smtp, port 587, port 465, command-line mail - all show the email as unencrypted.

Gmail can send TLS to me (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); is in the top header from Google - at least with the iRedMail settings.

Any idea what can be wrong or any way to debug?

8

Could you post all the headers?

9

This is the full headers from the reinstalled, base settings for postfix listed above & sent with command-line mail:

Delivered-To: email@gmail.com
Received: by 10.223.172.112 with SMTP id v103csp256550wrc;
        Fri, 17 Feb 2017 10:16:05 -0800 (PST)
X-Received: by 10.99.211.21 with SMTP id b21mr11819604pgg.166.1487355365449;
        Fri, 17 Feb 2017 10:16:05 -0800 (PST)
Return-Path: <person@mail.theserver.com>
Received: from mail.theserver.com (mail.theserver.com. [191.xxx.xxx.xx7])
        by mx.google.com with ESMTP id o19si10958171pgk.11.2017.02.17.10.16.04
        for <email@gmail.com>;
        Fri, 17 Feb 2017 10:16:05 -0800 (PST)
Received-SPF: pass (google.com: best guess record for domain of person@mail.theserver.com designates 191.xxx.xxx.xx7 as permitted sender) client-ip=191.xxx.xxx.xx7;
Authentication-Results: mx.google.com;
       spf=pass (google.com: best guess record for domain of person@mail.theserver.com designates 191.xxx.xxx.xx7 as permitted sender) smtp.mailfrom=person@mail.theserver.com
Received: by mail.theserver.com (Postfix, from userid 1000) id 607813008F8; Fri, 17 Feb 2017 13:16:03 -0500 (EST)
To: email@gmail.com
Subject: Hello
Message-Id: <20170217181603.607813008F8@mail.theserver.com>
Date: Fri, 17 Feb 2017 13:16:03 -0500 (EST)
From: person@mail.theserver.com

And here is the headers from the iRedMail settings & sent with Roundcube:

Delivered-To: email@gmail.com
Received: by 10.223.172.112 with SMTP id v103csp2225611wrc;
        Thu, 16 Feb 2017 19:22:31 -0800 (PST)
X-Received: by 10.98.202.139 with SMTP id y11mr6629482pfk.98.1487301751426;
        Thu, 16 Feb 2017 19:22:31 -0800 (PST)
Return-Path: <person@theserver.com>
Received: from mail.theserver.com (mail.theserver.com. [191.xxx.xxx.xx7])
        by mx.google.com with ESMTP id e7si8896912pfg.86.2017.02.16.19.22.30
        for <email@gmail.com>;
        Thu, 16 Feb 2017 19:22:31 -0800 (PST)
Received-SPF: pass (google.com: domain of person@theserver.com designates 191.xxx.xxx.xx7 as permitted sender) client-ip=191.xxx.xxx.xx7;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of person@theserver.com designates 191.xxx.xxx.xx7 as permitted sender) smtp.mailfrom=person@theserver.com
Received: from mail.theserver.com (mail.theserver.com [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.theserver.com (Postfix) with ESMTPSA id 62A2C200FFD for <email@gmail.com>; Thu, 16 Feb 2017 22:22:30 -0500 (EST)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Date: Thu, 16 Feb 2017 22:22:30 -0500
From: person <person@theserver.com>
To: email@gmail.com
Subject: Hello
Message-ID: <bf11ca285ab790f69816ac3acfde57d6@theserver.com>
X-Sender: person@theserver.com
User-Agent: Roundcube Webmail

This one has the initial internal connection as TLS, but the end connection to Gmail is unencrypted. smtp_tls_security_level is set to may with these iRedMail settings and won’t encrypt.

So strange.

All of the few tutorials I’ve found make it seem so easy to fix. What can possibly be going on?

10

Just to check the obvious: I assume you’ve restarted Postfix since making the config changes?

As a next debugging step I would run

sudo tcpdump 'port 25' -w packet.log -s 65535

to capture all port 25 traffic, then send a piece of mail, kill the tcpdump, copy packet.log locally, and open it with Wireshark. Do you see STARTTLS in the returned headers? Do you see Postfix attempting to use the STARTTLS command?

You could also change smtp_tls_security_level from “may” to “encrypt”, which is in general probably too strict, but would cause Postfix to error if it can’t use TLS, which might get you more information.

1 Like
11

You might also want to check your mail log. Most of the time something like /var/log/maillog or /var/log/mail.log.

12

Yes postfix has been restarted many times and has been restarted right before this packet with the postfix config listed above.

After running sudo tcpdump 'port 25' -w packet.log -s 65535 and sending an email from the command-line with echo "Text" | mail -s "Subject" email@gmail.com the email is received from Gmail (unencrypted :dizzy_face:).

Opening the packet with Wireshark shows no mention of STARTTLS anywhere

SMTP	122	S: 220 mx.google.com ESMTP i71si11922983pfa.238 - gsmtp
SMTP	95	C: EHLO mail.theserver.com
SMTP	122	S: 250 mx.google.com at your service, [191.xxx.xxx.xx7]
SMTP	141	S: 250 SIZE 157286400 | 250 8BITMIME | 250 ENHANCEDSTATUSCODES | 250 SMTPUTF8
SMTP	117	C: MAIL FROM:<person@mail.theserver.com> SIZE=331
SMTP	96	C: RCPT TO:<email@gmail.com>

In the past, I’ve tried changing smtp_tls_security_level to encrypt - causing the email to hold up until I change it back to may and then it delivers (unencrypted).

After checking the /var/log/mail.log after setting it to encrypt - I couldn’t see any noticeable information or indication of the postfix error that was happening. Not sure if postfix would log the error there or if the logging was set correctly. It was set to smtp_tls_loglevel=1 and tried smtp_tls_loglevel=4 but didn’t see anything with that change either.

Not sure if it means anything but I ran this command:

person@mail:~$ telnet localhost 25
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mail.theserver.com ESMTP Postfix (Ubuntu)
EHLO test
250-mail.theserver.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

And STARTTLS is in telnet. Again, not sure what that means - if anything at all. Also ESMTP should be ESMTPS or ESMTPSA, maybe. Maybe not in telnet - but definitely on the email headers. Again, not sure if any of that is relevant at all.

Are there more debug tests that can be run?

13

I believe the STARTTLS support of your SMTP server is not directly related to this. If you’re sending mail to gmail, gmail won’t contact your SMTP server.

What’s rather odd, on the other hand, is that Gmail’s SMTP server does not offer STARTTLS when you connect to it. When I connect to Gmail’s SMTP server, I get the following response:

$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.206.26...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP m139si3888485wmb.28 - gsmtp
EHLO example.com
250-mx.google.com at your service, [redacted]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

For some reason, that doesn’t seem to match what you’re getting. I see two possible explanations for this:

  • Gmail is selectively disabling some of those options based on the IP address you’re connecting from, or the hostname provided with the EHLO command. This could be tested relatively easily by using telnet from a different IP address, or using it with a different EHLO hostname (such as example.com). I’m not sure why they would be doing this, perhaps a false positive of some heuristic for SMTP servers that are known to have problems with STARTTLS. (I really hope something like that doesn’t exist in 2017, just guessing here.)
  • You’re not actually talking to Gmail’s SMTP server directly, but rather a man-in-the-middle. This could be some kind of “security” software, your ISP (perhaps in an attempt to prevent outgoing spam) or an actual attacker trying to intercept messages by stripping STARTTLS. Much like HTTPS interception software is known to decrease connection security and introduce vulnerabilities, it’s reasonable to expect SMTP proxies to suffer from similar problems, though I personally have not heard of such cases before.

Some further research shows that STARTTLS stripping does indeed appear to be a thing (coincidentally, that article was written by @jsha :smile:).

4 Likes
14

The ISP stripping STARTTLS is looking more and more likely. I’m thinking they strip the STARTTLS from the ipv4’s and leave it on the ipv6’s.

I get the STARTTLS when connecting from my mail server via telnet to gmail:

person@mail$ telnet gmail-smtp-in.l.google.com 25
Trying 2607:f8b0:400e:c04::1a...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP b28si12197106pfk.134 - gsmtp
EHLO mail.theserver.com
250-mx.google.com at your service, [2602:xxxx:xxx:xxx:xxx0::]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

Strangely, this command to disable ipv6:

sudo sh -c 'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6'

Causes the telnet response to remove STARTTLS:

person@mail:~$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.206.26...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP q6si15677925wra.224 - gsmtp
EHLO mail.theserver.com
250-mx.google.com at your service, [191.xxx.xxx.xx7]
250-SIZE 157286400
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 SMTPUTF8

But blacklisting ipv6 in /etc/modprobe.d/blacklist.conf with blacklist ipv6 alone keeps serving STARTTLS with that telnet to Gmail. Although the Wireshark packet headers listening on port 25 still don’t mention STARTTLS and the emails continue to be unecrypted.

Connecting from a different server:

otherperson@otherserver:~$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.198.27...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP c85si206283oig.73 - gsmtp
EHLO mail.theserver.com
250-mx.google.com at your service, [63.xxx.xxx.xxx]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

This other server is able to send encrypted email, TLS in top received header, free from the Gmail red lock. It is configured differently and has a different ISP.

Since the ipv6 does have STARTTLS listed, they must be stripping the STARTTLS from the ipv4’s and leaving it on the ipv6’s. Can the ipv6’s be used for email?

I was running into Undeliverable issues with Gmail saying that they didn’t like my ipv6 ip address for various reasons. I assumed that it was because it was missing the last 8 numbers 2602:xxxx:xxx:xxx:xxx0:: (double semicolon) so I set inet_protocols=ipv4 in main.cf and it accepted the ipv4.

Earlier, on a previous os install, I had setup a webserver proxy and the same thing happened when I tried to view Google’s homepage - said it detected strange behavior with the ipv6 and couldn’t continue - assuming the ipv6 was strange, I changed to ipv4 only and it accepted it.

Does my ipv6 look normal?

2602:xxxx:xxx:xxx:xxx0:: - double semicolon at end

It’s strange that any ISP would restrict STARTTLS to begin with. It’s forcing people to be less secure. Google would have a fit if they found that out.

15

Sure, no problem. Postfix supports IPv6.

2602 isn't in the list of special IPv6 addresses so should be good. Also, you can connect to an IPv6 address (i.e., GMail) with telnet perfectly, so all works :wink:

16

The only requirement that I’m aware of is that your IPv6 address needs to have a valid PTR record (reverse DNS), i.e. nslookup [your IPv6 address] should resolve to your mail server’s hostname and vice versa. (The same goes for IPv4, AFAIK.)

Another option you might want to explore if the IPv6 option doesn’t work out: Apparently postfix allows you to set the TLS security level on a per-domain basis - I’m not all that familiar with postfix, so I’m not sure about the specifics on this. Here’s an article with some details. I’m not sure how this would interact with the proxy and lack of STARTTLS command though, perhaps postfix will refuse to send the messages or the proxy will break things.

17

I was about to suggest @seveneleven could try to connect to GMail through port 465 with TLS to bypass STARTTLS, but it seems GMail doesn't offer that route (any more?). Yes yes, I know it's deprecated, but it could have been a solution to this MitM stuff.. But I'm afraid it's not.

18

I think 465 and 587 are only used for MUA to MTA communication in practice (i.e. something like Thunderbird connecting to Google’s outgoing SMTP server at smtp.gmail.com).

MTA to MTA communication is done on port 25 exclusively, IIRC.

1 Like
19

Well, there was a time STARTTLS didn’t exist and using a dedicated port for TLS was the only way to encrypt your mail :slight_smile:

20

@pfg is correct that MTA-to-MTA traffic is always on port 25. @osiris is also right that port 465 predated STARTTLS. It’s not clear to me whether it was every intended for SMTPS on 465 to take over MTA-to-MTA traffic, or whether it was intended just for MUAs.

I also agree that this looks very much like your ISP is stripping STARTTLS. Some off-the-shelf network equipment does this. I wouldn’t be surprised if the default configurations only did it for IPv6. @seveneleven, you should contact your ISP and ask them to disable that feature. They may not even be aware that it’s on!

2 Likes