How to meet Gmail's new (2016) email tls requirement - red lock

Can you post your Postfix config, and anything in syslog from Postfix (redacting sensitive info)?

Simplified it down to a reinstall of postfix & Ubuntu os to make sure a setting wasn’t conflicting somewhere. Here are my current settings that connect successfully to openssl s_client -connect your.postfix.server:25 -starttls smtp

syslog:

Feb 17 13:04:14 mail postfix/pickup[2751]: C121A3001E2: uid=1000 from=<person>
Feb 17 13:04:14 mail postfix/cleanup[3185]: C121A3001E2: message-id=<20170217180414.C121A3001E2@mail.theserver.com>
Feb 17 13:04:14 mail postfix/qmgr[2752]: C121A3001E2: from=<person@mail.theserver.com>, size=326, nrcpt=1 (queue active)
Feb 17 13:04:17 mail postfix/smtp[3187]: C121A3001E2: to=<email@gmail.com>, relay=gmail-smtp-in.l.google.com[64.233.184.27]:25, delay=2.5, delays=0.01/0.01/0.91/1.6, dsn=2.0.0, status=sent (250 2.0.0 OK 1487354657 v11si2558249wmg.13 - gsmtp)
Feb 17 13:04:17 mail postfix/qmgr[2752]: C121A3001E2: removed

main.cf:

smtpd_banner = $myhostname ESMTP $mail_name (Ubuntu)
biff = no
append_dot_mydomain = no
readme_directory = no
smtpd_tls_cert_file=/etc/ssl/certs/ssl-cert-snakeoil.pem
smtpd_tls_key_file=/etc/ssl/private/ssl-cert-snakeoil.key
smtp_tls_security_level=may
smtpd_use_tls=yes
smtpd_tls_session_cache_database = btree:${data_directory}/smtpd_scache
smtp_tls_session_cache_database = btree:${data_directory}/smtp_scache
smtpd_relay_restrictions = permit_mynetworks permit_sasl_authenticated defer_unauth_destination
myhostname = mail.theserver.com
alias_maps = hash:/etc/aliases
alias_database = hash:/etc/aliases
mydestination = localdomain, localhost, localhost.localdomain, localhost
relayhost =
mynetworks = 127.0.0.0/8 [::ffff:127.0.0.0]/104 [::1]/128
mailbox_command = procmail -a "$EXTENSION"
mailbox_size_limit = 0
recipient_delimiter = +
inet_interfaces = all
inet_protocols = ipv4
smtpd_tls_loglevel = 1

master.cf:

smtp      inet  n       -       -       -       -       smtpd
pickup    unix  n       -       -       60      1       pickup
cleanup   unix  n       -       -       -       0       cleanup
qmgr      unix  n       -       n       300     1       qmgr
tlsmgr    unix  -       -       -       1000?   1       tlsmgr
rewrite   unix  -       -       -       -       -       trivial-rewrite
bounce    unix  -       -       -       -       0       bounce
defer     unix  -       -       -       -       0       bounce
trace     unix  -       -       -       -       0       bounce
verify    unix  -       -       -       -       1       verify
flush     unix  n       -       -       1000?   0       flush
proxymap  unix  -       -       n       -       -       proxymap
proxywrite unix -       -       n       -       1       proxymap
smtp      unix  -       -       -       -       -       smtp
relay     unix  -       -       -       -       -       smtp
showq     unix  n       -       -       -       -       showq
error     unix  -       -       -       -       -       error
retry     unix  -       -       -       -       -       error
discard   unix  -       -       -       -       -       discard
local     unix  -       n       n       -       -       local
virtual   unix  -       n       n       -       -       virtual
lmtp      unix  -       -       -       -       -       lmtp
anvil     unix  -       -       -       -       1       anvil
scache    unix  -       -       -       -       1       scache
maildrop  unix  -       n       n       -       -       pipe
  flags=DRhu user=vmail argv=/usr/bin/maildrop -d ${recipient}
uucp      unix  -       n       n       -       -       pipe
  flags=Fqhu user=uucp argv=uux -r -n -z -a$sender - $nexthop!rmail ($recipient)
ifmail    unix  -       n       n       -       -       pipe
  flags=F user=ftn argv=/usr/lib/ifmail/ifmail -r $nexthop ($recipient)
bsmtp     unix  -       n       n       -       -       pipe
  flags=Fq. user=bsmtp argv=/usr/lib/bsmtp/bsmtp -t$nexthop -f$sender $recipient
scalemail-backend unix  -       n       n       -       2       pipe
  flags=R user=scalemail argv=/usr/lib/scalemail/bin/scalemail-store ${nexthop} ${user} ${extension}
mailman   unix  -       n       n       -       -       pipe
  flags=FR user=list argv=/usr/lib/mailman/bin/postfix-to-mailman.py
  ${nexthop} ${user}

Still missing (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); in the top received header when sending from my email server.

Normally I’m on iRedMail and have LetsEncrypt as the postfix cert and key. Tried sending emails through Roundcube, iPhone smtp, port 587, port 465, command-line mail - all show the email as unencrypted.

Gmail can send TLS to me (version=TLS1_2 cipher=ECDHE-RSA-CHACHA20-POLY1305 bits=256/256); is in the top header from Google - at least with the iRedMail settings.

Any idea what can be wrong or any way to debug?

Could you post all the headers?

This is the full headers from the reinstalled, base settings for postfix listed above & sent with command-line mail:

Delivered-To: email@gmail.com
Received: by 10.223.172.112 with SMTP id v103csp256550wrc;
        Fri, 17 Feb 2017 10:16:05 -0800 (PST)
X-Received: by 10.99.211.21 with SMTP id b21mr11819604pgg.166.1487355365449;
        Fri, 17 Feb 2017 10:16:05 -0800 (PST)
Return-Path: <person@mail.theserver.com>
Received: from mail.theserver.com (mail.theserver.com. [191.xxx.xxx.xx7])
        by mx.google.com with ESMTP id o19si10958171pgk.11.2017.02.17.10.16.04
        for <email@gmail.com>;
        Fri, 17 Feb 2017 10:16:05 -0800 (PST)
Received-SPF: pass (google.com: best guess record for domain of person@mail.theserver.com designates 191.xxx.xxx.xx7 as permitted sender) client-ip=191.xxx.xxx.xx7;
Authentication-Results: mx.google.com;
       spf=pass (google.com: best guess record for domain of person@mail.theserver.com designates 191.xxx.xxx.xx7 as permitted sender) smtp.mailfrom=person@mail.theserver.com
Received: by mail.theserver.com (Postfix, from userid 1000) id 607813008F8; Fri, 17 Feb 2017 13:16:03 -0500 (EST)
To: email@gmail.com
Subject: Hello
Message-Id: <20170217181603.607813008F8@mail.theserver.com>
Date: Fri, 17 Feb 2017 13:16:03 -0500 (EST)
From: person@mail.theserver.com

And here is the headers from the iRedMail settings & sent with Roundcube:

Delivered-To: email@gmail.com
Received: by 10.223.172.112 with SMTP id v103csp2225611wrc;
        Thu, 16 Feb 2017 19:22:31 -0800 (PST)
X-Received: by 10.98.202.139 with SMTP id y11mr6629482pfk.98.1487301751426;
        Thu, 16 Feb 2017 19:22:31 -0800 (PST)
Return-Path: <person@theserver.com>
Received: from mail.theserver.com (mail.theserver.com. [191.xxx.xxx.xx7])
        by mx.google.com with ESMTP id e7si8896912pfg.86.2017.02.16.19.22.30
        for <email@gmail.com>;
        Thu, 16 Feb 2017 19:22:31 -0800 (PST)
Received-SPF: pass (google.com: domain of person@theserver.com designates 191.xxx.xxx.xx7 as permitted sender) client-ip=191.xxx.xxx.xx7;
Authentication-Results: mx.google.com;
       spf=pass (google.com: domain of person@theserver.com designates 191.xxx.xxx.xx7 as permitted sender) smtp.mailfrom=person@theserver.com
Received: from mail.theserver.com (mail.theserver.com [127.0.0.1]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by mail.theserver.com (Postfix) with ESMTPSA id 62A2C200FFD for <email@gmail.com>; Thu, 16 Feb 2017 22:22:30 -0500 (EST)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII; format=flowed
Content-Transfer-Encoding: 7bit
Date: Thu, 16 Feb 2017 22:22:30 -0500
From: person <person@theserver.com>
To: email@gmail.com
Subject: Hello
Message-ID: <bf11ca285ab790f69816ac3acfde57d6@theserver.com>
X-Sender: person@theserver.com
User-Agent: Roundcube Webmail

This one has the initial internal connection as TLS, but the end connection to Gmail is unencrypted. smtp_tls_security_level is set to may with these iRedMail settings and won’t encrypt.

So strange.

All of the few tutorials I’ve found make it seem so easy to fix. What can possibly be going on?

Just to check the obvious: I assume you’ve restarted Postfix since making the config changes?

As a next debugging step I would run

sudo tcpdump 'port 25' -w packet.log -s 65535

to capture all port 25 traffic, then send a piece of mail, kill the tcpdump, copy packet.log locally, and open it with Wireshark. Do you see STARTTLS in the returned headers? Do you see Postfix attempting to use the STARTTLS command?

You could also change smtp_tls_security_level from “may” to “encrypt”, which is in general probably too strict, but would cause Postfix to error if it can’t use TLS, which might get you more information.

1 Like

You might also want to check your mail log. Most of the time something like /var/log/maillog or /var/log/mail.log.

Yes postfix has been restarted many times and has been restarted right before this packet with the postfix config listed above.

After running sudo tcpdump 'port 25' -w packet.log -s 65535 and sending an email from the command-line with echo "Text" | mail -s "Subject" email@gmail.com the email is received from Gmail (unencrypted :dizzy_face:).

Opening the packet with Wireshark shows no mention of STARTTLS anywhere

SMTP	122	S: 220 mx.google.com ESMTP i71si11922983pfa.238 - gsmtp
SMTP	95	C: EHLO mail.theserver.com
SMTP	122	S: 250 mx.google.com at your service, [191.xxx.xxx.xx7]
SMTP	141	S: 250 SIZE 157286400 | 250 8BITMIME | 250 ENHANCEDSTATUSCODES | 250 SMTPUTF8
SMTP	117	C: MAIL FROM:<person@mail.theserver.com> SIZE=331
SMTP	96	C: RCPT TO:<email@gmail.com>

In the past, I’ve tried changing smtp_tls_security_level to encrypt - causing the email to hold up until I change it back to may and then it delivers (unencrypted).

After checking the /var/log/mail.log after setting it to encrypt - I couldn’t see any noticeable information or indication of the postfix error that was happening. Not sure if postfix would log the error there or if the logging was set correctly. It was set to smtp_tls_loglevel=1 and tried smtp_tls_loglevel=4 but didn’t see anything with that change either.

Not sure if it means anything but I ran this command:

person@mail:~$ telnet localhost 25
Trying ::1...
Trying 127.0.0.1...
Connected to localhost.
Escape character is '^]'.
220 mail.theserver.com ESMTP Postfix (Ubuntu)
EHLO test
250-mail.theserver.com
250-PIPELINING
250-SIZE 10240000
250-VRFY
250-ETRN
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-8BITMIME
250 DSN

And STARTTLS is in telnet. Again, not sure what that means - if anything at all. Also ESMTP should be ESMTPS or ESMTPSA, maybe. Maybe not in telnet - but definitely on the email headers. Again, not sure if any of that is relevant at all.

Are there more debug tests that can be run?

I believe the STARTTLS support of your SMTP server is not directly related to this. If you’re sending mail to gmail, gmail won’t contact your SMTP server.

What’s rather odd, on the other hand, is that Gmail’s SMTP server does not offer STARTTLS when you connect to it. When I connect to Gmail’s SMTP server, I get the following response:

$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.206.26...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP m139si3888485wmb.28 - gsmtp
EHLO example.com
250-mx.google.com at your service, [redacted]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

For some reason, that doesn’t seem to match what you’re getting. I see two possible explanations for this:

  • Gmail is selectively disabling some of those options based on the IP address you’re connecting from, or the hostname provided with the EHLO command. This could be tested relatively easily by using telnet from a different IP address, or using it with a different EHLO hostname (such as example.com). I’m not sure why they would be doing this, perhaps a false positive of some heuristic for SMTP servers that are known to have problems with STARTTLS. (I really hope something like that doesn’t exist in 2017, just guessing here.)
  • You’re not actually talking to Gmail’s SMTP server directly, but rather a man-in-the-middle. This could be some kind of “security” software, your ISP (perhaps in an attempt to prevent outgoing spam) or an actual attacker trying to intercept messages by stripping STARTTLS. Much like HTTPS interception software is known to decrease connection security and introduce vulnerabilities, it’s reasonable to expect SMTP proxies to suffer from similar problems, though I personally have not heard of such cases before.

Some further research shows that STARTTLS stripping does indeed appear to be a thing (coincidentally, that article was written by @jsha :smile:).

4 Likes

The ISP stripping STARTTLS is looking more and more likely. I’m thinking they strip the STARTTLS from the ipv4’s and leave it on the ipv6’s.

I get the STARTTLS when connecting from my mail server via telnet to gmail:

person@mail$ telnet gmail-smtp-in.l.google.com 25
Trying 2607:f8b0:400e:c04::1a...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP b28si12197106pfk.134 - gsmtp
EHLO mail.theserver.com
250-mx.google.com at your service, [2602:xxxx:xxx:xxx:xxx0::]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

Strangely, this command to disable ipv6:

sudo sh -c 'echo 1 > /proc/sys/net/ipv6/conf/all/disable_ipv6'

Causes the telnet response to remove STARTTLS:

person@mail:~$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.206.26...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP q6si15677925wra.224 - gsmtp
EHLO mail.theserver.com
250-mx.google.com at your service, [191.xxx.xxx.xx7]
250-SIZE 157286400
250-8BITMIME
250-ENHANCEDSTATUSCODES
250 SMTPUTF8

But blacklisting ipv6 in /etc/modprobe.d/blacklist.conf with blacklist ipv6 alone keeps serving STARTTLS with that telnet to Gmail. Although the Wireshark packet headers listening on port 25 still don’t mention STARTTLS and the emails continue to be unecrypted.

Connecting from a different server:

otherperson@otherserver:~$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.198.27...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP c85si206283oig.73 - gsmtp
EHLO mail.theserver.com
250-mx.google.com at your service, [63.xxx.xxx.xxx]
250-SIZE 157286400
250-8BITMIME
250-STARTTLS
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-CHUNKING
250 SMTPUTF8

This other server is able to send encrypted email, TLS in top received header, free from the Gmail red lock. It is configured differently and has a different ISP.


Since the ipv6 does have STARTTLS listed, they must be stripping the STARTTLS from the ipv4’s and leaving it on the ipv6’s. Can the ipv6’s be used for email?

I was running into Undeliverable issues with Gmail saying that they didn’t like my ipv6 ip address for various reasons. I assumed that it was because it was missing the last 8 numbers 2602:xxxx:xxx:xxx:xxx0:: (double semicolon) so I set inet_protocols=ipv4 in main.cf and it accepted the ipv4.

Earlier, on a previous os install, I had setup a webserver proxy and the same thing happened when I tried to view Google’s homepage - said it detected strange behavior with the ipv6 and couldn’t continue - assuming the ipv6 was strange, I changed to ipv4 only and it accepted it.

Does my ipv6 look normal?

2602:xxxx:xxx:xxx:xxx0:: - double semicolon at end

It’s strange that any ISP would restrict STARTTLS to begin with. It’s forcing people to be less secure. Google would have a fit if they found that out.

Sure, no problem. Postfix supports IPv6.

2602 isn’t in the list of special IPv6 addresses so should be good. Also, you can connect to an IPv6 address (i.e., GMail) with telnet perfectly, so all works :wink:

The only requirement that I’m aware of is that your IPv6 address needs to have a valid PTR record (reverse DNS), i.e. nslookup [your IPv6 address] should resolve to your mail server’s hostname and vice versa. (The same goes for IPv4, AFAIK.)

Another option you might want to explore if the IPv6 option doesn’t work out: Apparently postfix allows you to set the TLS security level on a per-domain basis - I’m not all that familiar with postfix, so I’m not sure about the specifics on this. Here’s an article with some details. I’m not sure how this would interact with the proxy and lack of STARTTLS command though, perhaps postfix will refuse to send the messages or the proxy will break things.

I was about to suggest @seveneleven could try to connect to GMail through port 465 with TLS to bypass STARTTLS, but it seems GMail doesn’t offer that route (any more?). Yes yes, I know it’s deprecated, but it could have been a solution to this MitM stuff… But I’m afraid it’s not.

I think 465 and 587 are only used for MUA to MTA communication in practice (i.e. something like Thunderbird connecting to Google’s outgoing SMTP server at smtp.gmail.com).

MTA to MTA communication is done on port 25 exclusively, IIRC.

1 Like

Well, there was a time STARTTLS didn’t exist and using a dedicated port for TLS was the only way to encrypt your mail :slight_smile:

@pfg is correct that MTA-to-MTA traffic is always on port 25. @osiris is also right that port 465 predated STARTTLS. It’s not clear to me whether it was every intended for SMTPS on 465 to take over MTA-to-MTA traffic, or whether it was intended just for MUAs.

I also agree that this looks very much like your ISP is stripping STARTTLS. Some off-the-shelf network equipment does this. I wouldn’t be surprised if the default configurations only did it for IPv6. @seveneleven, you should contact your ISP and ask them to disable that feature. They may not even be aware that it’s on!

2 Likes

as @pfg says, google requires reverse DNS when using IPv6 (while it is only recommended for IPv6. It also almost-requires SPF and/or DKIM when using IPv6 – you can get by without it, but then it might mark most of your mails in Spam [as it did for me]). See https://support.google.com/mail/answer/81126 for details (“Additional guidelines for IPv6” section)

About gmail and postfix and ssl : I follow instruction here : Tutorial on ejabberd, postfix, dovecot and or nginx with letsencrypt

Then :

  1. Make my ssl for belar.example.net via certbot (i use webrot method, but think any method work too)
  2. update postfix (and dovecot) to use same certificate than my webserver
  3. restart postfix

Seems OK (for me)

Received: from belar.example.net (belar.example.net. [91.xx.xx.xx])
        by mx.google.com with ESMTPS id 1si721816wrl.82.2017.02.20.00.00.46
        for <shnoulle@gmail.com>
        (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
        Mon, 20 Feb 2017 00:00:46 -0800 (PST)

But right : i didn’t find the good way for IPv6

This happened before with some ISPs in 2014 (@jsha wrote about it at https://www.eff.org/deeplinks/2014/11/starttls-downgrade-attacks). It’s true that it’s not always immediately clear whether it was intentional on the ISP’s part; one incentive would be if the ISP wanted the ability to monitor, record, or censor outgoing e-mail, and was willing to block encrypted connections in order to do so.

You’re forgetting the most likely reason: on home networks, anything that calls out to port 25 is almost guaranteed to be a computer virus, spewing spam to the world. The ISP most likely wants to catch that before it causes problems for itself.

There is no good solution, and time machines don’t count. I personally use a VPN tunnel to work around problems like this.

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.