Windows update KB5018410 breaks SMTP with some certs

Since few days, users with Windows update KB5018410 are unable to use SMTP TLS (just google "KB5018410 smtp"). IMAP with the same cert works.

In my case it affects only one server with hone LE certificate. Both servers are completely the same (postfix/debian/openssl) versions and the same configuration.

On the affected server the connection fails during SSL/TLS setup for these clients. Renewing the LE certificate did not help. However trying to use LE certificate from the other server worked and these clients were able to connect (with warnings about wrong domain in cert of course).

Can you help me to find what is different in these certs and why one works and the other not? I tried using openssl and various tools to compare them and did not find anything. I compared the certificate chain and both present LE R3 (serial 91:...5A) and ISRG Root X1 (serial 40:...:B7). If needed to check connection, I can provide running postfix on some ports with both certs.

crt.sh | 7341323384 (original cert not working for some clients)

crt.sh | 7576728815 (cert from similar server, working)

Postfix log for failed connection:

Oct 14 14:40:47 bill3 postfix/smtpd[12167]: static-dsl-xx-197-101.telecom.sk[87.197.x.x]: TLS cipher list "aNULL:-aNULL:HIGH:MEDIUM:+RC4:@STRENGTH"
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:before SSL initialization
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:before SSL initialization
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:SSLv3/TLS read client hello
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:SSLv3/TLS write server hello
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:SSLv3/TLS write certificate
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:SSLv3/TLS write key exchange
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:SSLv3/TLS write server done
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept:error in SSLv3/TLS write server done
Oct 14 14:40:47 bill3 postfix/smtpd[12167]: SSL_accept error from static-dsl-xx-197-101.telecom.sk[87.197.x.x]: lost connection

Hi @marki555, and welcome to the LE community forum :slight_smile:

This is confusing (and needs more explanation):

More must have changed.
Those two certs are essentially the same [except for the FQDN they cover].

I'm shocked at how many ciphers your SMTP server supports:

mail.hostio.sk

ECDHE-RSA-AES256-GCM-SHA384
DHE-RSA-AES256-GCM-SHA384
ECDHE-RSA-CHACHA20-POLY1305
DHE-RSA-CHACHA20-POLY1305
DHE-RSA-AES256-CCM8
DHE-RSA-AES256-CCM
ECDHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-GCM-SHA256
DHE-RSA-AES128-CCM8
DHE-RSA-AES128-CCM
ECDHE-RSA-AES256-SHA384
DHE-RSA-AES256-SHA256
ECDHE-RSA-CAMELLIA256-SHA384
DHE-RSA-CAMELLIA256-SHA256
ECDHE-RSA-AES128-SHA256
DHE-RSA-AES128-SHA256
ECDHE-RSA-CAMELLIA128-SHA256
DHE-RSA-CAMELLIA128-SHA256
ECDHE-RSA-AES256-SHA
DHE-RSA-AES256-SHA
DHE-RSA-CAMELLIA256-SHA
ECDHE-RSA-AES128-SHA
DHE-RSA-AES128-SHA
DHE-RSA-SEED-SHA
DHE-RSA-CAMELLIA128-SHA
AES256-GCM-SHA384
AES256-CCM8
AES256-CCM
AES128-GCM-SHA256
AES128-CCM8
AES128-CCM
AES256-SHA256
CAMELLIA256-SHA256
AES128-SHA256
CAMELLIA128-SHA256
AES256-SHA
CAMELLIA256-SHA
AES128-SHA
SEED-SHA
CAMELLIA128-SHA

To me this sounds like a client problem, not a certificate problem.

2 Likes

Yes, I just pulled the LE cert from other server and these clients were able to connect. These are existing clients, existing LE cert and everything worked until 2 days ago. So it is a client problem, but everything worked before until that Windows update was installed.
I am also confused why only 1 cert is affected. Right now on port 587 is running postfix with the original (problematic) cert and on port 588 with the cert from other server, if you want to test.
Regarding the ciphers, this is default configuration in postfix 3.1.15 in debian 9 with openssl 1.1.0l, I did not configure them and should be the same on both servers.

This issue is very new (3 days) and I have found several other people on other forums having the same issue, also Plesk affected, but no details yet about what exactly causes it.

Would maybe tcpdump of the failing connection help? Or somehow decipher the tcpdump with the server private key?

1 Like

Possibly.

You shouldn't have to get that deep to "see" the problem.

3 Likes

My bet is on "multiple things having gone wrong".
Like:

  • that KB breaking something within the client affecting "some ciphers"
  • the server supported ciphers is bad and the order is helping to cause this problem
3 Likes

You should look for guidance on how to secure it better than using "the default".

3 Likes

This cipher suite configuration looks very weird, especially the enabled-then-disabled aNULL (which is interpreted as "enable aNULL" by the way) doesn't look like it is sane.

The aNULL is indeed causing OpenSSL to enable anonymous ciphers without authentication. This is quite dangerous, as it provides no security against active attackers. You are likely being saved by clients refusing to negotiate them though, but it's a very bad configuration in any case.

The config also tries to enable RC4 ciphers, which is luckily not even supported by your OpenSSL version - you don't want to use RC4 in any case.

 Testing cipher categories

 NULL ciphers (no encryption)                      not offered (OK)
 Anonymous NULL Ciphers (no authentication)        offered (NOT ok)
 Export ciphers (w/o ADH+NULL)                     not offered (OK)
 LOW: 64 Bit + DES, RC[2,4], MD5 (w/o export)      not offered (OK)
 Triple DES Ciphers / IDEA                         not offered
 Obsoleted CBC ciphers (AES, ARIA etc.)            offered
 Strong encryption (AEAD ciphers) with no FS       offered (OK)
 Forward Secrecy strong encryption (AEAD ciphers)  offered (OK)

I don't think that this suboptimal cipher suite configuration is the cause of your issues though (especially since it's the same on both ports).

The error messages are sadly not really helpful. Outlook 0x800CCC1A should be:

Your server does not support the connection encryption type you have specified. Try changing the encryption method. Contact your mail server administrator or Internet service provider (ISP) for additional assistance.

I would reverify the the settings in Outlook are correct (maybe it has gotten more sensitive to incorrect settings?). Encryption type should be STARTTLS.

The postfix log just indicates that some party disconnected (without a proper TLS error message) while finishing the handshake. This behaviour is most commonly seen with unhappy firewalls/TLS inspection (client and/or server-side).

4 Likes

Postfix specifically prefers anonymous ciphers when configured to use "opportunistic TLS" (which is typical for Internet-facing mail servers), since it figures that if it would have been fine to send the message plaintext it would also be fine to use TLS but without any authentication.

But yes, I agree that one should be specific about what ciphers one wants to use, and I'm personally not a big fan of what Postfix uses as a default. You could look at Mozilla's config generator for some ideas, but there are a lot of old, badly configured mail servers out there still that you might want your server to still be able to talk with.

5 Likes

It looks like the problem with the update isn't related to cipher suites, but to empty session ids provided by Postfix.

https://marc.info/?l=postfix-users&m=166585838304041&w=2

9 Likes

TLS 1.2 session tickets should be disabled anway (We need to talk about Session Tickets) [unless you're a large CDN with special needs and know what you're doing], so nothing lost. Session ID resumption should be used instead.

I personally disagree with this decision, but if that is the Postfix default well so be it. (I would discourage this setting because this completly removes the option of verifying certificates, so it can break DANE/MTA-STS. If clients want to use opportunistic TLS they can simply skip certificate verification, but aNULL doesn't even send a certificate in the first place. TLS 1.3 removed this handshake variant for a reason)

4 Likes

If Postfix is configured to use DANE, then if it finds the TLSA records it uses "mandatory" TLS so it won't negotiate the anonymous ciphers and does in fact verify the certificate.

Postfix doesn't have any outbound MTA-STS support built in (though I think there may be some systems that try to add it in somehow), but any host that it's configured to require TLS for similarly won't use the anonymous ciphers.

4 Likes

I guess that this is default because of buggy email servers which won't try without TLS if TLS negotiation fails for any reason. And as a SMTP itself, you want your customers to receive all legitimate emails.

I'm not sure how much of this is from the past and how much it still applies to SMTP servers currently in the wild, but I guess nobody wants to try&guess on production servers...
But what could be done is separate config for the submission port where only authentificated clients (customers) connect and enforce modern TLS settings here (similar as for web servers).

1 Like

That's for the client side, and that's exactly the problem I have with this: It relies on the TLS client to do the right thing. The cipher suite configuration shown above was for a TLS server socket. A server should be conservative and always avoid the usage of ciphers that are highly problematic - such as aNULL. The server configuration above a) had aNULL enabled b) had server cipher preference off. This configuration is basically "whatever the client likes", so it relies entirely on the client to not negotiate aNULL if the server in question is DANE/MTA-STS enabled.

While Postfix (as a client) may be smart enough to do this correctly, we have no idea if that's the case for ALL SMTP implementations out there. This is essentially saying "I don't know who you are, but here's a weapon, hopefully you won't use it unless you can handle it". While being completly redundant at the same time, because there's no need to have that weapon in the first place.

TLS negotiations fail all the time (in SMTP), so if the sending MTA won't retry all hope is already lost :stuck_out_tongue_winking_eye:. I know that for SMTP, you want to have best compatibility because "any form of TLS is better than no TLS", which is the usual fallback for MTA-to-MTA transfer. So many SMTP servers offer a huge variety of cipher suites, which you would never see in i.e. HTTPS. Still, while I can see this as an argument to have stuff like CBC or even Aria enabled on SMTP, I don't find aNULL to be worth their inherent misuse risks. It's going to be used in the wrong place at the wrong time eventually.

(Also, one point I completly forgot to mention: The TLS configuration we were originally talking about was SMTP Submission, which should always verify certificates. Opportunistic TLS on a submission port is a complete no-go for me. For internet-wide SMTP on port 25 you have to deal with buggy, outdated, and misconfigured servers, but for submission you are going to talk to much more recent devices that you can actually influence to some extent. Making this port vulnerable to MITM attacks is just outright dangerous)

4 Likes

Oh, of course. Now I feel a little silly for bringing it up. And the whole problem in this thread that I've managed to sidetrack is that a client seems to not be doing the Right Thing.

Well, I can tell you that when I was trying an ECDSA-only cert on my mail server, there were systems that failed to negotiate and would just keep retrying and failing each time.

Another great point, and clearly I've sidetracked things even more than I realized.

4 Likes

A similar thread on postfix-users mailing list: 'Outlook TLS errors after Microsoft Windows Update' thread - MARC

1 Like

So would clearing cached session tickets (in postfix) fix this issue or do they have to be disabled completely?

1 Like

Setting tls_ssl_options=NO_TICKET for the submission port seems to help. You can add it as -o tls_ssl_options=NO_TICKET into /etc/postfix/master.cf and reload postfix. Or set it globally in main.cf, but then it will be used also for all server-to-server connections on port 25.

3 Likes

Microsoft fix : KB5020435 out of band

https://support.microsoft.com/en-gb/topic/october-17-2022-kb5020435-os-builds-19042-2132-19043-2132-and-19044-2132-out-of-band-243f34de-2f44-4015-a224-1b68a4132ca5

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.