Mail server sending to postfix refusing TLS connection with "certificate expired", but it's not

[l3u]

Hi all

Since April 30, I'm seeing errors like that in my mail log:

May  1 02:27:27 afaron postfix/smtpd[2644268]: connect from r137.info.hofer.at[66.117.17.137]
May  1 02:27:27 afaron postfix/smtpd[2644268]: SSL_accept error from r137.info.hofer.at[66.117.17.137]: -1
May  1 02:27:27 afaron postfix/smtpd[2644268]: warning: TLS library problem: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired:../ssl/record/rec_layer_s3.c:1543:SSL alert number 45:
May  1 02:27:27 afaron postfix/smtpd[2644268]: lost connection after STARTTLS from r137.info.hofer.at[66.117.17.137]
May  1 02:27:27 afaron postfix/smtpd[2644268]: disconnect from r137.info.hofer.at[66.117.17.137] ehlo=1 starttls=0/1 commands=1/2

As far as I can grasp it, r137.info.hofer.at[66.117.17.137] refuses to send mail to my server, because it claims my SSL certificate would be expired.

I double-checked if the latest issed letsencrypt certificate one is actually used by postfix, and it is (via the live symlink – the server has been running for almost a year now without any problems). It's not expired. I even tried to force-update the cert, but the errors re-appeared. When I run openssl s_client -starttls smtp -showcerts -connect mail.l3u.de:25 -servername mail.l3u.de , I get a valid TLS session ticket.

Until now, r137.info.hofer.at[66.117.17.137] is the only mail server complaining. Im tried send mail from and to gmx.de, web.de, t-online.de, gmail.com, yahoo.com and outlook.de. All without a problem, both sending and receiving.

How can I track this down? Can this be some local problem due to some outdated cert in the chain of trust for my sertificate on my server? And how can I find it? Or is this a remote problem?

Thanks for all help!

[9peppe]

@l3u

Or is this a remote problem?

I am tempted to say yes.

This problem usually happens when the client is runnning OpenSSL 1.0.1 or 1.0.2.

You can mitigate it (if the actual problem is that one, and I have no evidence, just strong suspicions) by serving the short chain instead of the long one. For a mailserver shouldn't change much.

[l3u]

After having found and read through

Help, let'sencrypt does not work for Mail

I…

I force-renewed my server's certificate with --preferred-chain "ISRG Root X1" and triggered getting a mail from said server (it delivers Aldi Süd's Newsletter )

Apparently, the problem is gone now: The very same server now talked to my mail server. No error.

Did you mean that with "serving the short chain instead of the long one"? Sorry, I'm not really deep into SSL, until now, everything "just worked" …

[9peppe]

Yes, I did mean that

[l3u]

As far as I grasp it, this would only affect some legacy devices that haven't got updates for years, won't it? So I can be pretty sure my mail server won't be affected in a negative way, will it?

[9peppe]

Not more than before.

The issue you could see is with android < 7.1.1 clients, if you use the same certificate for port 25 and for the "client" ports (110, 143, 465, 587, 993, 995).

If there are mailservers around that haven't been updated since 2015, I am not sure I want to get mail from them. And they'll probably react like this one, if the certificate is expired and they do not ignore the expiration.

[l3u]

Well, then I'm, pretty sure I'm fine

Thanks for the support!

[PMc]

@l3u

As far as I grasp it, this would only affect some legacy devices that haven't got updates for years, won't it?

Yeah, that's the theory.

In practice, I found this a few days ago, when I started to thoroughly look into that technology and built a test environment.

The problem with the "expired cert" arises only and specifcally when both sides use Let'sEncrypt certs.
And it arises with very new software.

So I can be pretty sure my mail server won't be affected in a negative way, will it?

What should happen to the mailserver? What is worrying You?

Technically we call this a false positive (or false negative, as you want it): the certificate is flagged as untrustworthy although it is trustworthy. So what can happen is, communication does fail, and you don't get your mail.

And then you start reading the sourcecode evilgrin

I did, I read the code of sendmail (it's beautiful), and the bug was not there.
Now I assume it is in openssl. Interesting that You say it suddenly appeared a few days ago. Then it could be a regression.
Could You figure out what exactly has been changed/upgraded at that time?

[9peppe]

@PMc

both sides

That makes really little sense.

Are you sure the option we were talking about yesterday refers to the certificate chain and not to a root store?

[Rip]

No MX records that I can find.

[PMc]

@9peppe

@PMc

both sides

That makes really little sense.

It's just the fact.

Are you sure the option we were talking about yesterday refers to the certificate chain and not to a root store?

I don't remember option. I do remember what is in the files:

Alice uses a Let'sEncrypt cert. Bob has the expired DST X3 caroot in his root store, and the DST intermediate in his(!) chain. Kabumm!

It doesn't matter what certificate Bob actually uses.

I assume this is a bug where the local chain-data for authentication spills over into the chain-data for verification, inside of openssl.
To me this makes a lot of sense, because it is quite typical for bugs that were fixed, but forgot a corner-case.

The option? Maybe you mean the --preferred-chain? That nice for automation, but technically doesn't matter much, because I can as well take an editor and delete the offending cross cert out of the chain-file. The effect is the same: as soon as Bob does not have the DST internediate in his chain-file, the error is gone.

[9peppe]

I'm talking about this option:

@PMc

the key is the key, the cert is the cert, and the cacert.pem is the chain, i.e. the collection of intermediate certificates that are needed for the adversary to get to one of their known root ca certs, which obviousely must be sent to the adversary during handshake.

But here the doc talks about something different: it appears to say that the cacert.pem contains a "list of acceptable CAs".

"List of acceptable CAs" can mean "a root store" instead of "the certificate chain."

[Osiris]

@Rip

No MX records that I can find.

The hostname of the mailserver itself shouldn't have a MX RR, the MX RR is for the apex domain and should point to the mailservers hostname.

[l3u]

@PMc

Could You figure out what exactly has been changed/upgraded at that time?

That's the interesting part: Nothing. Not on my mail server at least …

[l3u]

@Osiris

The hostname of the mailserver itself shouldn't have a MX RR, the MX RR is for the apex domain and should point to the mailservers hostname.

Exactly. E.g. l3u.de has an MX record pointing to mail.l3u.de.

[Osiris]

Unless you'd like to receive emails for foobarbaz@mail.l3u.de

[9peppe]

That's when the backup mechanism triggers. If no MX is found, the fqdn itself is assumed as MX and MTAs will try the A record.

[Osiris]

Sure, but it's better, I think, to do everything as best as possible.

But a MX record for mail.l3u.de is probably unnecessary.

[9peppe]

Yes, but... I do have mx, srv, spf, dmarc, and tlsa records for my MX. I use its own name for "technical" mailboxes, like dmarc and mta-sts reports, postmaster, etc.

[9peppe]

Y'all know what the funny thing is?

My mailserver isn't using DST Root X3... For an entirely different reason: it's using the "experimental" E1-X2-X1 chain instead of R3-X1(-DST).