recently I did some experiments with "tlsserver" profile certificates. It mostly works.
Using such a certificate for TLS of a SMTP server works for all peers except of one.
A large German freemail provider cannot connect and I find following log entries in my logs:
warning: TLS library problem: error:0A00042F:SSL routines::tlsv1 alert insufficient security:../ssl/record/rec_layer_s3.c:1605:SSL alert number 71:
lost connection after STARTTLS from mout.gmx.net[212.227.15.15]
After recerting to "classic" profile incoming connections do work again. I did no other changes to cipher / TLS settings or the certificate. Changing back to the (previously issued) "tlsserver" certificate gives an error again. The behaviour seems to be consistent and reproducible.
I cannot tell which change is triggering this behaviour. It could be the missing common name, the missing client auth EKU or the missing subject key ID. (it is a EC certificate so there is no key encipherment KU anyway)
Are "tlsserver" certificates supposed to work in the SMTP context? Is it worth to report it to the postmaster of the sending provider?
Are SMTP TLS certificates "special" and do we need "classic" profile certificates for usage in mail servers?
Is that a newly acquired "classic" cert? Because LE recently changed the issuing intermediates and I just wonder if perhaps that mail client is wrongly validating the chain of the newly issued "tlsserver" cert.
Was "classic" cert also ECDSA? Is DANE involved?
Yes, I think so. You say it is the only one failing with "tlsserver" so they must be looking at something differently than others.
There are other better TLS SMTP experts on this forum. I just wanted to cover some basics.
Well, if by "supposed to" you mean "complies with all the current specs", then I think so. But there are a lot of mail servers out there that are running old or custom software, so I think a lot depends on who you're expecting to receive mail from. When I last tried using only an ECDSA cert and not also having an RSA one (which was a few years ago now, so things may have changed), there were quite a few senders that would either fall back to plaintext or just wouldn't deliver mail at all. (Many were financial institutions, interestingly enough, which I think were more likely to have worked to explicitly limit cipher suites in their configuration at some point and just didn't know about or care about ECDSA.) What I'm trying to say is that ECDSA is "supposed to work" in that it's a normal default kind of certificate, but not everyone is actually ready for it. And similarly, despite no-CN (and the other modern cert profile changes) being acceptable and even recommended for a long time now, there are still older systems out there that just haven't been updated accordingly for whatever reason.
I haven't tried out the "tlsserver" profile for my personal mail server yet, and I appreciate your testing and report that at least one system out there somewhere doesn't like it.
I mean, if you think you can get a hold of a human on the other end, I guess it wouldn't hurt to ask them to check things over. But most people, even most server administrators, have eyes that start to glaze over when you start talking about cipher suites and certificate field minutiae. I doubt many of them could tell you at all what the software they're running might be trying to do with the CN field. Keep in mind, most SMTP servers will still send plaintext just fine if it doesn't think the receiving server supports STARTTLS, as while DANE and MTA-STS support are growing there's still a long tail of servers out there that just haven't been updated in forever.
I am aware that there are a lot of "old" servers around. It is a private mail server and I am receiving mail mostly from known recipients and it does not hurt "bussiness" if some mail bounces thus it was worth an experiment.
I suppose the situation with ECDSA improved since you tried the last time - maybe I was lucky too but if you want / have to be compatible to ~100% of the servers you probably have to use the "really old stuff". I would not be surprised if some email servers bounce mails if you only support TLSv1.1 and higher.
E mail is a mess with a lot of different software around - the browser market has only a few big players and if one or most of them require new ciphers / protocols or make changes to certificates they accept you have to change. If Chrome rejects your certificate and displays an error page instead of the content you are simply out of bussiness.
I am no cryptographer but it is an interesting topic for me but I know that many peoply try to avoid to deal with encryption - it is very complex and is protocols / ciphers are rapidly changing. I will give it a try. If the person at the other end is an crypto enthusiast I might be lucky and he will try to fix this problem. Chances are high that he is struggling with more important daily bussiness and does not want to do a deep dive in X509.
At least now I know that it is not my "fault" - besides being foolish and daring to try out "the newest stuff". I always do my best to check my side before complaining that something does not work. Now I know that it is "according to the spec" but it is very bleeding edge.
I reverted to "classic" ECDSA for now but I learned something new therefore it was not wasted time for me.
Probably they cannot tell. The missing CN field is IMHO the most likely cause why the TLS handshake is failing. Nearly every certificate had an CN until now. But there is another one: client authentication EKU. It would make no sense to require client authentication EKU from a server but until now most certificates had both which is "dangerous" because some developers might require it by accident without knowing the difference.
If it is the missing client authentication EKU I want it get fixed before we are forced to use certificates without it. The transition is only some months away.
If the other server would just send without encryption I would not mind. The other end gives up and bounces the mail which is a more annoying failure mode.
MX mailservers are probably the one application where you best stick to the "classic" profile (and perhaps RSA) for the reasons you stated. At least that's what I did, with everything else (including Authenticated SMTP for clients and IMAP) switched over to the "tlsserver" profile (and ECC) without issues.
I doubt the client auth EKU is the issue, but if it is, their postmaster will soon become painfully aware of their problem.
It's more likely the omitted CN.
On TLS versions: TLS 1.2+ is more or less ubiquitious now even in SMTP land, as more and more MX mailservers are disabling TLS 1.0 and 1.1.
