Windows Live Mail revocation warning


#13

With reference to the following (which has been closed so I can’t add to it)…

We have received yet another complaint today about our emails not working in Windows Live Mail. Is there really nothing that can be done to deal with the underlying problem? Despite lack of support from MS, Windows Live Mail, like Outlook Express before it, isn’t going to disappear any time soon, and it’s much more recent an app than, say, Windows XP where LE didn’t work either when first released.

As far as users are concerned, it is

  • intensely annoying as they get loads of modal pop-ups, one per image, which each have to be dismissed individually
  • it appears to them only our emails are a problem, which isn’t true in general but is the perception
  • they have no idea that it is the images that are the problem - as far as they are concerned, our emails are broken, even though the reality is it is their email app that is out-of-date.

I don’t know much about CRL, but is there maybe some way to make a bridge back from OCSP data to CRL, enough to make the obsolete Live mail program work even if revocation isn’t possible there, or some way we might be able to patch Live Mail. Might an approach to Microsoft be helpful?


#14

I don’t think Let’s Encrypt is going to generate a CRL, just to make sure a single discontinued mail client works? They’ve made a choice to issue only signed OCSP statuses and don’t use CRL’s. So I’m guessing that was a deliberate choice, not something they decided lightly.

The mail client expects these CRL data inside the certificate, which is signed by Let’s Encrypt. Furthermore, it would be insane to generate a CRL without using it for revocation checking! That’s the whole point to a CRL! You can’t expect Let’s Encrypt to maintain a seperate bunch of certificates with a “fake” CRL. That’s probably against the CA/B Forum rules too…

I guess that’s something you should ask MS indeed, but don’t get your hopes up…


#15

(Reopened the original topic and merged the previous posts.)

Compared to OCSP, CRL is quite the bandwidth hog. It’s essentially a file containing a list of all revoked certificates from a particular CA (though some partitioning is possible to improve performance), and every user needs to fetch the whole file. Cloudflare had a blog post on this a few years ago during the Heartbleed fiasco (during which a lot of certificates were revoked). They estimated the bandwidth cost for the revocations of just one CA at about $400,000.

Let’s Encrypt probably has (at least) an order of magnitude more active certificates, so we might be talking about $4M for the next Heartbleed. That’s more than what it costs to run Let’s Encrypt for a year otherwise (about $3M). Even without a similar event, the cost for CRL would be significant - about 25,000 certificates are revoked each month (based on stats from December).

Technical reasons aside, it’s probably simply too costly.


#16

I understand the problems, which is why I was looking for some lateral thinking, a work-round of some kind, not necessarily a complete solution. It may be a “single discontinued email client”, but it is apparently, and to my surprise, still widely used, and it isn’t very old. LE did eventually work to find a solution for XP, which is much older and also obsolete.

Obviously an approach to MS from me is going to be completely ignored - that’s why I was asking, as it would need a much higher level approach than from little me. But even if they would consider making the warning non-modal, or turn off the message altogether as it is just wrong now, it would help.

The biggest problem is that as far as the recipient is concerned, it is our fault that we are sending them extremely annoying email faults! Telling them to change their email client, with all that entails to unknowledgeable people who have long since forgotten their email passwords just makes it worse.


#17

But in that case any solution on the client side would hardly be effective? If the clients are that unknowledgeable, something like a “patch” of “user setting” would possibly be difficult to achieve.


#18

XP compatibility was relatively easy to achieve, requiring only a small change to the intermediate certificate and a key signing ceremony. From what I can tell, rolling out CRL would require significantly more effort (both in terms of development effort and financially).

There doesn’t seem to be an easy fix here on Let’s Encrypt’s end, so I would personally look for workarounds elsewhere, like referencing images via a domain behind a certificate from a CA with CRL support. Perhaps proxying through Cloudflare or Amazon CloudFront (both offer free certificates; CloudFront charges for traffic) would work for you?


#19

At least it’s possible to lead someone through that though, and people are familiar now with updates to software they already have.

This thread is still all about pointing out reasons why it is a hard problem to solve, not what could be done to mitigate it!


#20

I don’t get it. You were told all your options, still you insist on Lets Encrypt to move.

If you need CRL information in your certificate, just buy a certificate with this (outdated) feature. The competition is searching for unique selling points, and backwards compatibility is one of them, so go and pay them the money.

If your clients need to connect to your site using the outdated tool, they can disable the checking, which is not a good idea, security wise, but using the old tool isn’t either.

Most users of LetsEncrypt, apparently all except you, don’t care about this feature. So they either went the way of ignoring their clients’ needs, or their clients just don’t use that kind of outdated tool.

Harsh words spoken, I hope you appreciate the summary of your options, nevertheless.


#21

Probably because there is not anything what could be done from Let’s Encrypts end ánd probably nothing to be done about it on the client end… In any case, not anything we can think of… Otherwise you would have heard it already :slight_smile:


#22

If your clients need to connect to your site using the outdated tool, they can disable the checking, which is not a good idea, security wise, but using the old tool isn’t either.

They aren’t aware they are “connecting to our site”. They’re just pictures in an email, as far as they are concerned, and everyone else’s pictures work.

No, of course they won’t disable checking, they’ll just delete the emails unread and report them as spam.


#23

You asked for a solution. That is a solution. If thát doesn’t work for you, how would any other client-side workaround work for you?

As said, there is most certainly absolutely nothing Let’s Encrypt can do about this from their end. Or at least nothing feasible.


#24

Most users of LetsEncrypt, apparently all except you, don’t care about this feature.

Well, the reason I’m raising it is because it has relevance for more than just me. It took us nearly a year before anyone even bothered telling us that this was a problem - most of the time, as I said, people just move on elsewhere. I suspect many people using images in email have the same problem, just don’t know it because none of their recipients has pointed it out to them. Now I know about it, I can indeed tell people that there is a setting they can change, though it seems to be “turn off all images from everyone” from what I can see so far, which won’t go down well either. I doubt most people using MS Live Mail know that it is obsolete.

…absolutely nothing Let’s Encrypt can do about this…

I understand the problem is not directly solvable, but for example, an approach to Microsoft is something LE could do that wouldn’t have an effect if it came from me. I can certainly try, but they will just say “unsupported, unsupported”.

The XP issue was also effectively deemed “impossible to solve” when it was first raised! I’m not looking for miracles, but this is a genuine problem. It may be small in percentage terms, but there must still be millions of active Live mail users out there, for whom images work in emails except where the sender uses a LE certificate.


#25

Microsoft is pretty stubborn when it comes to fixing things in products that aren’t supported anymore. Probably rightfully so - Pandora’s box and all that. I doubt this will be successful no matter who asks, especially since - from Microsoft’s perspective - workarounds (like using a CA with CRL support) exist.

I don’t think this is accurate. From what I recall, this is the first statement regarding XP support. Again, there’s no similar solution here, so there’s really nothing Let’s Encrypt can do, other than document the issue (which has happened).


#26

How do CRL and OCSP interact? If they are both indicated are they both checked, or does one take precedence? If OCSP is overwhelmingly used, is bandwidth a big deal, as it would only then be old clients that would be asking? Why don’t all CA’s have this issue - or do they, but they are large enough to cope and can fund it because people pay for certificates? What happens in browsers if a CRL doesn’t respond but OCSP does? Might that offer a route by which a CRL responder only works for the low-volume cases that can’t be solved by OCSP, and therefore reduces the overhead to very small?


#27

Hi @fas,

Thanks for checking in again on this issue. It probably is possible to fix this issue by implementing CRL. It would require a substantive investment in engineering time even to reproduce the issue and verify that providing CRL fixes. It would require significantly more engineering and ops time to design and build a CRL system for our end-entity certificates, and maintain it over the long term.

While I empathize with your issue, and would like to make Let’s Encrypt as broadly compatible as possible, I do have to balance costs and benefits. In this case, the cost of implementing CRL is very high, and the benefit-- supporting an older mail client with relatively low usage-- is not large enough to justify the cost. If we had evidence showing a large number of people reporting that Windows Live Mail support was a high priority, that balance might shift, but for now we can’t justify the work.

Is buying a certificate from another CA an option for you?


#28

Oh, and I forgot one more option you have: Just serve your images over http instead of https. That used to be the way by the time Outlook Express and Windows Live Mail were great, anyway.

BTW, and if the admins here don’t mind to link to commercial services I’m not affiliated with, I bought a really cheap ($14 for 3 years) Comodo PositiveSSL certificate from gogetssl.com two years ago which contains CRL entries. I’m pretty sure that they still offer this feature. Perhaps their free certificates have it, too, but I cannot tell.

OCSP, especially OCSP Stapling, typically takes precedence because of its better performance. Still, fetching OCSP responses leads to time-outs frequently, so most browsers just don’t care for that by default, even Internet Explorer.

Yes, only old clients are asking for CRL, and they look it up typically only if it’s enabled, or only for EV certificates. Most browsers and other clients have dropped support for it completely, years ago.

But remember, OCSP Stapling is a feature that a server admin can use to improve the availability of the OCSP result to the client as well as overall performance.

Other CAs don’t have the problem because they still support CRLs. Simply as that.

There is nothing that can’t be solved by OCSP which can be solved by CRL. Instead, OCSP offers more features and better performance than CRL altogether, that’s why it is so easy for most of us to switch there without looking back.


#29

They don’t, they are independend methods of revocation checking, both with their advantages and disadvantages.

That depends on the client.

Not every client uses OCSP, as OCSP without OCSP stapeling has privacy drawbacks/concerns.[quote=“fas, post:26, topic:26310”]
Why don’t all CA’s have this issue (…)
[/quote]

I expect this is indeed because of financial capabilities and/or less number of certificates. Let’s Encrypt is growing fast, because it’s free.


#30

Comodo’s free certificates are only valid for 90 days (same as LE), and last I checked were only for “trial purposes” and only once per domain. That said, many resellers, including NameCheap (my usual source for paid options like wildcards) have very affordable basic DV certs at a very low cost if you need “legacy” stuff like a CRL for old software.

Likely more because the major commercial entities were around before OCSP when CRL was the only mechanism for checking certificate revocation. They also make very good money from selling certificates, which rebalances the cost equation.


#31

Is buying a certificate from another CA an option for you?

Well, yes, clearly that’s an option. Rather defeats the point of LetsEncrypt though!

Another would be to host the images somewhere off our normal website (like flickr perhaps) for this purpose.

Just serve your images over http instead of https.

That is less satisfactory, not just for the obvious reasons, but that Outlook.com (why does it always seem to be Microsoft!) doesn’t handle this well - you get mixed content. (GMail handles this much better by proxying them).

I think maybe the best answer short of dealing with the problem directly is to use http image links which redirect to https except for the known problem clients. We still do this for IE8 on XP where SNI is a problem, so that mechanism is in place. The main issue then is for the people preparing the mailshot to understand the issue and not blindly put in URLs to the images straight from the website (i.e. Wordpress media library).


#32

Web clients that report mixed content errors will still consider this a mixed content error (because the initial HTTP fetch could have been snooped on or tampered with). So this is not the ideal plan, in my opinion.

This seems like a nice, simple approach that should work.