Concerns about CloudFlare's use of Let's Encrypt

Have a look at this: https://news.ycombinator.com/item?id=16497362

In case they delete it, I will quote the comment:
"We use several CAs to issue—Comodo, DigiCert, GlobalSign—and will be adding Let’s Encrypt once they support i) SHA-2/ECDSA signatures and ii) wildcards.

Having multiple issuers is important for us as each CA, at some point in time, has operational issues. Additionally, as you’ve seen with Symantec, browsers take action to distrust certain issuers/roots.

When either of these scenarios happens, our customers don’t care if it’s the third-party that’s down—they expect fast and reliable issuance from us (Cloudflare)."

Cloudflare uses wildcard certificates to provide SSL encryption to many customers, all of whom use completely different origin servers.

One of the greatest things about the Let’s Encrypt project is the dedication to the real goals of SSL encryption including security and Trust. Cloudflare discards trust and contributes to the mass erosion of security on the web. They should not be allowed to issue Let’s Encrypt certificates unless their model changes.

1 Like

Why do you consider this to be abuse?

I, too, dislike Cloudflare for muddying the waters when it comes to SSL/TLS. You see a site sitting behind Cloudflare proxy and you have absolutely no idea whether the origin is configured in a trustworthy way or not.

To be fair to them, they didn’t invent them problem, and they are far from the only major CDN to enable these murky modes of operation. It’s always been possible to terminate SSL in questionable ways that are not transparent to the visitor. Perhaps more an education/UX problem than anything else.

Somehow I do not think that Let’s Encrypt are surprised by this news, as they no doubt would have to work with and provide a rate limit exception to Cloudflare.

What I find interesting is to theorize about how sketchy something would need to be that Let’s Encrypt would terminate under the subscriber agreement. I had considered spinning up a variant of acme-dns that would work in a totally unsafe way (in that I would have all the power) but made it really easy for people to get instant, automatically renewing certificates just with the single step of pointing an _acme-challenge CNAME record. Would this be objectionable under the subscriber agreement? Would it be objectionable ethically if I made an attempt to educate the service’s users about the risks? You can already find sites like sslforfree that have users engage in really risky behavior without any disclaimer (they even have a misleading disclaimer about not leaking private keys, when in fact they control ACME account with re-usable authorizations to mint fresh certs), but I’ve never seen them called out in any serious way. How to even think about this problem?

1 Like

During the TLS-SNI-01 discussions, Let’s Encrypt floated the idea of blacklisting certain hosting providers from TLS-SNI-01 issuance… so I think they are willing to take whatever steps are necessary to ensure the ongoing trust value of the Let’s Encrypt certificate.

Which matters, I think. That trust matters a lot more than whatever short-term riches are to be made by some random internet companies who want to half-ass the process because their customers won’t know the difference.

That was for specific technical reasons, because the design of TLS-SNI and some hosting providers’ systems enabled unexpected people to succeed at domain validation even though no one would want them to.

No one is asserting such issues with Cloudflare or ACME DNS or HTTP validation.

As far as I know, the ISRG has not taken a moral position against reverse proxy CDNs.

Please keep in mind that the Let’s Encrypt community forum is not here for FUD or bashing CDN companies.

CloudFlare’s impact on user privacy is very complicated, and depends a lot on the particular site and threat model. It might well be that some users who understand that HTTPS is important are misled when sites use CloudFlare because of the lack of HTTPS to many origin servers (or because the users didn’t expect to have a CDN have access to the plaintext of their communications). Unfortunately, there are many things about sites’ back-end security that aren’t very evident to their visitors. :frowning:

I have contacts at CloudFlare and will gladly encourage them to take more steps to encourage users to provide a secure origin connection. But as @mnordhoff says, Let’s Encrypt has never said that reverse proxy CDN services are inherently improper. These services increase users’ security against many threats and I can’t imagine that we would ever say that they’re not welcome to use Let’s Encrypt certificates.

Like both @simbalion and @_az, I’m concerned that we don’t have a stronger incentive for site operators to provide secure origins when they choose to use CDNs. I think it’s clear just from the conversations on this forum that Let’s Encrypt has been helping a huge amount overall in this regard, because we have repeated questions from people who are already using CloudFlare or a similar CDN (and in some cases have been using it for years) and who have never before had a certificate on their origin servers, and who are now trying to get one from Let’s Encrypt because we’ve removed the financial barrier. So I think our net effect on this dynamic has been enormously positive, and I think everyone working on Let’s Encrypt is going to conclude that we won’t stop helping to increase encryption in one place just because we won’t always increase it in another place.

But I would be thrilled to hear more specific recommendations for CDNs or for anyone else in the picture, and, as I said above, I’d be happy to take the issue up with the CDNs. (I don’t think that the CDNs are likely to want to adopt a requirement that they have to use the same protocol schema that was used to contact the origin server, at least unless we can get many more hosting environments to offer HTTPS by default all the time without any user action—which is certainly something that we’re continuing to pursue.)

I agree that it will be sad if the UI changes in browsers about insecure origins, for example, mainly lead people to simply put their HTTP sites behind a CDN in order to make the UI warnings go away faster. :lock: Right now, I don’t think we know how frequently something like that happens.

4 Likes
That was for specific technical reasons

uh, no. The idea was floated because special corporate interests do not take priority over the security and “trust” of the entire Let’s Encrypt project. It was clear that if certain hosting providers were inconvenienced the burden would be on them to fix their approach to certificate issuance. And that is how it should be. The same standard can be applied to CDNs.

Most organizations have no problem making adjustments to their operations to adopt Let’s Encrypt in a responsible way. Exceptions should not be made for any corporation.

No one is asserting such issues with Cloudflare or ACME DNS or HTTP validation.

I am. I’m saying that Cloudflare’s approach to SSL makes SSL less trust-worthy. Sure traffic is encrypted, but they decrypt it before delivery and who knows if the origin you expected is the one it’s really coming from.

I think this forum exists to discuss topics exactly like this one.

Let’s Encrypt is the best thing to happen on the internet in years, and you can rest assured many of the largest corporations in the hosting marketplace would love to see LE fail. They’ve lost millions of dollars in laborless profit to this project, and those who haven’t adopted Let’s Encrypt stand to lose many millions more. They would love to tell people Let’s Encrypt certificates can’t be trusted.

I also personally don’t mind your raising the topic, but I think your proposed approach is kind of far outside of the way that Let’s Encrypt currently thinks about this. As I said in my reply, we’re getting more encryption overall, including more encryption of origin servers behind CDNs. While you’ve raised a legitimate concern, I don’t think the solution you’ve apparently proposed of essentially requiring (as a condition of Let’s Encrypt’s terms of service) the CDNs to use the same protocol that was used with the origin server holds appeal for us. What we’re doing now appears to be increasing the protection of a huge amount of users communications under many threat models, even in the CDN case.

I’d suggest pursuing one of the following approaches:

  • Bring some CDNs into the conversation. (I’m sure they’d also like to see HTTPS connections to origin servers, although they might disagree strongly with you about when and how this should be mandatory.)
  • Take this up with browsers, who are empowered to act on behalf of users in various ways (including security indications in the UI and changing requirements for certificate issuance). For example, the browsers could conceivably show a different security indication for a connection to a CDN when the encryption status of the origin connection was unknown or insecure, including asking the CDNs to expose this information somehow. Or the browsers could change the requirements for certificate issuance to impose the rule that you propose. Both of these things are difficult (and I would advocate against the second), but the browsers have the institutional role of protecting user security and keeping users informed about security measures, so they’re empowered to act on your concerns in some ways if they agree with them strongly enough.
  • Gather some more data about the state of origin security and the effects of Let’s Encrypt on it. (Also, is it different from CDN to CDN? Do different CDNs currently have different policies or documentation that’s having a differential impact? Are there any CDNs that do currently require HTTPS-to-origin for all users, and how hard has it been for them to obtain compliance with that policy so far?)
  • Suggest another course of action that CDNs and/or Let’s Encrypt could take that’s a less radical break with current practice than your original suggestion.

Again, I don’t want to dismiss your concerns, because I think they’re useful and you’ve raised a real issue; I just don’t think Let’s Encrypt is going to adopt your original policy suggestion, so I want to encourage you to pursue some other avenues. (I can also discuss the matter with some other colleagues who have more direct responsibility, but I also don’t think you can get a different result by just restating your concerns.)

1 Like

That’s also something to be vigilant about, but in this case you’ve also quoted a statement that

are all also issuing certificates to CloudFlare, and indeed I don’t know of any CA that’s specifically adopted the rule that you’ve suggested. So, this doesn’t seem to be a matter on which Let’s Encrypt stands alone.

The problem as I see it is that Wildcard certificates do not exist to be used the way Cloudflare uses them. Wildcards are meant to be used so a single organization, for example a microsoft.com, doesn’t need unique certs for every server on their network. Using a wildcard to encrypt dozens or hundreds of completely unrelated organizations and origins is a misuse of the technology, but as I understand it that is precisely what Cloudflare does. I don’t mean to imply Cloudflare are the only abusers, but they’re the ones I know about.

are all also issuing certificates to CloudFlare, and indeed I don’t know of any CA that’s specifically adopted the rule that you’ve suggested. So, this doesn’t seem to be a matter on which Let’s Encrypt stands alone.

Before Let’s Encrypt there were not any trusted CAs issuing free SSL certificates. There’s a first time for everything.

Yeah, that is true. My thought on this whole thing is that if CDNs end up using Let’s Encrypt for SSL certificates, it will be a-okay as long as Let’s Encrypt makes sure everything goes smoothly for CDNs like CloudFlare.

I’ve said it before, and I’ll say it again:
If LE provided domain name services, it could mitigate a lot of sketchy scenarios.

Using a wildcard to encrypt dozens or hundreds of completely unrelated organizations and origins is a misuse of the technology, but as I understand it that is precisely what Cloudflare does.

That is not how cloudflare works, nobody can use subdomains of your domain, just because it is hosted by cloudflare

2 Likes

On the one hand, I love the idea of using Licensing Terms to leverage CDNs into encrypting origin traffic. On the other hand, I think it’s more important for traffic to be encrypted on that last mile.

But in the case of Cloudflare, they provide an immediately tangible benefit to upgrading http traffic to https when the origin domains contain content that is politically restricted. I’m not sure the world would be a better place if they were restricted to work with only http traffic - that puts a burden on the HTTP host that may not be feasible.

Nobody is suggesting permanently banning CDNs from offering HTTPS. I am suggesting they should be forced to improve their operating model.

If website.com signs up with $Cloudflare’s CDN, then the certificate that $Cloudflare serves should only be good for website.com. It should not be for website.com and flowers.net and pornstar.org and hackercentral.us plus 30 others. Certainly the origin certificate served by the actual website.com server is not valid for any of those other domains. In my opinion this reduces the trust value to null. Without trust, how can you verify real data versus hijacked phishing site data? Trust is everything. Trust is the entire reason we use CAs in the first place, otherwise all SSL would be self-signed. Serving trusted CA certificates which have zero trust value undermines the entire structure of HTTPS and presents false security to the visitor’s browser.

This is not unreasonable, it’s not unrealistic. They operate the way they do currently because they’ve been getting away with it. If pressured to, they will adapt.

That is not how cloudflare works

That is exactly how Cloudflare works. I can’t speak for other CDNs, however.

Why not? Why does it matter?

4 Likes

But is it let’s encrypt or the browser vendors role to enforce good reverse proxy designs? All the SSL connection guaranties is a secure connection between your browser and the public facing server. What happens behind that server may be horribly insecure (like unprotected mangodb) and the client itself may be compromised (a local malware may read the content of your page) but this is not what SSL is designed to protect against.

I share @mnordhoff’s confusion here–why does this matter? It would be straightforward enough, I suppose, for Cloudflare to obtain separate certs for each domain and use SNI to determine the correct one to serve (though I’m not sure how much, if any, overhead this would add on their end), so they could avoid this issue if they chose, but why should they?

You’re welcome to your opinion, of course, but not to your own facts. The fact is that Cloudflare has demonstrated adequate control over each of those domains in accordance with the CA/B forum requirements, exactly as they would if website.com and flowers.net were separate certificates. The user can be assured that, when they browse to website.com and see the green padlock (or whatever the UI indication is), that they’re at website.com–which is all that a DV cert validates in any event.

They’ve “been getting away with it” because it’s a perfectly sensible (and legal under the existing rules) way to operate. What difference do you think it would make if they obtained separate certs for each domain?

3 Likes

The one downside of their SNI implementation is that it tends to reveal which domains are owned by the same accounts, but certainly not a practical security issue, and definitely not abusable by end-users :smiley:

You mean currently? No, the cert covers domains owned by multiple accounts, or at least mine does. All my domains are on the same cert, but a bunch of other domains are as well.