I was thinking about the impact of no OCSP to the greater PKI ecosystem; and a random thought hit me. Instead of sunsetting OCSP and switching to CRLs; why not force the Must Staple flag on every issued certificate?
Here me out… The two big concerns are privacy of users and infrastructure costs for Let’s Encrypt. In a world where every issued certificate is flagged with Must Staple, both of those concerns are addressed. Privacy wise (by and large) the same server/IP that requested the original cert will be the one asking for an OCSP response, so nothing new is revealed there. Performance wise, the vast majority of OCSP requests today are from clients… clients connecting to server which default to disabling OCSP stapling.
If the rollout is planned with a big enough notice (say 3-6 months), it’ll encourage the major software packages to enable OCSP stapling by default (for example, Apache). While not every server on the internet will get patched during the notice period, support for OCSP stapling has been built-in for many years to most major software packages. Often, it’s fairly easy for an admin to enable. Maybe a little prodding from certbot can also help nudge this along.
Obviously, this isn’t a perfect plan. But forcing OCSP stapling seems like a better solution than going back to CRLs. CRLs have a long history of being configured to fail-open, and being ignored completely once any discernible performance impact is detected.
Apparently, I've learned from this community, the (default) OCSP stapling implementations in major software like Apache and nginx are often buggy unfortunately. That would be a big problem for 100 % must staple.
I agree with both of you. The current state of OCSP stapling support is not great. But, why is that?
Simple, OCSP stapling has always been a “nice to have” feature. So major software packages have never prioritized implementing it and supporting it. The lack of OCSP stapling support has never been a deployment blocker.
But, what if it was? What if the world’s biggest CA says, “you want to use our free service? Okay, cool, but you can’t overwhelm our OCSP responders. Support OCSP Stapling, or use a different provider”. I think a major shift would happen amongst the largest software packages and operators. It’d force the greater ecosystem to finally make OCSP stapling ubiquitous.
This is not without precedent. Many will remember in 2014 when Google announced they’d increase site’s page rank if they supported HTTPS. This lead to a huge spike in the adoption of HTTPS. Many complained how hard it would be, and how many incompatibilities would erupt, but the world figured it out, and we now have a much more private web because of it.
There are a lot of servers out there that aren't really being "administrated". If someone needed to upgrade its software or reconfigure it in order for it to continue working, it will probably just stop working. Some of them might make their way here, asking us how they're supposed to log into their system to update it.
But I think the most practical version of what you're suggesting is what they are currently considering, of continuing to support OCSP for must-staple certificates, and only must-staple. This takes a lot of load off of their OCSP systems, and gives the benefit of stapling for servers that do have it set up properly.
I'm not sure how you'd convince the people running all the various servers to either switch to Caddy or to configure however stapling works best on their existing servers, but doing so sounds great to me. It may be that Let's Encrypt trying to push for it might help, but there are many other free CAs that are just a ACME-server configuration option away, so I'm not sure that they really have as much influence as you're hoping. (Certainly Google saying that sites using must-staple would get an SEO boost would be more helpful, but I tend to doubt they'd go that direction.)
Though I do like that idea, the big problem is it's opt-in. History has shown, the vast majority of people leave settings at default.
My hope is all the major platforms (i.e. Apache, Nginx, Postfix, etc) will finally be incentivized to clean up their OCSP stapling code, and enable it by default. Meaning, that for the vast majority of sysadmins, all they'd do is install the latest update and they're ready to go.
So what if some users leave? Let's Encrypt is not a for-profit business, there are no shareholders to please, no quarterly growth targets to hit. I suspect every free CA has problems funding their operations, those alternatives very well might be forced to make the same decision once their usage goes up.
That's an excellent idea. I can see Google being a fan of that too. since it'd put less pressure on CRLSet. Another option would be the for-profit CAs giving a sizable discount for including the Must Staple flag. Under the presumption that such a certificate would reduce the load on their OCSP infrastructure. Obviously, neither is solution that Let's Encrypt could implement.
Thank you for your thoughtful and detailed response!
Because this idea has now come up a few times here (also in this thread: Sunsetting of OCSP in favor of older technology?) I have setup a public test site so that you can see whether your browser/TLS client actually supports OCSP must-staple.
The site is deliberately broken: It uses a cert with must-staple extension, but doesn't actually staple OCSP. A conforming TLS client must fail the connection. If your client connects nevertheless, it does not support OCSP must staple.
That's generally what happens when crypto standards evolve though (for example when SHA-1 certificates were phased out, or more recently, TLS 1.0 and 1.1). Implementations have to follow, or they become incompatible with the rest of the world.
But this usually happens on the scale of years, not months as suggested here.
And for OCSP specifically, the industry trend seems to be to move away from it, rather than towards.
Speaking as a former Firefox Crypto Engineer, I suspect this would quickly lead to having to use Remote Settings to issue a pref flip for security.ssl.enable_ocsp_must_staple to turn it off as a hotfix, because it’d be back to “breaks in Firefox, works in Chrome/Safari.”
Requiring must-staple goes well beyond just the handful of well-known "servers". The large CDNs don't readily support it today either. AWS and Cloudflare's CDNs have a total market share in that category above 70%. Not small niche providers
The web ecosystem is vast and material changes take years not months. The long-term direction is clearly automated short-lived certs.
I checked AWS CloudFront at some length and could not find a way to enable it on the standard ACM cert. Perhaps I could create a cert separately with must-staple and upload it to CloudFront but I found no docs that said their edges would honor that. Same if using an AWS Private CA regarding the edges. I found one (credible) thread that said the edges automatically staple OCSP based on volume of requests at that edge. This leaves considerable number of requests not stapled at all let alone the unknown about must-staple.
I checked two unrelated Cloudflare Universal SSL certs and each got OCSP responses. But, neither had Must-Staple enabled. One of these was their blog.cloudflare.com domain which boasted back in 2017 (link here) that they were so confident in their OCSP support they would set must-staple on their blog domain to prove it. Somewhere along the line must-staple was dropped. A post in their Community's Feature Request (link here) asked for must-staple in 2023 but only got one like and no official reply.
They had it on 3 Comodo-issued certificates all issued in 2017 (2x RSA [1][2], 1x ECC [3], the first one apparently reissued due to dropping the www prefix)
All later certificates (first issued approximately 10 months later) no longer had the must-staple set, up to the present day. This may or may not have been related to a CA change from Comodo (now Sectigo) to DigiCert that happened in this timeframe. In any case, it shows that they really couldn't bother too much, despite claiming how important it is to them.
