Plea to extend Let's Encrypt support to OCSP

LE downtime is rare and even when occurs is short.

I have LE's downtimes on record if you are so inclined to read them, and they are frequent.

If that weekly cronjob is doing the renewal that isn't best practice. Checking and trying renewal twice / day is default for Certbot, for example. No one would have advised to only do that once / week.

The cronjob has been adjusted to cope with delays, but when LE is down, it fails hard, and when DNS slaves do not update in time, it fails hard after a few hours of trying.

When things fail on the other side of the wire, there is nothing you can do about it. I made my recommendations, based on experience, but LE refused to listen. This problem would simply disappear if LE would read the DNS master, and then updates would take seconds.

LE walks the DNS tree from the root down to the authorative nameservers. That's just how it works. I'm pretty sure they're mandated to do so, without making a distinction between primary or secondary.

In the end you can give all the advice you want, but if LE isn't able to then you just have to accept that. And don't sulk that "[they] refused to listen". Because, is it refusal? Or unable to?

LE is a publicly trusted CA and needs to adhere to the CA/Browser Forum Baseline Requirements. They can't just do things differently because a single (!) user wants them to.

4 Likes

For anyone who were too lazy to check: OP had a thread here 4 years ago:

They had serious misconceptions on how DNS functions then, and apparently they still do…

3 Likes

When you mentioned two weeks, does that mean that your DNS configuration can potentially provide an inconsistent global view for two weeks after any change? That sounds very bad. Authoritative DNS should generally become consistent within minutes, maybe hours at most.

I'm not sure why availability is an issue. I use an ACME client that attempts to renew at 1/3rd lifespan remaining. If renewal fails it automatically retries after a gradually lengthening backoff period. This is after a preflight check that the DNS record is in place and correctly resolving globally.

Maybe Acme-DNS would be better and a way to move the challenge elsewhere, but your renewal process seems brittle and prone to errors.

DNS does not have a concept of master and slave, at least from a requestors perspective. If I am a client attempting to connect, I can query any of the DNS resolvers published for your zone and expect to get consistent results. If I do not, it is a problem on the server operator to fix.

5 Likes

Hear hear..

I think we define "LE downtime" differently. LE downtime, or outage if you will, affects a collection of people. Perhaps certain geographies, or certain types of connections, or even a bug across the LE API or Validation centers. Not failures to authenticate domains due to known configuration problems.

I think you define it as a certificate request that fails for you. And I am not surprised you see these often given your description of your system. Others have suggested methods to improve your reliability.

The picture below is from the Let's Encrypt 2024 Annual Report page 7. This is what I was describing

Source: https://www.abetterinternet.org/documents/2024-ISRG-Annual-Report.pdf

6 Likes

If they're so frequent, why has no one else noticed them? And why do you want to rely on an OCSP responder with so much downtime?

1 Like

Letsencrypt's DNS validation server does not check master DNS server, neither it checks slave DNS server. It checks only authoritative DNS servers. That can be a master DNS server and multiple DNS slave servers in a traditional configuration. Or, it can be only slave servers, if the master DNS server is a stealth master server. Or, it can be only master DNS servers, if the configuration is multi-master.

The important thing, that being authoritative that counts for checking. Being master or slave does not matter. By the way, a DNS resolver can't even be sure what is master and what is slave server to check if it tries to go the way you wish. The server entry in the SOA record is not reliable.

5 Likes

@patch-work : Welcome to the community.

Woah!! So lots of folks have posted in this thread and their contributions are extremely valid. There's lots of experience here and I want to acknowledge that, especially from longtime leaders like @petercooperjr and @MikeMcQ and @Osiris (The Good Doctor) and more...

That said, I think the thread has started to drift into other areas that don't really focus on the actual concern of the title. "A plea to extend Let's Encrypt support for OCSP"

Topics like "downtime", "DNS propagation delays", "registrar behavior" (OMG)... these are real issues, but they're separate discussions and they should not be in this thread. They should live in their own threads and many of them are not in the scope of this forum.

So I'd like to bring this back to the original point.

If you're dealing with this change and you want to stay focused on the real technical shift, and the end of "must-staple" certificates, My response below is intended to address exactly that.

Take a hard look at your Acme clients or client, whichever, and how you're requesting certificates. If you're still trying to request "must-staple", you're going to hit a wall. Can't avoid it. But your vhost certificate config, including OSCP stapling, that can stay.

It's still valid and useful. It's just not required anymore.

So let's stay on track and try to resolve the issue you mentioned in the title of this thread and that's what the thread and forum are meant for:

I see your point, and I get it. But it's important to separate two things that are now often confused by experienced system administrators and novices alike. It's a fact that needs to be addressed in many cases.

Requesting "must-staple" certificates from Let's Encrypt via ACME clients like Certbot, acme.sh, or any of the other multiple clients is no longer going to happen. It's just not possible. You'll get an exception error, a 403, maybe some kind of unauthorized response or something. Whatever the case is, your script or client will fail, and the cert will not be issued.

Let's Encrypt discontinued support for this feature, and they've announced it long in advance. They've "drawn the line in the sand". No more certs with the TLS feature extension that require stapled OCSP responses. That's it. That's just the way it is, and we have to accept it, deal with it, and just move forward.

On the other hand, using OCSP stapling in your server configuration for Apache, NGINX, Lighttpd, whatever server you choose is still completely valid, and there's nothing wrong with it. It does not violate best practices, and it will work. I'm doing it myself, and I'm very happy with the results. Your mileage may vary depending on your configuration. And of course, this is not an "Web-Server" configuration forum, but we're talking about certificates here, so I think this is appropriate.

The only difference now is that the certificate you're stapling no longer has the "Must-Staple" flag.

It's important to note that and that's perfectly fine. There's nothing wrong with it. There's nothing broken. Nothing to fix. You can still staple OCSP responses to your TLS handshakes if you want. It's up to you. It's just not mandatory or enforced by the certificate anymore.

So, of course, you can staple OCSP... You just can't require it when you're using your client to request a certificate. That's an important point, and that's where Let's Encrypt has made the change and "drawn the line" and opted to discontinue the feature.

Even if you try to require it... and this is an important note...most browsers don't care. They ignore it anyway. They don't pay attention. Chrome, Edge, Safari, Opera, whatever browser you're using... or one of your clients, even someone coming from the outside that you may or may not know... none of them consistently enforce "Must-Staple" anymore. It's just not happening.

So, in reality, you're not going to gain any meaningful security guarantees or advantages from the "Must-Staple" feature. That's why Let's Encrypt pulled the plug. It wasn't providing real-world value, and it just added complexity and unnecessary overhead. They decided to discontinue it, and that's their decision. It worked.

So, if OCSP stapling makes sense in your setup and you want to keep doing it, there's nothing wrong with that. It's perfect. It's a good decision. I do it. Go ahead and do it. It's good to do. It's not going to break anything.

But if your clients (and I'm hoping they're automated) are still trying to request "Must-Staple" certs from Let's Encrypt, they're going to fail. Quickly and utterly. It's just not going to work. Flat out... it's not the thing to do anymore.

So, do a reset. Let's make stuff work and make your system secure. That's the point.
The goal... the real solution:

Get rid of "Must-Staple" requests from your ACME clients that are trying to get OCSP Must-Staple certs.

Leave your virtual host configurations the way they are. They will work.

Then, the problem will be resolved, and the end result will not break your customers’, clients’, or visitors’ ability to visit your stuff or take in your services or enjoy your sites.

I did not this intend to become a "book". My apologies. I have experienced many of the same emotions/feelings relating to this subject.
Best to you all.

This is my "two cents"
Go figure.
RIP ;@)

5 Likes

To be clear: you will not be able to staple OCSP responses for Let's Encrypt certificates after August 6, 2025. As we've already announced, on that date we will shut our OCSP Responder service down. It will become impossible to obtain an OCSP response for any certificate issued by Let's Encrypt.

Keeping stapling configured in your webservers will probably be fine; I expect they fail fairly gracefully if there's no OCSP URL in the cert. And that's already the case today (we stopped including those in our certs on May 7), so you can test it easily. But it won't actually be doing anything, as there will be no OCSP response to staple.

7 Likes

The enforcement of this one-sided decision from LE and Google causes non-compliance in a highly-regulated sector. For the non-bureaucratically inclined, this means compliance with cyber-security and privacy laws that call for international standards and industry best practice. We can not write "we no longer comply with OCSP because two major companies are not using it". Compliance is stronger than a company's one-sided decision. If a product or service does not comply, then this product or service must be removed from the supply-chain, by law. If you know what I am talking about, good for you. If you do not know it, then you do not need to know it. I am not here do educate anyone on such matters.

Is there an RFC in the standards track that deprecates OCSP in favor or the new efficient CRL mechanism used by LE, Google and Mozilla? If one such RFC exists, then compliance with the new mechanism is doable. If it does not, then I must remove LE and Google from the supply chain.

As far as I know, there are no standards that prefer OCSP over CRLs or vice versa.

If you believe you need OCSP, then you should switch CAs. We cannot provide every feature for every user that may want it.

5 Likes

OCSP was made optional in favour of CRLs for CAs meeting the Baseline Requirements for the Issuance and Management of Publicly-Trusted TLS Server Certificates in SC063V4.

3 Likes

Highly regulated sectors are one area I would absolutely recommend a commercial CA.

If you have specific requirements, a commercial CA will have the resources and ability to work with you on a solution that meets all necessary requirements.

2 Likes

As far as I know, there are no standards that prefer OCSP over CRLs or vice versa.

If you believe you need OCSP, then you should switch CAs. We cannot provide every
feature for every user that may want it.

By shutting down OCSP, you are unable to comply with the latest Baseline requirements for TLS server certificates, version 2.1.5, 2025-05-16.

The CA SHALL make these records available to its Qualified Auditor as proof
of the CA’s compliance with these Requirements.

The CA SHALL record at least the following events:
1.6. Signing of OCSP Responses (as described in Section 4.9 and Section 4.10);
2.6. Signing of OCSP Responses (as described in Section 4.9 and Section 4.10).

Ref. page 77

The key words “MUST”, “MUST NOT”, “REQUIRED”, “SHALL”, “SHALL NOT”,
“SHOULD”, “SHOULD NOT”, “RECOMMENDED”, “MAY”, and “OPTIONAL” in these
Requirements shall be interpreted in accordance with RFC 2119.

Ref. page 35

[1] MUST This word, or the terms "REQUIRED" or "SHALL", mean that the definition is an absolute requirement of the specification.

Ref. RFC 2119 - Key words for use in RFCs to Indicate Requirement Levels

You are a CA, so you are expected to comply with the current baseline requirements.

LOL, funny you.

You're quoting parts about the auditing. If there's nothing being signed, there's nothing to be logged to be audited.

Please see:

where OCSP was made optional. This ballot has been linked earlier in Plea to extend Let's Encrypt support to OCSP - #36 by MaxHearnden already, just 12 hours ago.

3 Likes

The ballot is dated 2023. The baseline is dated 2025.

I need to see a published document where the Qualified Auditor certifies that LE's shutting down of OCSP complies with the baseline requirements. If this document exists, then I am satisfied and I shall deprecate OCSP. If this document does not exist, then I shall remove LE from the supply chain.

And it's current. OCSP has not been made mandatory again in more recent ballots.

I know these kinds of documents are often hard to read and even CAs make mistakes in reading them, but in this case I'm infinitely close to 100 % sure you're incorrect.

I'm pretty sure that's not how CA audits work.

3 Likes

4.10.2 Service availability

The CA SHALL operate and maintain its CRL and optional OCSP capability with
resources sufficient to provide a response time of ten seconds or less under normal
operating conditions.

(emphasis mine) Feels like there's a bit of a conflict between the actual requirements and audit requirements?

3 Likes

The WebTrust audit and ISRG Combined Certificate Policy and Certification Practice Statement are here: Policy and Legal Repository - Let's Encrypt

The published policies clearly state that OCSP is optional in line with Baseline Requirements

I don't think you will be shown any other audit than the one above. If that isn't sufficient you may need to adjust your business practices as you've described.

TL;DR: The audit did not report any compliance issues for its covered scope.

3 Likes