Revoking certain certificates on March 4

A post was split to a new topic: Renewing Certs with acme.sh

Jillian - Are you able to provide some more details about when you might start revocations? We’re in panic mode over here as we attempt to figure out which of our customers this might effect. Delaying the cert revocation would definitely give us more time to respond

@jillian, I sincerely believe that if the requirements are open to interpretation in any way, you should at a minimum avoid revoking certificates until 2020-03-04T23:59Z to give people at least another 23.5 hours to fix their certs, if that falls within the interpretation of the regulations to which LE are bound.

In fact, having read the incident report, if the bug was confirmed at 2020-02-29 03:08 UTC, then 5 days should be 2020-03-05 03:08 UTC.

How long does LE expect the revocation process to take?

1 Like

Yeah, tell me about it… What about people using this in an SaaS application!!! This will kill the app and not just ‘visitors will see security warnings’ guh!!

We are still assessing how long it will take to revoke this many certificates with our tools. As soon as we have an estimate, we will update the community forum and other communication channels for when we will start. Thank you for your patience and understanding while we gather this information.

I have updated the following question in our F.A.Q at the top of our page:

5 Likes

Understood, that is why I posted my question at on 04 March 2020 00:15 UTC. We are now past that window and I am growing concerned as my CDN is still deploying certificates.

1 Like

https://checkhost.unboundtest.com/checkhost

Is down (no longer)

Same here, still waiting for validation for 2 customers(around 600 hostnames) we manage in CDN. Unfortunately validation is not as quick as normal causing some headaches over here. It would be best to raise the timeframe where you’re going to revoke these certificates. We’re 7,5 hours in since being notified and taking action and still waiting for challenges being generated and validated and this is assuming a happy flow where validation is succesful, if someone has taken records out of dns, there’s no time to respond and act accordingly.

I hope Let’s Encrypt waits until the end of it’s policy deadline to revoke. We’re working as fast as we can to obtain replacement certs, but with about 1000 to re-issue, we’re only 25% complete.

2 Likes

@kf6nux,

Would you mind sharing how you’re requesting certificates so that we could potentially parallelize it and speed up the process?

2 Likes

For next time, checkhost.unboundtest.com needs two additional features:

  1. The ability to test a specified port. Not all LE certs are on port 443.
  2. The ability to pass a cert number rather than a domain name. This will simplify the checking of internal hosts without having to download the entire caa-rechecking-incident-affected-serials.txt.gz
2 Likes

We’re requesting them via ACMEv1. We have our own client. We tried to parallelize it and quickly got into rate limits. In the hustle to try to get everything renewed quickly, we might have leaked some authz as well. I’m trying to clear out some of our long-standing authz to make room for these replacement certs.

2 Likes

Care to share which rate limit you’re hitting? We’ve been applying global rate limit overrides as the need occurs. We do not want to have anyone rate limited and be left behind due to our bug.

4 Likes

A post was split to a new topic: K8s cert-manager duplicate cert ratelimited

Our logging isn’t good enough to say with certainty, but I suspect we were receiving retry-after headers from LE.

The error from the client library is 429 : 429 Too Many Requests

This is one of the errors we’ve been seeing

<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

We’re still making progress, but it’s slow.

1 Like

Lots of Rechecking CAA failures as well, likely due to DNS failures (SERVFAIL, timeout) for domains using Network Solutions/Web.com/worldnic.com nameservers

2 Likes

@kf6nux,

We’re currently discussing a plan to reduce the QPS that we send to Worldnic/NetSol by adding more outbound IP addresses. I’m not sure when this will come to fruition, but it’s a route we’re pursuing.

3 Likes

Thank you for your feedback and suggestions. We will keep them in mind if we have to provide this tool/service in the future.

1 Like

This is probably not the best place for it, but MS are terrible at responding to anything and this is extremely short notice - Does anyone know if the team at Azure are aware of this imminent revoking and have a plan in place? I have certs provisioned that haven’t been renewed (and they don’t have the ability to manually renew CDN certificates)… I imagine they are a large consumer of letsencrypt certs and there would be a lot of people in the same boat.

1 Like