@jxman they have mentioned in the edits at the top of this thread that they do not have a locked down specific time, but that they suggest that you should consider your certificates as having been revoked as of 2020-03-04T00:00Z (midnight at the start of the 4th March UTC)
We have not started revocations but stated that 00:00 UTC on 04 March 2020 would be the earliest we would start that process. When we begin the revocations, we will post an update here.
A post was split to a new topic: Replacing certificates with acme.sh
A post was split to a new topic: Renewing Certs with acme.sh
Jillian - Are you able to provide some more details about when you might start revocations? We’re in panic mode over here as we attempt to figure out which of our customers this might effect. Delaying the cert revocation would definitely give us more time to respond
@jillian, I sincerely believe that if the requirements are open to interpretation in any way, you should at a minimum avoid revoking certificates until 2020-03-04T23:59Z to give people at least another 23.5 hours to fix their certs, if that falls within the interpretation of the regulations to which LE are bound.
In fact, having read the incident report, if the bug was confirmed at 2020-02-29 03:08 UTC, then 5 days should be 2020-03-05 03:08 UTC.
How long does LE expect the revocation process to take?
Yeah, tell me about it… What about people using this in an SaaS application!!! This will kill the app and not just ‘visitors will see security warnings’ guh!!
We are still assessing how long it will take to revoke this many certificates with our tools. As soon as we have an estimate, we will update the community forum and other communication channels for when we will start. Thank you for your patience and understanding while we gather this information.
I have updated the following question in our F.A.Q at the top of our page:
Understood, that is why I posted my question at on 04 March 2020 00:15 UTC. We are now past that window and I am growing concerned as my CDN is still deploying certificates.
Is down (no longer)
Same here, still waiting for validation for 2 customers(around 600 hostnames) we manage in CDN. Unfortunately validation is not as quick as normal causing some headaches over here. It would be best to raise the timeframe where you’re going to revoke these certificates. We’re 7,5 hours in since being notified and taking action and still waiting for challenges being generated and validated and this is assuming a happy flow where validation is succesful, if someone has taken records out of dns, there’s no time to respond and act accordingly.
I hope Let’s Encrypt waits until the end of it’s policy deadline to revoke. We’re working as fast as we can to obtain replacement certs, but with about 1000 to re-issue, we’re only 25% complete.
Would you mind sharing how you’re requesting certificates so that we could potentially parallelize it and speed up the process?
For next time, checkhost.unboundtest.com needs two additional features:
- The ability to test a specified port. Not all LE certs are on port 443.
- The ability to pass a cert number rather than a domain name. This will simplify the checking of internal hosts without having to download the entire caa-rechecking-incident-affected-serials.txt.gz
We’re requesting them via ACMEv1. We have our own client. We tried to parallelize it and quickly got into rate limits. In the hustle to try to get everything renewed quickly, we might have leaked some authz as well. I’m trying to clear out some of our long-standing authz to make room for these replacement certs.
Care to share which rate limit you’re hitting? We’ve been applying global rate limit overrides as the need occurs. We do not want to have anyone rate limited and be left behind due to our bug.
A post was split to a new topic: K8s cert-manager duplicate cert ratelimited
Our logging isn’t good enough to say with certainty, but I suspect we were receiving
retry-after headers from LE.
The error from the client library is
429 : 429 Too Many Requests
This is one of the errors we’ve been seeing
<head><title>400 Bad Request</title></head> <body> <center><h1>400 Bad Request</h1></center> <hr><center>nginx</center> </body> </html>
We’re still making progress, but it’s slow.
Rechecking CAA failures as well, likely due to DNS failures (SERVFAIL, timeout) for domains using Network Solutions/Web.com/worldnic.com nameservers