A post was split to a new topic: Renewing Certs with acme.sh
Jillian - Are you able to provide some more details about when you might start revocations? Weāre in panic mode over here as we attempt to figure out which of our customers this might effect. Delaying the cert revocation would definitely give us more time to respond
1 Like
@jillian, I sincerely believe that if the requirements are open to interpretation in any way, you should at a minimum avoid revoking certificates until 2020-03-04T23:59Z to give people at least another 23.5 hours to fix their certs, if that falls within the interpretation of the regulations to which LE are bound.
In fact, having read the incident report, if the bug was confirmed at 2020-02-29 03:08 UTC, then 5 days should be 2020-03-05 03:08 UTC.
How long does LE expect the revocation process to take?
2 Likes
Yeah, tell me about itā¦ What about people using this in an SaaS application!!! This will kill the app and not just āvisitors will see security warningsā guh!!
1 Like
We are still assessing how long it will take to revoke this many certificates with our tools. As soon as we have an estimate, we will update the community forum and other communication channels for when we will start. Thank you for your patience and understanding while we gather this information.
I have updated the following question in our F.A.Q at the top of our page:
6 Likes
Understood, that is why I posted my question at on 04 March 2020 00:15 UTC. We are now past that window and I am growing concerned as my CDN is still deploying certificates.
2 Likes
https://checkhost.unboundtest.com/checkhost
Is down (no longer)
1 Like
Same here, still waiting for validation for 2 customers(around 600 hostnames) we manage in CDN. Unfortunately validation is not as quick as normal causing some headaches over here. It would be best to raise the timeframe where youāre going to revoke these certificates. Weāre 7,5 hours in since being notified and taking action and still waiting for challenges being generated and validated and this is assuming a happy flow where validation is succesful, if someone has taken records out of dns, thereās no time to respond and act accordingly.
1 Like
I hope Letās Encrypt waits until the end of itās policy deadline to revoke. Weāre working as fast as we can to obtain replacement certs, but with about 1000 to re-issue, weāre only 25% complete.
3 Likes
Would you mind sharing how youāre requesting certificates so that we could potentially parallelize it and speed up the process?
3 Likes
For next time, checkhost.unboundtest.com needs two additional features:
- The ability to test a specified port. Not all LE certs are on port 443.
- The ability to pass a cert number rather than a domain name. This will simplify the checking of internal hosts without having to download the entire caa-rechecking-incident-affected-serials.txt.gz
3 Likes
Weāre requesting them via ACMEv1. We have our own client. We tried to parallelize it and quickly got into rate limits. In the hustle to try to get everything renewed quickly, we might have leaked some authz as well. Iām trying to clear out some of our long-standing authz to make room for these replacement certs.
3 Likes
Care to share which rate limit youāre hitting? Weāve been applying global rate limit overrides as the need occurs. We do not want to have anyone rate limited and be left behind due to our bug.
5 Likes
A post was split to a new topic: K8s cert-manager duplicate cert ratelimited
Our logging isnāt good enough to say with certainty, but I suspect we were receiving retry-after headers from LE.
The error from the client library is 429 : 429 Too Many Requests
1 Like
This is one of the errors weāve been seeing
<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>
Weāre still making progress, but itās slow.
2 Likes
Lots of Rechecking CAA failures as well, likely due to DNS failures (SERVFAIL, timeout) for domains using Network Solutions/Web.com/worldnic.com nameservers
3 Likes
Weāre currently discussing a plan to reduce the QPS that we send to Worldnic/NetSol by adding more outbound IP addresses. Iām not sure when this will come to fruition, but itās a route weāre pursuing.
4 Likes
Thank you for your feedback and suggestions. We will keep them in mind if we have to provide this tool/service in the future.
2 Likes
This is probably not the best place for it, but MS are terrible at responding to anything and this is extremely short notice - Does anyone know if the team at Azure are aware of this imminent revoking and have a plan in place? I have certs provisioned that havenāt been renewed (and they donāt have the ability to manually renew CDN certificates)ā¦ I imagine they are a large consumer of letsencrypt certs and there would be a lot of people in the same boat.
1 Like