Revoking certain certificates on March 4

We’re requesting them via ACMEv1. We have our own client. We tried to parallelize it and quickly got into rate limits. In the hustle to try to get everything renewed quickly, we might have leaked some authz as well. I’m trying to clear out some of our long-standing authz to make room for these replacement certs.

2 Likes

Care to share which rate limit you’re hitting? We’ve been applying global rate limit overrides as the need occurs. We do not want to have anyone rate limited and be left behind due to our bug.

4 Likes

A post was split to a new topic: K8s cert-manager duplicate cert ratelimited

Our logging isn’t good enough to say with certainty, but I suspect we were receiving retry-after headers from LE.

The error from the client library is 429 : 429 Too Many Requests

This is one of the errors we’ve been seeing

<head><title>400 Bad Request</title></head>
<body>
<center><h1>400 Bad Request</h1></center>
<hr><center>nginx</center>
</body>
</html>

We’re still making progress, but it’s slow.

1 Like

Lots of Rechecking CAA failures as well, likely due to DNS failures (SERVFAIL, timeout) for domains using Network Solutions/Web.com/worldnic.com nameservers

2 Likes

@kf6nux,

We’re currently discussing a plan to reduce the QPS that we send to Worldnic/NetSol by adding more outbound IP addresses. I’m not sure when this will come to fruition, but it’s a route we’re pursuing.

3 Likes

Thank you for your feedback and suggestions. We will keep them in mind if we have to provide this tool/service in the future.

1 Like

This is probably not the best place for it, but MS are terrible at responding to anything and this is extremely short notice - Does anyone know if the team at Azure are aware of this imminent revoking and have a plan in place? I have certs provisioned that haven’t been renewed (and they don’t have the ability to manually renew CDN certificates)… I imagine they are a large consumer of letsencrypt certs and there would be a lot of people in the same boat.

1 Like

We believe the Web.com (Network Solutions) issue is mostly resolved, and affected subscribers should be seeing more success getting new certificates:

4 Likes

We’re in the same boat as @kf6nux. We have thousands of customer domains on WorldNic/Netsol and spread across our certs so those certs will keep failing.

1 Like

We saw the worldnic issue get a lot better about an hour ago and @JamesLE posted that it’s either fixed or mostly fixed here: DNS failures (SERVFAIL, timeout) for domains using Network Solutions/Web.com/worldnic.com nameservers

6 Likes

In order to complete revocations before the deadline of 2020-03-05 03:00 UTC, we are planning to start revoking affected certificates at 2020-03-04 20:00 UTC (3:00pm US EST). Please continue to renew and replace affected certificates in the meantime. If there are any changes to this start time, updates will be provided in this thread. Thank you all very much for your patience, understanding, and help as we work through this issue.

10 Likes

I have updated the top level FAQ to reflect this information

4 Likes

As of 06:45 UTC 04/03/2020 no email notification.
Saw an article in “The register” and took 30 seconds to renew a certificate.

More good luck than good management!!
Should have checked before renewing, I suppose. Checked after and it seems OK now

A post was split to a new topic: HTTPSConnectionPool(host=‘acme-v02.api.letsencrypt.org’, port=443): Read timed out

A post was split to a new topic: Certificate renew with Kubernetes cert-manager

A post was split to a new topic: How to reissue cert for QNAP NAS

Bit of an odd one, the email I received shows the domain and serial number of the affected domains but searching the caa-rechecking-incident-affected-serials.txt.gz the serial number isn’t there. So do I still need to re-issue the certificate?

@RobC-CTL Can you post the specifics?

Are you sure it isn’t just a syntax issue or something – some software displays serial numbers with colons and some doesn’t, and a simple grep won’t match them.

1 Like