New certs failed with "A rate limit prevents DCV"

[Osiris]

@timeofmind

So you are (…)

Note that most of the users here, myself included, are just volunteers of this Community and not related to Let's Encrypt in any way, besides being users of it like yourself. I'm just re-iterating statements made earlier in combination with my views/interpretation of the situation.

@timeofmind

Unlike CPanel, we keep track of when our domains need renewal in a table, so that we do not have to check every cert on our servers, so we are not affected by the same bug affecting CPanel and our request rate has not increased. Now we are being punished for a bug in CPanel....

That migh be the case indeed. As in, sounds likely to me, without exactly knowing the migitation deployed by Let's Encrypt.

And yes, that's very annoying and I can understand your issue with such a broad mitigation. However, I'm fairly certain that if Let's Encrypt had a better way of mitigating this cPanel (#)$)(#$(#), they would have. I'm not sure if "the good suffer for the bad" is a proverb in English, but this seems to be a class book example of it.

[JamesLE]

Yes, the rate limit on this user-agent is a temporary emergency measure.

[ismailgorkemkara]

@JamesLE

Evet, bu kullanıcı aracısındaki oran sınırı geçici bir acil durum önlemidir.

So, is it clear when this limit will be lifted or eased? We manage a large network, it causes us problems with our customers.

[ismailgorkemkara]

The solution that cPanel has published will have worked in the next /scripts/upcp task on all servers. Many server administrators have already implemented the solution manually.

[jcjones]

We disabled the temporary rate limit 13 minutes ago, though that has prompted another bout of extremely high traffic volume. We might need to re-impose a (lighter this time) limit as we monitor.

[Osiris]

So to summerise: in effect, a buggy cPanel is DDoS-ing the Let's Encrypt ACME API, mandating mitigating actions which unfortunately affect all users with the same user agent. As Let's Encrypt just has the user agent to work with, it cannot distinguish cPanel users from non-cPanel users.

[ismailgorkemkara]

@Osiris

So to summerise: in effect, a buggy cPanel is DDoS-ing the Let's Encrypt ACME API, mandating mitigating actions which unfortunately affect all users with the same user agent. As Let's Encrypt just has the user agent to work with, it cannot distinguish cPanel users from non-cPanel users.

Yes, I can understand your situation, but it is a difficult process for us as well. I hope you understand.

@jcjones

We disabled the temporary rate limit 13 minutes ago, though that has prompted another bout of extremely high traffic volume. We might need to re-impose a (lighter this time) limit as we monitor.

I'm grateful to you, I think the traffic involved will decrease before it's too long. Currently connecting servers that have not been able to connect for hours.

[jessika]

There is still a foot problem and the following error

WARN AutoSSL failed to create a new certificate order because the server’s Let’s Encrypt account (https://acme-v02.api.letsencrypt.org/acme/acct/208383230) has reached a rate limit. (429 urn: ietf: params: acme: error: rateLimited (The request exceeds a rate limit) (Error creating new order :: too many certificates (5) already issued for this exact set of domains in the last 168 hours: mail. mydomain.com, www.mydomain.com: see Rate Limits - Let's Encrypt)) You may contact Let's Encrypt to request a change to this rate limit.
ERROR “Let’s Encrypt ™” general error (www.mydomain.com): A rate limit prevents DCV.
ERROR “Let’s Encrypt ™” general error (mail.mydomain.com): A rate limit prevents DCV.
ERROR Impediment: TOTAL_DCV_FAILURE: Every domain failed DCV.

[jcjones]

@jessika

: too many certificates (5)

This is a more normal rate limit error, that the FQDN has had too many duplicate this period (a sliding 7 day window). Rate Limits - Let's Encrypt

Sadly, each certificate requires lots of upkeep over its 90 day lifespan, so we do limit the number of duplicates out there to keep our total issuance volume within our maintenance capacity.

[jcjones]

@ismailgorkemkara

I'm grateful to you, I think the traffic involved will decrease before it's too long. Currently connecting servers that have not been able to connect for hours.

I hope so too! Right now we're at just a little over 2x normal volume, which is better than the 4x volume from earlier, and if we can just stay at this volume we should be fine.

4x volume starts to get into issues with overloading external certificate transparency logs and all kinds of fun stuff.

[eva2000]

@jcjones

We disabled the temporary rate limit 13 minutes ago, though that has prompted another bout of extremely high traffic volume.

How's that AMD EPYC issuance server doing? I thought there was enough additional capacity there? After the dust has settled, would be curious to know the issuance numbers/rates it was handling

[eva2000]

@Osiris

As Let's Encrypt just has the user agent to work with, it cannot distinguish cPanel users from non-cPanel users.

Cpanel should pass on their own custom user agent. I do for my issuances via acme.sh for my users' automatic Nginx HTTPS generated sites.

[tarimehr]

we have same issue
all of our hosting site over 2,000 site cert expired (Lets Encrypt)
i removed lets encrypt script and re-installed it and new letsencrypt version is installed.
but we have an error:
MASTER DCV: A rate limit prevents DCV.

Please tell me, What to do

thanks

[Osiris]

@eva2000

Cpanel should pass on their own custom user agent.

It probably should, yes.

@tarimehr

Please tell me, What to do

Please give us the exact error message provided by the ACME server. Because unless the temporary cPanel rate limit has been re-instated, it probably is a different rate limit you're hitting.

[jcjones]

@Osiris

Because unless the temporary cPanel rate limit has been re-instated,

It has not been reinstated. cPanel has no specific rate limits right now.

[jcjones]

There's a lot of overhead for certain things, but various other systems assume relatively steady load and need a bit of tweaking to go beyond a short burst every now and then.

The obvious one is other organizations' certificate transparency logs, since we want to be good netizens. But that's going well off topic.

[hassankhalid]

@jcjones

There's a lot of overhead for certain things, but various other systems assume relatively steady load and need a bit of tweaking to go beyond a short burst every now and then.

I am facing the similar issue

[eva2000]

@jcjones

The obvious one is other organizations' certificate transparency logs, since we want to be good netizens. But that's going well off topic.

Ah did notice the certificate transparency logs were slower to show newly issued certs lately! Though I'd think this is a good real world load test case, so folks can plan for further scalability