I'm experiencing a puzzling issue where both issuing of new certificates as well as renewals are failing for a specific domain.
Both result in the same error:
{
"type": "urn:ietf:params:acme:error:rateLimited",
"detail": "Error creating new order :: too many certificates already issued for: tickoweb.be: see Rate Limits - Let's Encrypt",
"status": 429
}
It is my understanding that renewals (i.e. certificates with Subjects and SubjectAlternativeNames identical to those issued ~60 days ago) should always work.
The list of all issued certificates can be seen here:
If I count correctly, for this domain, in the past 7 days there have been 26 "new" certificates (one occurence in the list), and 81 "renewals" (two occurences in the list).
It is not clear to me which limit might be hit here.
As an example, running the command below for guest.brabantsepijl.tickoweb.be (which is due for renewal today), yields the error shown above:
The rate limit being hit is Certificates per Registered Domain (50 per week).
Let's Debug says:
The 'Certificates per Registered Domain' limit (50 certificates per week that share the same Registered Domain: tickoweb.be) has been exceeded. There is no way to work around this rate limit. The next non-renewal certificate for this Registered Domain should be issuable after 2021-09-24 12:40:11 +0000 UTC (1h49m0s from now).
As for the renewals, I don't know the rate limits well enough to answer what's going wrong here. Maybe the system is confused and doesn't see your renewals as renewals.
Thanks for the pointer to letsdebug.net, that's useful information. I'll have to have a look at how the 50-per-week limit was hit in this case.
I am indeed mostly concerned about the renewals at this point, as I don't really understand why these would be affected. Any insights there would be greatly appreciated.
At least for this cert, you actually have several more weeks of use left in it - so it is not critical at this point.
If any other FQDNs are much more critical...
I would say that you may be able to use another FREE CA until this problem gets cleared up here.
Thanks for the hint about using another CA in the worst case. Yes, indeed, what will have to be the solution if we can't solve it.
We only have the issue since a day or two, and as we always renew after 60 days, we still have some time to figure out a solution. I hope someone can point me in the right direction here.
It looks like your site is set up to have an unbounded number of subdomains (one for each event using your platform), but that you host and control all of the content on every subdomain (they're not handed out to your clients for them to control, so they don't need to be sandboxed from each other). Why not use a single wildcard cert for this?
Although wildcard cert use would minimize the number of certs and renewals.
Multiple IPs are seen and also multiple subdomains are being used.
[not all FQDNs have only two dots - some have three dots]
So some care would have to be taken to include the right number of wildcard SAN entries on each of the systems in use.
Even the base domain resolves to yet another IP.
Not saying it's impossible... Just saying it might not be as super simple as their current process nor as one would expect/hope with implementing wildcard certs correctly.
[in this particular scenario]
But yes, I do agree that this is also a viable workaround to the current situation.
We are considering a wildcard for everything under the main domain (*.tickoweb.be), and specific certificates for others (foo.bar.tickoweb.be). It's on the wishlist, just not implemented yet. It's not as straightforward as, indeed, the infra here is slightly more complicated than the average setup.
While it is true that this would (currently) work around the bulk of the issues now, it still leaves the question as to why this is happening at all. Indeed, we would still have the same problem with the 4th-level subdomains.
Thanks for all your input so far, it's greatly appreciated!
LetsEncrypt staff identified this as an unintended regression caused by recent changes. This issue will likely affect many other users until resolved, but it is indeed a bug and the LE Staff will be addressing it.
I am sorry for being sparse on the details here. The LE Staff is busy addressing this and some other matters, and hastily provided us (community moderators) with a high level explanation. I don't want to paraphrase any of the information they've shared incorrectly.
We've got a temporary SRE hotfix coming shortly (within the next 20 minutes) followed by a boulder fix later on.
Edit: We deployed a temporary certificatesPerName override from 50/week to 150/week which should take care of most folks. The eventual boulder fix will take care of everybody.
@aarongable As you've mentioned the "no expiry e-mails send at all" bug was not related to this current thing. Is that also true for the newly opened issue you've posted just now?
Thank you so much for the swift identification and workaround!