Cert request throttling

Facing the limit on cert update going for 2 weeks now.

We use automatic routine to renew expiring certs daily using http-base domain validation. For one of the certs the process has been failing repeatedly for over a week, validation failure of one of SAN domains due to dns misconfiguration(not pointing to the right http server any longer). That triggered throttling on LE side:

{
"type": "urn:ietf:params:acme:error:rateLimited",
"detail": "Error creating new order :: too many certificates already issued for: schibsted.io: see Rate Limits - Let's Encrypt",
"status": 429
}

Limits ocumentation states that:

  1. "Exceeding the Duplicate Certificate limit is reported with the error message too many certificates already issued for exact set of domains."

However, the request still bounces immediately when I alter SAN domain list : removing one domain from SubjAltName and adding another. This is unexpected since doc says "already issued for exact set of domains". Key word is "exact". Technically since I changed SAN domain list in request it's supposed to be treated as new request.
Can someone elaborate on it? Is the doc inaccurate or am I misreading it?

  1. "We use a sliding window, so if you issued 25 certificates on Monday and 25 more
    certificates on Friday, you’ll be able to issue again starting Monday."

Seemingly the failed requests still count toward max threshold. I.e. even though the actual cert is never issued each attempt counts toward "5 per week".
The question is - do immediately bounced requests count toward max limit as well or not?
The reason asking is that it's unclear what is the true "wait" time until I can re-issue cert request without being bounced/limited.

Other details:
certbot version: 0.36.0

actual request:

[DEBUG] 2022-10-06T14:13:29.118Z 3d467498-77ca-4370-8fb6-80d3f78a9160 Arguments: ['certonly', '-n', '--agree-tos', '--email', 'xxx@myorg.com', '--server', 'https://acme-v02.api.letsencrypt.org/directory', '--manual', '--manual-public-ip-logging-ok', '--preferred-challenges', 'http', '--manual-auth-hook', '/var/task/auth-http.sh', '--manual-cleanup-hook', '/var/task/cleanup-http.sh', '-d', 'cmp.aftenbladet.no', '-d', 'cmp.aftenposten.no', '-d', 'cmp.aftonbladet.se', '-d', 'cmp.av-avis.no', '-d', 'cmp.avointyopaikka.fi', '-d', 'cmp.bareasane.no', '-d', 'cmp.bt.no', '-d', 'cmp.byas.no', '-d', 'cmp.bygdanytt.no', '-d', 'cmp.dinepenger.no', '-d', 'cmp.e24.no', '-d', 'cmp.finn.no', '-d', 'cmp.friflyt.no', '-d', 'cmp.friflytbestill.no', '-d', 'cmp.godare.se', '-d', 'cmp.godt.no', '-d', 'cmp.helthjem.no', '-d', 'cmp.honkhonk.dk', '-d', 'cmp.honkhonk.fi', '-d', 'cmp.honkhonk.no', '-d', 'cmp.honkhonk.se', '-d', 'cmp.jeger.no', '-d', 'cmp.kesaduuni.org', '-d', 'cmp.klart.se', '-d', 'cmp.klarttest.net', '-d', 'cmp.landevei.no', '-d', 'cmp.ledenicheur.fr', '-d', 'cmp.letsdeal.no', '-d', 'cmp.letsdeal.se', '-d', 'cmp.minmote.no', '-d', 'cmp.norsk-klatring.no', '-d', 'cmp.oikotie.fi', '-d', 'cmp.omakotimessut.net', '-d', 'cmp.omataloyhtio.fi', '-d', 'cmp.omni.se', '-d', 'cmp.omniekonomi.se', '-d', 'cmp.oyposten.no', '-d', 'cmp.peil.no', '-d', 'cmp.pent.no', '-d', 'cmp.podme.com', '-d', 'cmp.pricespy.co.uk', '-d', 'cmp.prisjakt.no', '-d', 'cmp.prisjakt.nu', '-d', 'cmp.privacy-dev.schibsted.io', '-d', 'cmp.privacy-pre.schibsted.io', '-d', 'cmp.privacy-pro.schibsted.io', '-d', 'cmp.puutarha.net', '-d', 'cmp.qasa.fi', '-d', 'cmp.rakentaja.fi', '-d', 'cmp.rakentaja.pro', '-d', 'cmp.randaberg24.no', '-d', 'cmp.rrmessut.fi', '-d', 'cmp.sahkovertailu.fi', '-d', 'cmp.strilen.no', '-d', 'cmp.svd.se', '-d', 'cmp.talovertailu.fi', '-d', 'cmp.tek.no', '-d', 'cmp.terrengsykkel.no', '-d', 'cmp.tori.fi', '-d', 'cmp.tv.nu', '-d', 'cmp.utemagasinet.no', '-d', 'cmp.vestnytt.no', '-d', 'cmp.vg.no', '-d', 'cmp.vgd.no', '-d', 'cmp.vglive.no', '-d', 'cmp.vgnett.no', '-d', 'cmp.vgtv.no', '-d', 'cmp.wellness.se', '-d', 'psi-dev.svd.se', '-d', 'psi.svd.se', '-d', 'samtykke.bilbasen.dk', '-d', 'samtykke.dba.dk', '-d', 'cmp.honk.no', '--config-dir', '/tmp/config-dir/', '--work-dir', '/tmp/work-dir/', '--logs-dir', '/tmp/logs-dir/']

Any help is much appreciated as the cert is about to expire in few days and we're stuck with hitting the limit.

Hello @sergeisp, welcome to the Let's Encrypt community. :slightly_smiling_face:

The first thing I noticed is

which is very old, I suggest updating Certbot.
What version of OpenSSL do you use?

Also double check you are not using less than TLS v1.2 and that you are using SHA-2 256 or newer.

Testing and debugging are best done using the Staging Environment as the Rate Limits are much higher. Rate Limits are per week (rolling).

And to assist with debugging there is a great place to start is Let's Debug.

2 Likes

You are correct about the rate limit of 5 / week for exact same set of names. But, there is another limit which is what I think you are seeing here. A portion fom the Rate Limit page:

The main limit is Certificates per Registered Domain (50 per week). A registered domain is, generally speaking, the part of the domain you purchased from your domain name registrar

The odd thing to me is I don't see any certs issued for that domain name. Not ever. So, I don't know how it could cause this rate limit. (maybe something wrong with log lookups today - don't know)

I saw some certs for schibsted.no but not schibsted.io. That shouldn't count since it's not the same domain of course. But, it's the closest thing to a clue I could find :slight_smile:

As a test, can you get a cert with just schibsted.io in it?

8 Likes

That's not the correct rate limit you're looking at. This one is for the entire domain, i.e., 50 certs per domain per week.

6 Likes

Does that mean that if I have 3 subdomains for one domain in the cert request and it ran once a day for a week I effectively exhausted 3*7=21 "certs", plus any manual cert requests would contribute another 3 on top?

Last time a cert for that domain (set of domains) was sucessfuly generated back I Jul. None since then.

Are you getting new certs daily?

4 Likes

No. I have automation which would try to renew the cert with expiration <30d. It will keep retrying daily.
Attempts to renew that particular cert failed due to failure to validate one of the SAN domains.
I always though only successful cert requests contribute to the limit threshold, apparently not the case.

I confirm the problem is due to excessive numbers of certs for the apex domain schibsted.io

There have been problems viewing the public Certificate Transparency logs today but I just got a good retrieval.

The rate limit described above is 50 such certs per week. I did not count them all but there are a lot. Below is just a small sample of recent certs.

You may have seen there is an option to ask for increases in some cases. See this topic

In the meantime, you may be able to get a cert for the other domains just by excluding subdomains for this problem domain.

6 Likes

Depends on the rate limit. Yes, only issued certs contribute to the duplicate certs rate limit and certs per domain rate limit, but e.g. the Failed Validation rate limit obviously does not depend on issued certs, but failed attempts.

The Rate Limits - Let's Encrypt page is pretty much self-explanatory. If it is NOT, we'd like to know so we can make the page better.

6 Likes

FYI - Certbot 1.31.0 Release

2 Likes

Thank you Mike! This finally makes sense now.
Much appreciated!

2 Likes

Thank you.
Guessing I took the wrong track with the "5 certs for domain set per week" from the very beginning, which creted a lot of confusion.
Things make sense now.

2 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.