Hacker bypassing rate-limit, causing DoS for my domain

[rahvee_tufts]

My domain is:
tufts.edu

I ran this command:
certbot certonly --preferred-chain 'ISRG Root X1' --non-interactive --preferred-challenges dns --authenticator dns-standalone --dns-standalone-address=0.0.0.0 --dns-standalone-ipv6-address=:: --dns-standalone-port=53 -d '*.apps.k8s-lab.it.tufts.edu'

It produced this output:
Error creating new order :: too many certificates already issued for "tufts.edu".

My web server is (include version):
n/a

The operating system my web server runs on is (include version):
n/a

My hosting provider, if applicable, is:
n/a

I can login to a root shell on my machine (yes or no, or I don't know):
n/a

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
certbot 1.29.0

I have reviewed

Rate Limits - Let's Encrypt

and
https://crt.sh/?Identity=tufts.edu&exclude=expired

As you can see, the number of new (not renewal or duplicate) certs issued in a 7-day rolling window exceeded 50 on 2/14/2023, when 13 new certs were issued (not renewal or duplicate). There have continued to be dozens of new certs issued per day since that time, which should all be denied according to the rate-limits. How / why were these certs allowed to be created, greatly exceeding the rate limits? I repeat: The number of new (not renewal or duplicate) certs per day were:

2023-02-14,13
2023-02-05,3
2023-02-06,0
2023-02-07,3
2023-02-08,18
2023-02-09,4
2023-02-10,0
2023-02-11,10
2023-02-12,4
2023-02-13,4
2023-02-14,13
2023-02-15,21
2023-02-16,7
2023-02-17,12
2023-02-18,22
2023-02-19,24
2023-02-20,19
2023-02-21,61
2023-02-22,12
2023-02-23,2

And the 7-day sliding window greatly exceeded 50. The peak I see was 166 on 2/21.

A hacker was exploiting a public wildcard dns entry *.platform01.tufts.edu until 2/22/2023 when we removed it. Since DNS is under control now and the new certs per day is again under control, the 7-day sliding window should allow me to generate certs, but I am still being rate-limited.

How was the hacker able to obtain so many certs, exceeding the rate-limit? I think they should have been blocked, just like me, starting on 2/14, but as you can see from the crt.sh logs, they kept on generating dozens of certs per day.

[Osiris]

Please note that crt.sh by default also shows the pre-certificates. You likely want to add deduplicate=y to the URL.

[mcpherrinm]

I spot-checked a few certificates for platform01.tufts.edu, like totomacauhariini.platform01.tufts.edu and they were all issued by a hosted CMS platform which has a rate-limit override for their platform account.

From your post, I'm guessing *.platform01.tufts.edu was pointed at a provider which allowed spammers to register accounts which included provisioning certificates for any domain pointed at them, and so they hosted spam or phishing pages there.

As a large institution, you (or others at Tufts) may want an override for tufts.edu too, since there's presumably many different groups operating under that domain. There's a straightforward form linked from the instruction page you've already seen: Rate Limits - Let's Encrypt

[rahvee_tufts]

OOOOoooohhhh! Figured it out, I think. A colleage just sent me this:

Configure DNS and Provision HTTPS | Pantheon Global CDN

It seems, Pantheon has increased their rate-limit, and they're constantly mixing and matching certificates that include the tufts.edu domain, so they are able to exceed the rate limit that we are not able to exceed, even though it's our domain.

The solution will have to be, I'll have to request an increase to our rate-limit to match pantheon's (even though we will never generate nearly as many as they do) because when they generate these certs, they count against our rate-limit too.

[rahvee_tufts]

It seems you and I arrived at the same conclusion at the same time. Thank you for your response.