Rate limit - SANs

Help
1

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is: cornell.edu

I ran this command:
docker service create --name traefik --constraint=node.role==manager --publish 80:80 --publish 443:443 --publish 8080:8080 --mount type=bind,source=/var/run/docker.sock,target=/var/run/docker.sock,readonly --mount type=bind,source=/mnt/docker1/is/traefik/acme,target=/etc/traefik/acme --network traefik-net traefik:1.3.1 --docker --docker.swarmmode --docker.domain=traefik --docker.watch --web --entryPoints=“Name:http Address::80 Redirect.EntryPoint:https” --entrypoints=“Name:https Address::443 TLS” --defaultentrypoints=“http,https” --acme.entryPoint=https --acme.email=***@cornell.edu --acme.storage=/etc/traefik/acme/acme.json --acme.onHostRule=true --debug --loglevel=DEBUG

It produced this output:
time=“2017-11-01T13:35:39Z” level=error msg="map[***.cornell.edu:acme: Error 429 - urn:acme:error:rateLimited - Error creating new cert :: too many certificates already issued for: cornell.edu]"
time=“2017-11-01T13:35:39Z” level=error msg=“Error getting ACME certificates [***.cornell.edu] : Cannot obtain certificates map[***.cornell.edu:acme: Error 429 - urn:acme:error:rateLimited - Error creating new cert :: too many certificates already issued for: cornell.edu]+v”

My web server is (include version):
traefik 1.4.0

The operating system my web server runs on is (include version):
Centos 7, Docker Datacenter (UCP version 2.2.3)

My hosting provider, if applicable, is:

I can login to a root shell on my machine (yes or no, or I don’t know):
yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):
No, but apparently someone else is…

Additional info: This works fine when I add this flag to the command above
–acme.caServer=https://acme-staging.api.letsencrypt.org/directory

I downloaded lectl, and that matches the output here https://crt.sh/?identity=%cornell.edu&iCAID=16418&p=1&n=100

cornell.edu appears in many of these *pantheon.io results as a SAN.

I’m confused to how pantheon (apparently?) can exceed the request limit rate for cornell.edu as a SAN, but I can’t get a single valid cert.

2

If you can't, so won't they at the moment. The rate limit is a sliding window though, so if on the "far end" (least recent) a certificate "falls off" (the window of exactly 7 days), one can issue a new certificate which includes the rate limited domain.

3

If SANs count, then they clearly are exceeding the limit.

They have a dozen SANs with *.cornell.edu just today…and nearly every other day, as shown in the link I provided.

lectl shows:
Sorry, you can’t issue any certificate, you already issued 20 certificates on last 7 days
You could issue next certificate on Tuesday 2017-Nov-07 11:59:00 EST

And I got the same message for the last week - but they continue to get many new certs with cornell.edu listed as a SAN every day.

4

At a guess they have agreed a rate exception with Let’s Encrypt. Pantheon rings a bell for me, didn’t they used to operate a subCA? I understand that organisations like Cornell can be intimidatingly big but it probably can’t hurt to try reaching out to them to find out if there’s a way you and other users can benefit from rate exception for Cornell names, or if there’s some practical alternative.

5

I’ve reached out within our organization.

Without someone from LetsEncrypt explaining exactly how this is happening, I’m unlikely to find the consensus required to make such a request (rate limit change).

I’m currently assuming that pantheon provides some type of website administration service, and so a cert is required on their side. That’s all well and good - except if they’re given an exception for their CN, and can add any of their customers domains, they can break processes at any of these relatively large organizations to the extent that I can NEVER request a LetsEncrypt cert…

6

I believe a rate limit exemption can be given for the combination of a particular domain name and a particular ACME account, in which case it doesn’t benefit other people trying to get certificates for the associated domain name (and indeed, might indirectly harm them). Or a rate limit exemption can be given for a domain name as a whole. However, I only think the former because the ACME account is mentioned in the rate limit request form… I don’t know for sure how it’s implemented on the back end.

Maybe @cpu could check for us what rate limit exemptions currently apply to Cornell.

7

There is not currently a rate limit exception for Cornell, though we’d welcome a rate limit override request from Cornell.

It looks like Pantheon has an override specific to their account id (rather than by domain name), which is why you see more than 20 certificates per week from them that contain subdomains of Cornell.edu.

If Cornell wants an exception, the decision between doing it for a single account id vs a domain name depends on Cornell’s IT structure. At most universities, IT is fairly decentralized, so various departments are likely to get their own certificates without coordination. In that situation you’d want a rate limit override by name, which is what most universities ask for. If, however, Cornell’s IT is highly centralized, and issuance for most subdomains will be managed by one organizations, you might choose instead to ask for an override by account id.

4 Likes
8

By the way, you are probably affected by this bug/misfeature in how renewal rate limiting works, which we are (slowly) trying to fix. That would improve your situation a lot, but fixing it has a few dependencies, including a large database migration, so it will take a little while.

9

@jsha

IT is fairly decentralized at Cornell, at least with respect to domain names, ie, administrators each have full control as far as dns and certificates of their own subdomains.

Who should make the request for the exception, and to whom/how should they make the request?

Also, I assume different limits are applied when a request is granted (not the complete removal of limits). Is there a way to determine the proper limit, possibly based on Pantheon’s activity/limit?

Finally, I agree, that the bug you described is at least partially the source of the problem. I assume Pantheon is renewing these certs, since if I search any of the cornell fqdns that are listed, I see pantheon acquiring a new cert roughly every couple days to at least several times a month and the subject name matches on them.

Thanks for clarifying this perplexing problem…

10

There's a form to fill out:

As for who should do it, I'd guess anyone who can reasonably speak for Cornell IT could do it, but I might be wrong.

2 Likes
11

@mnordhoff is correct, anyone who can reasonably speak for Cornell IT can do it. Sounds like that might be you. :smile:

After the renewal bug is fixed, I would say "estimate how many new subdomains you except to need certs in a given week, and then double that." However, so long as Pantheon is issuing for Cornell certificates, you will run into this bug as long as your limit is less than theirs. So write in the notes that it should be at least as high as Pantheon's, or maybe a bit higher.

12

@jsha

Our security office will be doing the request, I think.

I think we’re also talking to Pantheon about fixing their process (ie stop renewing certs 2-3x per week when they’re valid for 90 days).

Thanks again.

13

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.