Rate limits - how to known the exact limit reason please?

Help
1

My domain is: comptoirdespecheurs:com / www comptoirdespecheurs:com / fishingthespot:uk / fishingthespot:co.uk / fishingthespot:us

I ran this command: certbot --nginx -d comptoirdespecheurs:com -d www.comptoirdespecheurs:com -d fishingthespot:uk -d fishingthespot:us -d fishingthespot:co.uk

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Plugins selected: Authenticator nginx, Installer nginx

You have an existing certificate that contains a portion of the domains you
requested (ref: /etc/letsencrypt/renewal/comptoirdespecheurs.com.conf)

It contains these names: comptoirdespecheurs:com, www.comptoirdespecheurs:com

You requested these names for the new certificate: comptoirdespecheurs:com,
www.comptoirdespecheurs:com, fishingthespot:uk, fishingthespot:us,
fishingthespot:co.uk.

Do you want to expand and replace this existing certificate with the new
certificate?

(E)xpand/(C)ancel: e
Renewing an existing certificate
An unexpected error occurred:
There were too many requests of a given type :: Error creating new order :: too many certificates already issued for: comptoirdespecheurs:com: see https://letsencrypt.org/docs/rate-limits/
Please see the logfiles in /var/log/letsencrypt for more details.

My web server is (include version): nginx 1.14.0-0ubuntu1.6

The operating system my web server runs on is (include version): Ubuntu Bionic

My hosting provider, if applicable, is: SoYouStart

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): certbot 0.31.0

Hello,
Few days ago, our servers was on a Kubernetes cluster with Traefik for automatic TLS requests. Last week, we've moved to a temporary server, without Kubernetes/Traefik/Docker, a pretty old method with all services on the same server.
The TLS for the principal website (www).comptoirdespecheurs:com works and the TLS cert is ok. But I'm trying to add our international websites ( fishingthespot.* ) but we reach a rate-limit since last week, but we don't understood why.

On https://crt.sh/?q=comptoirdespecheurs.com we haven't reach 50 requests / week.

May be it's caused by our staging server, still on previous server with Kubernetes / Traefik, BUT it's not the same IP address, all subdomains on our domains (excepted www) are associated to a specific GIT branch of our development (with htpasswd). Excepted for one subdomain (master), all other new request for others subdomains are also refused for the same rate-limit error.

Our production servers DNS go on 94.23.31.39, and all our subdomains DNS (excepted www) go to CNAME staging.fishingthespot:com and redirect to 51.83.88.183

Since many months, on our Kubernetes/Traefik, we have requested many more TLS for all our future international website ccTLD ( .cn .it, .es ... ), but we've found many errors last weeks about our .cn ccTLD, so I've removed the Traefik TLS request for not yet used international websites ( so only TLS for .co.uk .uk and .us )

We've reach some rate-limits, but we don't known which ones, may be too much errors ( caused by .cn ) but the documentation explain it's while 1 hour maximum. So may be it's the 50 requests per Registred Doamin per week, but on crt.sh, this doesn't seems to have reach 50 requests...

Any help will be really appreciated ! :slight_smile:

2

Missing a wildcard there. Try this: Let's Debug Toolkit

3

Ok, so this list contain a LOT of subdomains was not requested by us !
Could we be victim of hacking ?
We’ve moved from our previous server ( Kubernetes/Traefik ) to another server because we had a lots of attacks and we would like to reduce all the stack to the minimum to be sure it’s not a problem of configuration.

Is it possible someone try multiple fake subdomain to prevent us to create our legitimate TLS certs ? :-/

4

Do you recognize the most recent domains from the last 2 hours? Such as cpanel.joomla.comptoirdespecheurs.com and mx4.comptoirdespecheurs.com?

If new certificates are being issued without your authorization, I would be in contact with OVH support.

It’s not possible for somebody to “fake” certificates like this without actually controlling your domain or k8s deployment.

5

No, cpanel.joomla and mx4 haven’t been requested by us.
Our development branch / subdomains are named “master”, “evol-XXX”, “bugfix-XXX”, and a special old “backup-php5”

6

Hi @noxdigital

there are a lot of subdomains - 113 in the last 7 days. Perhaps your system has a catch-all dns setup and renews certificates you don't need.

Checking your main domain you have a wrong redirect - https://check-your-website.server-daten.de/?q=comptoirdespecheurs.com

Domainname Http-Status redirect Sec. G
• http://www.comptoirdespecheurs.com/
94.23.31.39 301 https://comptoidespecheurs.com/ Html is minified: 107,78 % 0.060 E
• http://comptoirdespecheurs.com/
94.23.31.39 No GZip used - 10305 / 32238 - 31,97 % possible 200 Html is minified: 142,58 % 0.114 H
• https://www.comptoirdespecheurs.com/
94.23.31.39 301 https://comptoidespecheurs.com/ Html is minified: 107,78 % 3.243 B
• https://comptoirdespecheurs.com/
94.23.31.39 No GZip used - 10305 / 32238 - 31,97 % possible Inline-JavaScript (∑/total): 5/3289 Inline-CSS (∑/total): 1/882 200 Html is minified: 142,58 % 3.396 I
• https://comptoidespecheurs.com/ -1 0.043 R
NameResolutionFailure - The remote name could not be resolved: 'comptoidespecheurs.com' 
comptoirdespecheurs
comptoidespecheurs

A missing r.

Same with http + /.well-known/acme-challenge/random-filename. Maybe the failed authorization limit is hitted.

7

Yes there is a catch-all on subdomains DNS to the IP to our staging server, because for our CI/CD process, we permit to our developers to create a new GIT branch and then to test with the subdomain of the name of the branch, without the need to update DNS

May be I should restrict to a subdomain, like *.testing.comptoirdespecheurs:com, this could surely limit the surface of attacks.

8

How do you create ingresses for the new subdomains? Hopefully it is not done dynamically when somebody connects to your webserver … because that would explain the random certificates.

9

It's dynamic, so if you try for example « xzazmlkemkcc » on a subdomain, Traefik will try to route it to a container, but as there is no route that match it, there is no existing ingress rules for this subdomain, so « I suppose » Traefik don't request a TLS.

May be am I wrong, may be Traefik try even if there is no matching routes / container / ingress. That could explain this.

Here is the ingress YAML that we uses on previous production server (and actual staging server):

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: {{ template "fts-web.fullname" . }}-{{ .Values.http.name }}-ingress
annotations:
kubernetes.io/ingress.class: traefik
helm.sh/hook-weight: "40"
labels:
app: {{ template "fts-web.fullname" . }}-{{ .Values.http.name }}
chart: {{ .Chart.Name }}-{{ .Chart.Version }}
release: {{ .Release.Name }}
heritage: {{ .Release.Service }}
spec:

rules:
- host: {{ .Values.http.subdomain }}fishingthespot:com
http: &fts
paths:
- backend:
serviceName: {{ template "fts-web.fullname" . }}-{{ .Values.http.name }}
servicePort: 80
- host: {{ .Values.http.subdomain }}comptoirdespecheurs:com
http: *fts
- host: {{ .Values.http.subdomain }}fishingthespot:uk
http: *fts
- host: {{ .Values.http.subdomain }}fishingthespot.co:uk
http: *fts
- host: {{ .Values.http.subdomain }}fishingthespot:us
http: *fts

10

I’ve just updated our DNS to remove the wildcard on .com .co.uk .uk and .us, and I’ve keep only our master branch/subdomain.
So, this mean I need to manually update DNS of all our domains each time a developer create a new branch.
But this could block this kind of attacks.

Is it possible to reset the rate-limits or do we need to wait one week ?
Our international domains (co.uk / .uk, .us ) are off since few days, it’s really annoying…

Regards

11

@_az @JuergenAuer Do you known if we can reset the rate-limits ? I’ve removed our wildcard DNS and this seems to fix the new certs requests , no more since the 23th at 14:29UTC

But we are still blocked, our international websites are all down since many days because we can’t create new cert :frowning:

12

That's not possible if you have hitted the 50 certificates per week - limit.

If you’ve hit a rate limit, we don’t have a way to temporarily reset it. You’ll need to wait until the rate limit expires after a week.

If you have hitted the duplicate certificate limit (5 identical certificates), you can add a new domain name. But not with the 50 certificates / week limit.

Checking your domain there are duplicates - https://check-your-website.server-daten.de/?q=comptoirdespecheurs.com#ct-logs

Some certificates are created two times.

You have to change your setup, that's a bad configuration.

13

@JuergenAuer Thank for your answer :slight_smile:

In reality, we haven’t hitted the 50 certificates per week limit, it’s not us, we haven’t requested the majority of theses listed subdomains. We have respected the limit, we are victim of an attack / bug of bad configuration of Traefik I don’t known.

When I see the name of subdomain ( im-eu, homeart, mail2, mail10, mx5, chiatai, cpanel, … ), this seems to be an hacking tool that try to connect to multiple subdomain to find a security hole.

So, our Traefik configuration is set to generate TLS Cert onDemand, as all our subdomains go to our staging IP server. But, the ingress+routes configuration exists only if we have deployed into our Kubernetes the containers of our nginx+ingress.

If we try “random.” without anything, may be Traefik try to contact Let’s Encrypt before checking if a route/ingress exists. I’ll check with the Traefik dev team.

But, is it normal Let’s Encrypt count theses failed certs in our limit 50 / week ?

14

You have. It's your server and it's your wrong / terrible configuration.

I have an own subdomain service - customer name -> customer subdomain. There is a wildcard certificate, every customer subdomain is added manually (to the dns).

Some customers add own cname entries:

special-name.main-domain-of-the-customer.com -> CNAME server-daten.de

But I have to add a relation

special-name.main-domain-of-the-customer.com -> Goto that customer database + subfolder

So no random domain name works via connecting the ip and sending a domain name or a server-daten.de subdomain as hostname. And no certificate is created.

A wildcard dns entry + a wildcard webserver is a critical configuration. Because it's too open.

Failed validations have their own limit, that's not the 50 / week limit.

15

Ok, thanks for your help :slight_smile:

16

Is this issue one that could affect other Traefik users and therefore one that we might want to be in touch with the Traefik developers about in order to warn their users?

17

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.