Need unblock IP

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. https://crt.sh/?q=example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

My domain is:unused

I ran this command:letsencrypt-auto certonly

It produced this output:
An unexpected error occurred:
There were too many requests of a given type ::
Your IP, 207.148.109.63, has been blocked due to ridiculously excessive traffic.
Once this is corrected you may request this be reviewed on our forum https://community.letsencrypt.org

My web server is (include version):caddy

The operating system my web server runs on is (include version):centos 7

My hosting provider, if applicable, is:vultr

I can login to a root shell on my machine (yes or no, or I don’t know):yes

I’m using a control panel to manage my site (no, or provide the name and version of the control panel):noVNC

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you’re using Certbot):none

Hi @znssky

check

same problem.

I haven’t heard of Certbot getting blocked like that.

Have you used other ACME clients before? In particular, older versions of cert-manager, or the software mentioned in the other thread?

How long have you had that IP address? Has it always been blocked?

One year.
I`d like to try caddy this month,but the ip has been blocked.

This is probably related to the issue at IP has been blocked. we recently blocked a bunch of IPs from misconfigured Caddy instances, possibly related to a bundle called V2Ray (more details in the other thread).

1 Like

Yes,i used V2Ray with mKCP as transport protocols by one year.
I’d like to try Caddy in this month.

1 Like

Thank you for the information. Please post this extra information:

  • When did you first install Caddy?
  • What commands did you use to configure Caddy?
  • What is your Caddy config?
  • What is in your Caddy logs?
  • What is in your system logs?
  • What is your domain name?

Thanks,
Jacob

1 Like

When did you first install Caddy?
The other day

What commands did you use to configure Caddy?
V2Ray

What is your Caddy config?
ns.sc2000.tk {
tls 18597543936@gmail.com
timeouts none
proxy / 127.0.0.1:35716 {
websocket
}
}

What is in your Caddy logs?
Activating Privacy features… done.
Serving HTTP on port 2015
http://:2015

What is in your system logs?
v2ray.com/core/transport/internet/websocket:
failed to server http for WebSocket >
accept tcp [::]:35716: use of closed network connection

What is your domain name?
ns.sc2000.tk

(One of your posts was caught by the forum’s automatic spam filter for a few minutes. Sorry.)

@znssky Thanks, we’re making progress. Can you please attach the full systemd Caddy process logs? I’d like to know what ACME errors your instance received from the beginning… that would be very helpful.

You can attach or link to the full log file.

That’s all

@mholt

Thanks, but I am looking for the logs from when the error started occurring, or from when you first started running Caddy. If running with systemd, you should be able to see some history with journalctl.

Thanks for the update! According to crt.sh no certificate was ever issued for ns.sc2000.tk. However, your IP address was sending new-acct requests for a long time. It sounds like maybe what happened was this:

  1. V2Ray set up a default Caddy config (with no hostname) that made Caddy try to create an account over and over again.
  2. We (Let’s Encrypt) blocked traffic.
  3. You tried to configure Caddy and then run it interactively, and got an error message.

Does that sound accurate?

@mholt, with the default config built by https://github.com/233boy/v2ray/blob/master/src/caddy-config.sh, would Caddy go ahead and create an account? Or would it wait to have a domain configured? What would happen if $domain in that script were empty?

Yes, I think it would create an account, but I don’t think $domain in that script is empty.

I’m willing to bet that the process was being restarted over and over without regard for exit code, though: CertMagic’s retry loop has a hard limit at 3 attempts before returning an error: https://sourcegraph.com/github.com/mholt/certmagic@403b55b6b7f6c31e35abae5a5e176d494e1142c2/-/blob/client.go#L117-119

If $domain was empty in this case, Caddy should also exit with an error before even getting to this point since server blocks without any keys are not allowed, so it’d be a parse error, basically, before the server even starts up.

So that’s why I think $domain is probably set, but I don’t know for sure without full log history.

Remember that before /new-acct is called, lego needs to set up a client which requires invoking the directory URL – only once that is successful can lego proceed to call /new-acct. So, it wasn’t IP-blocked from the beginning. Something else caused it to return an error at startup originally.

I am willing to bet it can’t write to storage. If it could, it would have re-used that account and not called /new-acct.

UNLESS, of course, the random email address is changing with every process restart. A new email address means a new account, even if it is saving to disk correctly.

I recently pushed a commit to CertMagic which tests the storage before attempting any ACME calls: https://sourcegraph.com/github.com/mholt/certmagic@403b55b6b7f6c31e35abae5a5e176d494e1142c2/-/blob/client.go#L75-105, but this will not solve the underlying problem…

1 Like

From what I seen in our logs, each IP address consistently uses the same random email address consistently, so I don’t think that’s what’s happening.

BTW one quirk that I hadn’t mentioned before: For a given IP address, when it first starts sending traffic with this problem, it sends clusters of five requests: /directory, /acme/new-nonce, /acme/new-acct, /acmt/new-acct, /acme/new-acct. The first new-acct request succeeds and returns 201. However, the next two new-acct requests generate an error: 400 :: malformed :: No embedded JWK in JWS header. Then the process repeats 4 seconds later, with the same pattern, until the requests start returning 429 (rate limited).

Unfortunately we don’t have the raw JWS header, but if you can think of any situation that could cause Caddy to send a new-acct request that’s missing a JWK, that might be an interesting avenue.

2 Likes

One more question, @znssky: Where did you install V2Ray from? What commands did you use to install V2Ray?

Thanks,
Jacob

Thanks for the details. I’m almost certain it has an error writing to storage. I’ll explain what I think is happening to show why:

  1. When Caddy tries to obtain a cert, it first must create an ACME client, which is a struct initialized in memory with some state resulting from a negotiation from the ACME CA. newACMEClient gets called, and the very first thing it does is try to get a user: https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/client.go#L89

  2. getUser will load the user with the given email address from storage, or specifically if it doesn’t exist, it will create a new user in memory (basically just generates a private key): https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/user.go#L168-172 – it does NOT store the user at this point (no point in doing so, since the user does not have an account).

  3. On the first run, a user gets created as expected. After some input validation, we see that no ACME client for the given CA & email exists in memory yet, so we have to create a new one. To do so, we first need to make a lego config, which takes the user as input: https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/client.go#L125-131

  4. After preparing the lego config, we finally ask lego to create an ACME client: https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/client.go#L148 - this is where the initial call to /directory happens. (Probably /new-nonce too, actually. I’m not super familiar here.) Before attempting to register a new user, we cache the client in memory for future use, to avoid hitting /directory more than once: https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/client.go#L153

  5. In any case, the very next thing to do is ensure that the user has an account. If not, one is created: https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/client.go#L170-180 - you can see that immediately after creating the registration/account, the user (account info) is saved to storage. I believe writing this resource to storage is what fails.

  6. If an error is returned at this point, it goes up to the caller in this loop: https://sourcegraph.com/github.com/mholt/certmagic@v0.7.3/-/blob/client.go#L70-73 – it retries (newer versions of the code wait before retrying, and only if the HTTP status is not 429).

  7. We repeat from step 1 again, except this time, when we get to step 3, an ACME client for the user email already exists in our in-memory cache, because we already queried /directory and we don’t need to do that again. So we skip to step 5. The cached client’s user does not have an account because there was an error storing it. This fails again, and the loop repeats one more time until the process exits.

  8. The process is apparently restarted in a tight loop.

That should explain the “quirk” you described with the pattern of requests.

Aha, sorry, upon a closer reading, I see that you’ve verified that the initial new-acct succeeds, and the subsequent requests have:

This is interesting, probably some mismatch between how lego expects to be used and how it’s actually being used. I’ll look into that.

However, I do think that’s a red herring to what’s actually going on at the crux of the excessive requests.

https://git.io/v2ray.sh
bash <(curl -s -L https://git.io/v2ray.sh)
@jsha
Thanks

Perhaps relevant - https://github.com/233boy/v2ray/issues/347

[ERROR] Making new certificate manager: could not save user: mkdir /etc/ssl/caddy/acme: read-only file system

2 Likes