Need unblock IP

That definitely looks relevant! There’s also lots of other closed/open issues there of folks having problems with caddy. People are getting linked to the v2ray.sh script from this wiki on the same repo:

2 Likes

Bingo. Nice find, thanks.

That’s the crux of the problem.

The latest CertMagic has a check to detect when the storage is inoperable before doing any ACME operations (do any other ACME clients do this?). It may lighten the load on LE’s servers, but it won’t fix the underlying problem.

@sydneyli Yeah, I’ve seen a few as I scoured GitHub, but strictly speaking this could happen with any ACME client: if the storage isn’t writeable, it will produce an error. And simply restarting the process endlessly in the hopes that the error goes away will naturally result in lots of network traffic.

I am not really sure where the problem originated and why so many people have read-only file systems, but that definitely needs to be addressed to fix this at large.

1 Like

Maybe https://github.com/caddyserver/caddy/issues/2698 ? systemd’s more exotic features are a bit of a pain in the ass to use safely because Linux distros have a very wide spread of versions …

1 Like

Which is exactly why we don’t have an official systemd unit. :slight_smile: (For now. v2 will have one.)

There’s a lot of history in the community-maintained systemd service file, and I’m not really up on all of it, but the Caddy docs are pretty clear as to how it should be wielded: https://caddyserver.com/docs/cli#exit-codes – as are our guidelines/recommendations for automated deployments: https://github.com/caddyserver/caddy/tree/master/dist/init#guidelines

Anyway, yeah, systemd is kinda the worst and nobody knows how to use it properly – I know I certainly don’t – all I can do for now is do my best to document how Caddy works and how it should be handled, then leave it up to the sysadmins to know how to use their computers.

I don’t think so—your stuff is used in far more fully automated and hands-off circumstances than most other ACME clients. (This is 100% meant as a compliment!)

1 Like

The idea is actually from @jsha. And unfortunately, this feature won’t be released until the next tag. I wish the current v2ray clients could have it already.

1 Like

Certbot has it somewhat accidentally - I think the lock files introduced in 0.14.0 would prevent a similar incident.

3 Likes

Thanks! I tried this script on a temporary host, and it was able to successfully set up Caddy and get a certificate. From the discussion at https://github.com/233boy/v2ray/issues/347, it sounds like 233boy recently pushed a fix to the script to address the systemd interaction reported in https://github.com/caddyserver/caddy/issues/2698. However, it’s not clear where the code is maintained. The 233boy/v2ray repo’s codebase just says “Removed.” @sydneyli do you have any idea where the source is maintained now?

One other interesting finding: On my test install, I get this systemd unit for Caddy, which has Restart=always and RestartSec=3, both of which contribute to excessive-traffic situations. If anyone has any idea where this systemd unit originates, I’d like to get it fixed.

[Unit]
Description=Caddy HTTP/2 web server
Documentation=https://caddyserver.com/docs
After=network.target
Wants=network.target

[Service]
Restart=always
RestartSec=3
Environment=CADDYPATH=/root/.caddy
ExecStart=/usr/local/bin/caddy -log stdout -agree=true -conf=/etc/caddy/Caddyfile -root=/var/tmp
ExecReload=/bin/kill -USR1 
KillMode=mixed
KillSignal=SIGQUIT
TimeoutStopSec=5s
LimitNOFILE=1048576
LimitNPROC=512

[Install]
WantedBy=multi-user.target

Change to the master branch. The (default) rm branch was last changed in May, but master is being actively maintained.

Strategy to hide from search engines? IDK.

2 Likes

Nice find, thanks! Looks like the script was previously getting the community-maintained Caddy config from the Caddy repo, but switched over to an inline-generated systemd unit when that stopped working: https://github.com/233boy/v2ray/commit/5fbdf869180f8d042257c55f8c49446494b370c8#diff-e16fccbf00a60d8781b481f7547dffe5R37-R58.

I can try sending the author a pull request.

1 Like

This one fixes Caddy’s systemd unit: https://github.com/caddyserver/caddy/pull/2798

And I posted to the related V2Ray issue asking them to change their systemd unit or go back to using Caddy’s: https://github.com/233boy/v2ray/issues/347#issuecomment-540270522.

5 Likes

It works
Thanks all of you
@JuergenAuer
@mnordhoff
@jsha
@mholt
@_az
@sydneyli
@schoen

1 Like