Persistent 404 errors when trying to obtain SSL certificate

[utccnu]

Please fill out the fields below so we can help you better. Note: you must provide your domain name to get help. Domain names for issued certificates are all made public in Certificate Transparency logs (e.g. crt.sh | example.com), so withholding your domain name here does not increase secrecy, but only makes it harder for us to provide help.

I've been banging my head against the wall for days trying to obtain an SSL certificate for my domain. No matter what I try, I keep running into the same issue, and I'm at my wit's end. weirdly, i've been using this domain for days, but the cert got invalidated suddenly, and neither swag nor caddy nor certbot(in docker) nor certbot directly in shell works! I am 100% sure my vps has no firewall(ufw) enabled, and I did try nc from another machine connecting to port 80 without problem.
BTW, I also tried DNS challenge, also in vain.

My domain is:
gende.164136.xyz

I ran this command:

1. certbot certonly --standalone --debug --preferred-challenges http -d gende.164136.xyz
2. docker run -it --rm --network host certbot/certbot --staging certonly --standalone --preferred-challenges http --register-unsafely-without-email --non-interactive --agree-tos -d gende.164136.xyz

It produced this output:

Saving debug log to /var/log/letsencrypt/letsencrypt.log
Requesting a certificate for gende.164136.xyz

Certbot failed to authenticate some domains (authenticator: standalone). The Certificate Authority reported these problems:
Domain: gende.164136.xyz
Type: unauthorized
Detail: 88.99.50.117: Invalid response from http://gende.164136.xyz/.well-known/acme-challenge/BlP4nbvDrBcTN7sd52DV04QzUuPpXrRdU5yOGdW9ZsE: 404

Hint: The Certificate Authority failed to download the challenge files from the temporary standalone webserver started by Certbot on port 80.

My web server is (include version):
debian 12, docker(swag, caddy...)

The operating system my web server runs on is (include version):
debian 12

My hosting provider, if applicable, is:
hetzner

I can login to a root shell on my machine (yes or no, or I don't know):
yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
no

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
2.11.0

[MikeMcQ]

Welcome to the community @utccnu

Yes, I also see 404 errors for any kind of a request - not just Let's Encrypt HTTP challenges.

An HTTPS request uses a default cert from TRAEFIK. You didn't mention that in your first post. But, it looks like some Traefik system is handling all requests.

How is that involved in your local HTTP(S) request handling?

openssl s_client -connect gende.164136.xyz:443 | head

subject=CN = TRAEFIK DEFAULT CERT
issuer=CN = TRAEFIK DEFAULT CERT
notBefore=Jul  2 14:36:25 2024 GMT
notAfter=Jul  2 14:36:25 2025 GMT

[utccnu]

Thanks Mike!
I was confused at first at this, then it came to that I have been struggling with k3s and swarm lately as a newbie. Now it seems the problem is indeed the default ingress traefik router would listen on ports 80/443 on all nodes, and I didn't know that... so it messes everything up.

While i have another server on AWS tokyo not in a cluster but also facing similar problem. I'll look into it later, will get back if I cannot make it.

[webprofusion]

If you are using k8s (which is a complex environment for very large scale services) the conventional choice is to use cert-manager because it knows how to integrate with ingress and answer http challenges for you.