Certmanager 400 HTTP-01 challenge propagation

[deibl]

Hi, we are using certmanager on K8s for our certs. They used to work but since a few weeks (without us changing anything we are aware of) the renewal is not working anymore, multiple cm-acme-http-solver (for different certs) are now stuck with "starting listener". The weird thing, for our other environment (different K8s Cluster) the work fine. We see this weird error in the challenge: "Waiting for HTTP-01 challenge propagation: wrong status code '400', expected '200'". Anyone has a hint what could be wrong?

My domain is:
https://example.com

I ran this command:
cmctl status certificate cert-tls

It produced this output:

...
Order:
  Name: cert-tls-5rkzg-1067775509
  State: pending, Reason: 
  Authorizations:
    URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/xxx, Identifier: example.com, Initial State: pending, Wildcard: false
Challenges:
- Name: cert, Type: HTTP-01, Token: xxx, Key: xxx, State: pending, Reason: Waiting for HTTP-01 challenge propagation: wrong status code '400', expected '200', Processing: true, Presented: true

The operating system my web server runs on is (include version):
K8s

[rg305]

@deibl

status code '400', expected '200'

Can you reach the site from the Internet?
Can you also reach a file placed into the expected challenge location?

[deibl]

"Can you reach the site from the Internet?"
Yes, exposed endoint is reachable from public internet:
http://auth.raya-diagnostics.com/.well-known/acme-challenge/*token*
"Can you also reach a file placed into the expected challenge location?"
To be hones, I am not sure what you mean. Challenges exists as K8s resources. Which location do you mean?

[rg305]

@deibl

Which location do you mean?

Something like:

@deibl

http://auth.raya-diagnostics.com/.well-known/acme-challenge/*token*

[that is the ACME challenge path]

[deibl]

Ah, I see, so I can reach the file and the site from the internet.
I did update certmanager to latest version, nothing change but the error message is now slightly different (used to be 400 instead of 404):
propagation check failed" "error"="wrong status code '404', expected '200'"
*edit:
In the cm-acme-http-solver i keep seeing this logs multiple time, without it terminating:

cert-manager/acmesolver "msg"="validating request" "base_path"="/.well-known/acme-challenge" "host"="example.com" "path"="/.well-known/acme-challenge/xxx" "token"="xxx"
cert-manager/acmesolver "msg"="comparing host" "base_path"="/.well-known/acme-challenge" "expected_host"="example.com" "host"="example.com" "path"="/.well-known/acme-challenge/xxx" "token"="xxx"
cert-manager/acmesolver "msg"="comparing token" "base_path"="/.well-known/acme-challenge" "expected_token"="xxx" "host"="example.com" "path"="/.well-known/acme-challenge/xxx" "token"="xxx"
cert-manager/acmesolver "msg"="got successful challenge request, writing key" "base_path"="/.well-known/acme-challenge" "host"="example.com" "path"="/.well-known/acme-challenge/xxx" "token"="xxx"

*edit2:
found out, upper logs was me when using curl

[deibl]

after removing certmanager completely and reinstalling it (with latest version) the issue is gone. Not sure what caused the issue initialy and how it is resolved now, i am sorry.