(kubernetes/cert-manager) failed to perform self check GET request

Help
1

I have an existing website which uses let's encrypt to renew its certificates.

Recently I deployed a new kubernetes cluster along with one application. I'm now working to get let's encrypt working to renew its certs, so that I can transition over to the new server.

  • kubernetes v1.32.4
  • name: nginx-ingress-controller
    version: 11.6.13
    repository: oci://registry-1.docker.io/bitnamicharts

I essentially copied over the same let's encrypt ClusterIssuer configuration so I'm not sure why things aren't working.

Each steps seems to have worked up until the challenge step, which results in the error mentioned before.

I can use public web testing tools to access the website. If I watch the nginx log I can see the test connections coming in and making their way to the nginx controller.

The "cm-acme-http-solver-pzn8g" reports all is well:

I0515 23:42:09.271088       1 solver.go:112] "got successful challenge request, writing key" logger="cert-manager.acmesolver" host="mail.aarr.xyz" path="/.well-known/acme-challenge/6osV5p-p9lJ7UhzFd0YWhsdyiLIYK9oq_yv2begXM1o" base_path="/.well-known/acme-challenge" token="6osV5p-p9lJ7UhzFd0YWhsdyiLIYK9oq_yv2begXM1o" headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/f3ffb86641f75d94d01e5a2606b9871ff89645ef"],"X-Forwarded-For":["172.16.99.192"],"X-Forwarded-Host":["mail.aarr.xyz"],"X-Forwarded-Port":["8080"],"X-Forwarded-Proto":["http"],"X-Forwarded-Scheme":["http"],"X-Real-Ip":["172.16.99.192"],"X-Request-Id":["a07e052d4c294df379b9b18df68ed43f"],"X-Scheme":["http"]}

additional:

$ k get clusterissuers.cert-manager.io 
NAME                         READY   AGE
cluster-letsencrypt-issuer   True    2d20h

$ k get challenges.acme.cert-manager.io mailu-certificates-1-91564087-381687296 
NAME                                      STATE     DOMAIN          AGE
mailu-certificates-1-91564087-381687296   pending   mail.aarr.xyz   12m

Since I appear to be able to browse all the way in from the outside, not sure where the problem is.

2

The incoming connection looks like:

172.16.99.192 - - [16/May/2025:00:02:00 +0000] "GET /.well-known/acme-challenge/...redacted... HTTP/1.1" 200 87 "-" "cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/...redacted..." 264 0.000 [mailu-cm-acme-http-solver-v8f2h-8089] [] 172.16.29.63:8089 87 0.000 200 2699c1989e39ddf4531409cc60d68b9b
3

That's from cert-manager self test, not LE itself:
"User-Agent":["cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/f3ffb86641f75d94d01e5a2606b9871ff89645ef"] "X-Real-Ip":["172.16.99.192"]

looks like it didn't seen any actual challenges traffic from LE?
call mail.aarr.xyz from outside and see where traffic routered: I hit by 308 redirect to https version than nginx 503 with Kubernetes Ingress Controller Fake Certificate

2 Likes
4

it will redirect unless you get the full path right, which is specified in the ingress, otherwise it uses another ingress to an application which isn't fully running yet, or in the case of an error returns a static image

5

I don't see any image from error page: just nginx default 503

1 Like
6

the ingress is like so:

spec:
  ingressClassName: nginx
  rules:
  - host: mail.aarr.xyz
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-v8f2h
            port:
              number: 8089
        path: /.well-known/acme-challenge/redacted
        pathType: ImplementationSpecific

I'm hesitant to give out the full path of course

7

that token is change every time new acme order is passed, so unless http-solver can update ingress rule too it wouldn't work

2 Likes
8

I just went to browser.lol and put the url in there, and it worked from there; so which seems to verify that the request should be able to make it all the way in

i can delete the ingresses and let them recreate ... let me see if i can restart the processes

9

it can work now but at next renewal token changes, than it wouldn't able to renew

2 Likes
10

I think it creates the ingress on the fly whenever it needs one to renew, then it goes away, if I recall correctly. It was working before for years, no issue.

11

since i was able to do it via the virtual website, that makes this especially weird, i deleted a bunch of things and the were automatically recreated as i would have expected to perform the renewal

12

the the lets encrypt pod it looks good showing things like:

"validating request" logger="cert-manager.acmesolver"
"comparing token" logger="cert-manager.acmesolver"
"got successful challenge request

repeating every 12 seconds

13

think problem is cert-manager doesn't call LE to come and see your challenge, can we see full log from LE pod?

2 Likes
14 
I0516 00:56:06.072239       1 solver.go:89] "validating request" logger="cert-manager.acmesolver" host="mail.aarr.xyz" path="/.well-known/acme-challenge/...redacted..." base_path="/.well-known/acme-challenge" token="...redacted..." headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/f3ffb866496871ff89645ef"],"X-Forwarded-For":["172.16.99.192"],"X-Forwarded-Host":["mail.aarr.xyz"],"X-Forwarded-Port":["8080"],"X-Forwarded-Proto":["http"],"X-Forwarded-Scheme":["http"],"X-Real-Ip":["172.16.99.192"],"X-Request-Id":["647fcb0cbf90eb593"],"X-Scheme":["http"]}
I0516 00:56:06.072266       1 solver.go:97] "comparing host" logger="cert-manager.acmesolver" host="mail.aarr.xyz" path="/.well-known/acme-challenge/...redacted..." base_path="/.well-known/acme-challenge" token="...redacted..." headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/f3ffb865d9498789645ef"],"X-Forwarded-For":["172.16.99.192"],"X-Forwarded-Host":["mail.aarr.xyz"],"X-Forwarded-Port":["8080"],"X-Forwarded-Proto":["http"],"X-Forwarded-Scheme":["http"],"X-Real-Ip":["172.16.99.192"],"X-Request-Id":["647fcb90a90eb593"],"X-Scheme":["http"]} expected_host="mail.aarr.xyz"
I0516 00:56:06.072273       1 solver.go:104] "comparing token" logger="cert-manager.acmesolver" host="mail.aarr.xyz" path="/.well-known/acme-challenge/...redacted..." base_path="/.well-known/acme-challenge" token="...redacted..." headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/f3ffb86b9871ff89645ef"],"X-Forwarded-For":["172.16.99.192"],"X-Forwarded-Host":["mail.aarr.xyz"],"X-Forwarded-Port":["8080"],"X-Forwarded-Proto":["http"],"X-Forwarded-Scheme":["http"],"X-Real-Ip":["172.16.99.192"],"X-Request-Id":["647fcb0ceb593"],"X-Scheme":["http"]} expected_token="...redacted..."
I0516 00:56:06.072280       1 solver.go:112] "got successful challenge request, writing key" logger="cert-manager.acmesolver" host="mail.aarr.xyz" path="/.well-known/acme-challenge/...redacted..." base_path="/.well-known/acme-challenge" token="...redacted..." headers={"Accept-Encoding":["gzip"],"User-Agent":["cert-manager-challenges/v1.17.2 (linux/amd64) cert-manager/f3f871ff89645ef"],"X-Forwarded-For":["172.16.99.192"],"X-Forwarded-Host":["mail.aarr.xyz"],"X-Forwarded-Port":["8080"],"X-Forwarded-Proto":["http"],"X-Forwarded-Scheme":["http"],"X-Real-Ip":["172.16.99.192"],"X-Request-Id":["647fcb0e93"],"X-Scheme":["http"]}
15

starting to look through cert-manager github issues ...

1 Like
16

its definitely the cert-manager challenge failing the self-check as you say, just no idea why ... i can ssh into the worker node and do a curl and it works

17

image
image1263×436 29.7 KB

18

I redeployed the ClusterIssuer to using the 'cert-manager' namespace and things started working. Given that its a Cluster resource, not sure why the namespace would matter, yet here we are ...

2 Likes
19

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.