Certbot cannot renew in k8s

Hi all,
I have K8S cluster bare-metal , installing by kubeadm , with metallb, ingress-nginx.
This is my web app ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
  name: web-ingress
  namespace: web
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt-production
    cert-manager.io/acme-challenge-type: http01
    acme.cert-manager.io/http01-edit-in-place: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
  - hosts:
      - web.com.vn
    secretName: web-tls
  - host: web.com.vn
      - path: /
        pathType: Prefix
            name: web-svc
              number: 80

It used to worked fine, I had let's encrypt cert for my web app 2 months ago.
Today I receive an email about my web app cert will expire in 19 days, so I start to troubleshoot why certbot cannot renew my web app cert.

kubectl -n web get pod,secret,certificaterequest,challenge,certificate
NAME                                   READY   STATUS    RESTARTS      AGE
pod/cm-acme-http-solver-6jwlx          1/1     Running   0             39m
pod/wev-server-d69f9b46f-hz4cg   1/1     Running   0             39m

NAME                         TYPE                DATA   AGE
secret/web-tls         kubernetes.io/tls   2      70d
secret/web-tls-qrnxc   Opaque              1      39m

NAME                                                     APPROVED   DENIED   READY   ISSUER                   REQUESTOR                                         AGE
certificaterequest.cert-manager.io/web-tls-rc4md   True                False   letsencrypt-production   system:serviceaccount:cert-manager:cert-manager   39m
certificaterequest.cert-manager.io/web-tls-rpdks   True                True    letsencrypt-production   system:serviceaccount:cert-manager:cert-manager   70d

NAME                                                                       STATE     DOMAIN             AGE
challenge.acme.cert-manager.io/itlglobal-tls-rc4md-1655542073-1501607821   pending   web.com.vn   39m

NAME                                        READY   SECRET          AGE
certificate.cert-manager.io/web-tls   True    web-tls   70d

kubectl -n web describe challenge.acme.cert-manager.io/web-tls-rc4md-1655542073-1501607821
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64': Get "http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  State:       pending
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    57m   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  57m   cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

From my computer I can

curl http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64

Please give me some advice, thank you very much.

But, from the public internet that domain cannot be reached. The Let's Debug test site is helpful to test connections


I can't connect either:

curl http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64
curl: (56) Recv failure: Connection reset by peer

Actually I changed my web app domain , it isn't web.com.vn
After delete and create all resources (deployment,service,ingress,secret...) many times, my web app can get let's encrypt cert successfully.
Maybe because my web app is too old (nginx+php5.3)
Other (newer) apps in my k8s work just fine with let's encrypt + nginx ingress.

Are you sure that ./well-known/acme-challenge requests to that domain are being directed to the cert-manager http challenge solver? Otherwise they may be trying to reach the web application itself which doesn't know the right response to provide.


This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.