Certbot cannot renew in k8s

Help
1

Hi all,
I have K8S cluster bare-metal , installing by kubeadm , with metallb, ingress-nginx.
This is my web app ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: web-ingress
  namespace: web
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: letsencrypt-production
    cert-manager.io/acme-challenge-type: http01
    acme.cert-manager.io/http01-edit-in-place: "true"
    nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
spec:
  tls:
  - hosts:
      - web.com.vn
    secretName: web-tls
  rules:
  - host: web.com.vn
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: web-svc
            port:
              number: 80

It used to worked fine, I had let's encrypt cert for my web app 2 months ago.
Today I receive an email about my web app cert will expire in 19 days, so I start to troubleshoot why certbot cannot renew my web app cert.

kubectl -n web get pod,secret,certificaterequest,challenge,certificate
NAME                                   READY   STATUS    RESTARTS      AGE
pod/cm-acme-http-solver-6jwlx          1/1     Running   0             39m
pod/wev-server-d69f9b46f-hz4cg   1/1     Running   0             39m

NAME                         TYPE                DATA   AGE
secret/web-tls         kubernetes.io/tls   2      70d
secret/web-tls-qrnxc   Opaque              1      39m

NAME                                                     APPROVED   DENIED   READY   ISSUER                   REQUESTOR                                         AGE
certificaterequest.cert-manager.io/web-tls-rc4md   True                False   letsencrypt-production   system:serviceaccount:cert-manager:cert-manager   39m
certificaterequest.cert-manager.io/web-tls-rpdks   True                True    letsencrypt-production   system:serviceaccount:cert-manager:cert-manager   70d

NAME                                                                       STATE     DOMAIN             AGE
challenge.acme.cert-manager.io/itlglobal-tls-rc4md-1655542073-1501607821   pending   web.com.vn   39m

NAME                                        READY   SECRET          AGE
certificate.cert-manager.io/web-tls   True    web-tls   70d

kubectl -n web describe challenge.acme.cert-manager.io/web-tls-rc4md-1655542073-1501607821
Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64': Get "http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
  State:       pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    57m   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  57m   cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

From my computer I can

curl http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64
38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64.vRQoD0SeZx3qzzYPIuWT-TgbcWS7yxL0oaXoIA9qApY

Please give me some advice, thank you very much.

2

But, from the public internet that domain cannot be reached. The Let's Debug test site is helpful to test connections

5 Likes
3

I can't connect either:

curl http://web.com.vn/.well-known/acme-challenge/38G9DdwdnaLlLLuDUFtAc9_ZHI9saEVdyDEU0J2BG64
curl: (56) Recv failure: Connection reset by peer
3 Likes
4

Actually I changed my web app domain , it isn't web.com.vn
After delete and create all resources (deployment,service,ingress,secret...) many times, my web app can get let's encrypt cert successfully.
Maybe because my web app is too old (nginx+php5.3)
Other (newer) apps in my k8s work just fine with let's encrypt + nginx ingress.

5

Are you sure that ./well-known/acme-challenge requests to that domain are being directed to the cert-manager http challenge solver? Otherwise they may be trying to reach the web application itself which doesn't know the right response to provide.

2 Likes
6

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.