Issues with getting a certificate in kubernetes

iferencik · October 9, 2022, 11:58pm

Hello there,

I am trying to set up a kube clusterissuer using cert-manager and ingress.
My subdomain is: titiler-dev.undpgeohub.org

kubectl get certificate -n geodemo
NAME READY SECRET AGE
geodemo-cert False geodemo-cert 41m

Status:
Conditions:
Last Transition Time: 2022-10-09T23:09:37Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: True
Type: Issuing
Last Transition Time: 2022-10-09T23:09:37Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Next Private Key Secret Name: geodemo-cert-w9zpm
Events:
Type Reason Age From Message

Normal Issuing 42m cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 42m cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "geodemo-cert-w9zpm"
Normal Requested 42m cert-manager-certificates-request-manager Created new CertificateRequest resource "geodemo-cert-b48kq"

and into the request

kubectl describe certificaterequest geodemo-cert-b48kq -n geodemo

Status:
Conditions:
Last Transition Time: 2022-10-09T23:09:37Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2022-10-09T23:09:38Z
Message: Waiting on certificate issuance from order geodemo/geodemo-cert-b48kq-3670509394: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message

Normal cert-manager.io 44m cert-manager-certificaterequests-approver Certificate request has been approved by cert-manager.io
Normal OrderCreated 44m cert-manager-certificaterequests-issuer-acme Created Order resource geodemo/geodemo-cert-b48kq-3670509394

next i looked at the order

kubectl describe order geodemo-cert-b48kq-3670509394 -n geodemo

Status:
Authorizations:
Challenges:
Token: EhxmKzrsFKrGgIoGx39VnNjMvXZWosyGX8cOIrCB3Oc
Type: http-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/162759015656/QIkh8Q
Token: EhxmKzrsFKrGgIoGx39VnNjMvXZWosyGX8cOIrCB3Oc
Type: dns-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/162759015656/GHr9LA
Token: EhxmKzrsFKrGgIoGx39VnNjMvXZWosyGX8cOIrCB3Oc
Type: tls-alpn-01
URL: https://acme-v02.api.letsencrypt.org/acme/chall-v3/162759015656/mAnf-g
Identifier: titiler-dev.undpgeohub.org
Initial State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/authz-v3/162759015656
Wildcard: false
Finalize URL: https://acme-v02.api.letsencrypt.org/acme/finalize/768432736/133100419896
State: pending
URL: https://acme-v02.api.letsencrypt.org/acme/order/768432736/133100419896
Events:

And this is where i stopped because I can not figure out what is happening.

at the same time the cert-manager pod logs said:

E1009 23:30:54.254553 1 controller.go:166] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="create not allowed while custom resource definition is terminating" "key"="geodemo/geodemo-cert-b48kq-3670509394"
E1009 23:41:29.765609 1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="[error reading (cluster)issuer "letsencrypt-staging": clusterissuer.cert-manager.io "letsencrypt-staging" not found, namespaces "dev" not found]" "key"="dev/tls-secret-lclvt-2994752350-2717540044"
E1009 23:52:14.263448 1 controller.go:166] cert-manager/orders "msg"="re-queuing item due to error processing" "error"="create not allowed while custom resource definition is terminating" "key"="geodemo/geodemo-cert-b48kq-3670509394"

My cluster-issuer is letsencrypt

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
name: letsencrypt
spec:
acme:
server: https://acme-v02.api.letsencrypt.org/directory
email: xxx@yyy
privateKeySecretRef:
name: letsencrypt
solvers:
- dns01:
cloudflare:
email: xxx@yyy
apiTokenSecretRef:
name: cloudflare-api-token-secret
key: api-token
selector:
dnsZones:
- 'undpgeohub.org'
- '*.undpgeohub.org'

so it makes no sense for the cert-manegr to complain about letsencrypt-staging

The cluster is deployed in AKS

Any hints are appreciated

griffin · October 10, 2022, 4:25am

Welcome to the Let's Encrypt Community!

I have been immersed in this type of setup (AKS and ingress-nginx) for several months now at my company. You do not explicitly request a certificate with this setup. It occurs automatically based upon deployment of an ingress manifest. From what I can tell from your outputs, it appears that you might have conflicting installations/configurations. Your best bet is usually to uninstall your cert-manager and ingress controller(s), fix your deployment configurations, then reinstall everything. I highly recommend reviewing the documentation below.

iferencik · October 10, 2022, 11:11am

Hi @griffin and thanks so much for replying.

I am aware the certificate is requested automatically by the ingress manifest:
this is the one I use

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: geodemo-ssl-tls-ingress
  namespace: geodemo
  annotations:
    kubernetes.io/ingress.class: nginx
    #nginx.ingress.kubernetes.io/force-ssl-redirect: "true"
    cert-manager.io/cluster-issuer: letsencrypt
    
spec:
  tls:
    - hosts:
      - titiler-dev.undpgeohub.org # update IP address here
      secretName: geodemo-cert
  rules:
  - host: titiler-dev.undpgeohub.org # update IP address here
    http:
      paths:
        - path: "/"
          pathType: Prefix
          backend:
            service:
              name: titiler
              port:
                number: 80

I actually used the link you sent. The truth is this is the first time in my life I am dealing with certificates
given that I have a geospatial background and I find it it quite complicated

I will delete everything and try from scratch.

I have on more question.

The cluster issuer uses cloudflare API token to solve the challenges. I stored the token as a secret.
Does the Token need to be a base64 string?

solvers:
      - dns01:
          cloudflare:
            email: ioan.ferencik@undp.org
            apiTokenSecretRef:
              name: cloudflare-api-token-secret
              key: api-token

and the secret

apiVersion: v1
kind: Secret
metadata:
  name: cloudflare-api-token-secret
  namespace: geodemo
type: Opaque
stringData:
  api-token: API_TOKEN_VALUE

my ingress does not get an address, and I believe it should

kubectl get ingress -n geodemo
NAME                      CLASS    HOSTS                        ADDRESS   PORTS     AGE
geodemo-ssl-tls-ingress   <none>   titiler-dev.undpgeohub.org             80, 443   1s

griffin · October 10, 2022, 3:19pm

You should get your ingress working (i.e. accessible from wherever you want access) before attempting to acquire a certificate (and before installing cert-manager at all for that matter). Typically a load balancer is provisioned during the ingress-nginx deployment process with an IP address that is connected one step further up in your network toward your public ingress point (e.g. a private IP address with a route from your corporate firewall or an Azure public IP address). That load balancer connects to your ingress-nginx replica instances (usually two, sometimes more depending upon your needs). There are many online examples of how to configure this (though I'm unsure of the exact addresses at the moment). They typically involve passing a yaml file to the ingress-nginx installer (via helm) that contains a configuration specifying the IP address of the load balancer to be deployed.

iferencik · October 10, 2022, 5:21pm

Hello @griffin

thanks again for you guidance. I looked into my ingress-controller and it is an Azure flavor of nginx.

The whole issue was that in my ingress resource I was using examples from internet. The ingress pod logs were complaining about the wrong ingress class. I then found out my AKS cluster uses a different class addon-http-application-routing not nginx

    #kubernetes.io/ingress.class: nginx
    kubernetes.io/ingress.class: addon-http-application-routing

once I updated the annotation I was able to get my ingress running.

I will look now into the cert-manager as the ingress is working.

Thanks for your advice.

system · November 9, 2022, 5:21pm

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Issue with certificate using cert-manager Kubernetes Help	3	2003	April 29, 2020
Cert-Manager, Letsencrypt in Kubernetes not giving me a Certification Help	2	4716	April 28, 2023
Unable to create certificate for Kubernetes web-app. cert-manager/challenges: propagation check failed" err="wrong status code '503', expected '200'" Help	8	1637	July 29, 2023
Certificate renewal using cert-manager on kubernetes cluster Help	3	4064	January 22, 2020
Problem with cert-manager and first time install on kubernetes cluster Help	44	5400	February 23, 2023

Issues with getting a certificate in kubernetes

Related topics