Cert-manager: Unable to get certificate for domain even the issuer is correctly working

Help
1

Hello Team,

TLS certificate is not coming from Let's encrypt even the issuer is correctly working as below and certificates status shows in false state.
I am pasting the output of certificaterequest please help to get that certificate for our domain

k get issuer
NAME READY AGE
letsencrypt-kc-prod True 29h
letsencrypt-key-cloak-staging True 25m

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-key-cloak-staging
namespace: default
spec:
acme:
email: "xyz@abc.com"
preferredChain: ""
privateKeySecretRef:
name: letsencrypt-key-cloak-staging
server: https://acme-staging-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: nginx

My domain is: idppreviewkc.centralus.cloudapp.azure.com

I ran this command:

It produced this output:
kubectl describe certificaterequest letsencrypt-key-cloak-tls

Status:
Conditions:
Last Transition Time: 2023-04-04T11:43:17Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2023-04-04T11:43:17Z
Message: Waiting on certificate issuance from order default/letsencrypt-key-cloak-tls-j8bkd-483700965: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message

Normal cert-manager.io 10m cert-manager-certificaterequests-approver Certificate request has been approved by cert-manager.io
Normal OrderCreated 10m cert-manager-certificaterequests-issuer-acme Created Order resource default/letsencrypt-key-cloak-tls-j8bkd-483700965
Normal OrderPending 10m cert-manager-certificaterequests-issuer-acme Waiting on certificate issuance from order default/letsencrypt-key-cloak-tls-j8bkd-483700965: ""

k get secrets | grep -i letsencrypt-key
letsencrypt-key-cloak-staging Opaque 1 237d
letsencrypt-key-cloak-tls Opaque 1 237d
letsencrypt-key-cloak-tls-qflh5 Opaque 1 34s

kubectl describe certificaterequest letsencrypt-key-cloak-tls

Status:
Conditions:
Last Transition Time: 2023-04-04T11:43:17Z
Message: Certificate request has been approved by cert-manager.io
Reason: cert-manager.io
Status: True
Type: Approved
Last Transition Time: 2023-04-04T11:43:17Z
Message: Waiting on certificate issuance from order default/letsencrypt-key-cloak-tls-j8bkd-483700965: "pending"
Reason: Pending
Status: False
Type: Ready
Events:
Type Reason Age From Message

Normal cert-manager.io 10m cert-manager-certificaterequests-approver Certificate request has been approved by cert-manager.io
Normal OrderCreated 10m cert-manager-certificaterequests-issuer-acme Created Order resource default/letsencrypt-key-cloak-tls-j8bkd-483700965
Normal OrderPending 10m cert-manager-certificaterequests-issuer-acme Waiting on certificate issuance from order default/letsencrypt-key-cloak-tls-j8bkd-483700965: ""

kubectl describe certificate letsencrypt-key-cloak-tls

Status:
Conditions:
Last Transition Time: 2023-04-04T11:43:17Z
Message: Issuing certificate as Secret does not contain a certificate
Observed Generation: 1
Reason: MissingData
Status: True
Type: Issuing
Last Transition Time: 2023-04-04T11:43:17Z
Message: Issuing certificate as Secret does not contain a certificate
Observed Generation: 1
Reason: MissingData
Status: False
Type: Ready
Next Private Key Secret Name: letsencrypt-key-cloak-tls-qflh5
Events:
Type Reason Age From Message

Normal Issuing 11m cert-manager-certificates-trigger Issuing certificate as Secret does not contain a certificate
Normal Reused 11m cert-manager-certificates-key-manager Reusing private key stored in existing Secret resource "letsencrypt-key-cloak-tls"
Normal Requested 11m cert-manager-certificates-request-manager Created new CertificateRequest resource "letsencrypt-key-cloak-tls-j8bkd"

2

This is probably due to a networking issue with your system. I don't know your config well enough to help you diagnose that. You might try the below link while waiting to see if anyone else here will help.

Also here:

3 Likes
3

I would like to delete the given below secrets, but they auto-provision eveytime when I delete those secrets.
Is this due to cause from Lets encrypt end?

k get secrets | grep -i letsencrypt-key
letsencrypt-key-cloak-staging Opaque 1 237d
letsencrypt-key-cloak-tls Opaque 1 237d
letsencrypt-key-cloak-tls-qflh5 Opaque 1 34s

4

No. Let's Encrypt is an ACME Server. Your system is the ACME Client. The LE Server only responds to requests from the Client. The LE Servers cannot make changes to your system.

Something on your system must be doing this auto-provision

3 Likes
5

This issue comes when I delete the certificate also, it is also auto-provison again and again, even after removing the service account it is autoprovisioned.
even after deleting the issuer, the certificate is auto-provisioned.
any hints please appreciatable

k get certificates | grep -i letsencrypt
letsencrypt-key-cloak-tls False letsencrypt-key-cloak-tls 2m42s

6

Is there a cron type job running that automatically issues certs?

3 Likes
7

no, nothing like that

8

Then you should keep looking :eyes:

3 Likes
9

I have issued multiple certs in past, but this is first time I faced the issue where I delete certs. and it is auto -provision, even secrets also gets auto-provisioned.for the same domain

10

I would search for "staging" within all configs.

3 Likes
11

Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt-key-cloak-staging
Secret Name: letsencrypt-key-cloak-tls
Usages:
digital signature
key encipherment
Status:
Conditions:
Last Transition Time: 2023-04-04T15:19:40Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: True
Type: Issuing
Last Transition Time: 2023-04-04T15:19:40Z
Message: Issuing certificate as Secret does not exist
Observed Generation: 1
Reason: DoesNotExist
Status: False
Type: Ready
Next Private Key Secret Name: letsencrypt-key-cloak-tls-p78nw
Events:

12
2 Likes
13

k describe challenges letsencrypt-key-cloak-tls-9gchp-483700965-3867639369
Name: letsencrypt-key-cloak-tls-9gchp-483700965-3867639369
Namespace: default
Labels:
Annotations:
API Version: acme.cert-manager.io/v1
Kind: Challenge
Metadata:
Creation Timestamp: 2023-04-04T15:27:33Z
Finalizers:
finalizer.acme.cert-manager.io
Generation: 1
Managed Fields:
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:finalizers:
.:
v:"finalizer.acme.cert-manager.io":
Manager: cert-manager-challenges
Operation: Update
Time: 2023-04-04T15:27:33Z
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:metadata:
f:ownerReferences:
.:
k:{"uid":"4dcf66e4-a2f9-4c64-bc1a-3c033a243e28"}:
f:spec:
.:
f:authorizationURL:
f:dnsName:
f:issuerRef:
.:
f:group:
f:kind:
f:name:
f:key:
f:solver:
.:
f:http01:
.:
f:ingress:
.:
f:class:
f:token:
f:type:
f:url:
f:wildcard:
Manager: cert-manager-orders
Operation: Update
Time: 2023-04-04T15:27:33Z
API Version: acme.cert-manager.io/v1
Fields Type: FieldsV1
fieldsV1:
f:status:
.:
f:presented:
f:processing:
f:reason:
f:state:
Manager: cert-manager-challenges
Operation: Update
Subresource: status
Time: 2023-04-04T15:27:35Z
Owner References:
API Version: acme.cert-manager.io/v1
Block Owner Deletion: true
Controller: true
Kind: Order
Name: letsencrypt-key-cloak-tls-9gchp-483700965
UID: 4dcf66e4-a2f9-4c64-bc1a-3c033a243e28
Resource Version: 165144437
UID: e669cbe0-575d-4e4a-a1b5-ebfa7e18b6c2
Spec:
Authorization URL: https://acme-staging-v02.api.letsencrypt.org/acme/authz-v3/5998688304
Dns Name: idppreviewkc.centralus.cloudapp.azure.com
Issuer Ref:
Group: cert-manager.io
Kind: Issuer
Name: letsencrypt-key-cloak-staging
Key: qEhuVoQ4dE3sL5NzQ4GrIgYVXIPpLmMU3SnTmGXpv4c.xelk_G1FnYvQ82UZ6qLYFz9NbQaeVAqIhmH4-493MKI
Solver:
http01:
Ingress:
Class: nginx
Token: qEhuVoQ4dE3sL5NzQ4GrIgYVXIPpLmMU3SnTmGXpv4c
Type: HTTP-01
URL: https://acme-staging-v02.api.letsencrypt.org/acme/chall-v3/5998688304/zgG5Ig
Wildcard: false
Status:
Presented: true
Processing: true
Reason: Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
State: pending
Events:

14

That will not produce a "real" cert.

2 Likes
15

I checked about

Got 404 status code

If your challenge self-check fails with a 404 not found error. Make sure to check the following:

  • you can access the URL from the public internet
  • the ACME solver pod is up and running
  • use kubectl describe ingress to check the status of the HTTP01 solver ingress. (unless you use acme.cert-manager.io/http01-edit-in-place, then check the same ingress as your domain)

k get pods
NAME READY STATUS RESTARTS AGE
cm-acme-http-solver-ghpcn 1/1 Running 0 68m

k get ingress
NAME CLASS HOSTS ADDRESS PORTS AGE
cm-acme-http-solver-ljrsf idppreviewkc.centralus.cloudapp.azure.com 80 69m

All things mentioned is working infact the domain is publicly accessible.

16

k get certificates letsencrypt-key-cloak-tls -o yaml

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  creationTimestamp: "2023-04-04T15:19:40Z"
  generation: 1
  name: letsencrypt-key-cloak-tls
  namespace: default
  ownerReferences:
  - apiVersion: networking.k8s.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Ingress
    name: keycloak-service-ingress
    uid: 55135089-7600-45d0-8a7b-126c0afa6346
  resourceVersion: "165141185"
  uid: f9bb8b2a-8c56-4e76-9625-fffe695f8df6
spec:
  dnsNames:
  - idppreviewkc.centralus.cloudapp.azure.com
  issuerRef:
    group: cert-manager.io
    kind: Issuer
    name: letsencrypt-key-cloak-staging
  secretName: letsencrypt-key-cloak-tls
  usages:
  - digital signature
  - key encipherment
status:
  conditions:
  - lastTransitionTime: "2023-04-04T15:19:40Z"
    message: Issuing certificate as Secret does not exist
    observedGeneration: 1
    reason: DoesNotExist
    status: "True"
    type: Issuing
  - lastTransitionTime: "2023-04-04T15:19:40Z"
    message: Issuing certificate as Secret does not exist
    observedGeneration: 1
    reason: DoesNotExist
    status: "False"
    type: Ready
  nextPrivateKeySecretName: letsencrypt-key-cloak-tls-p78nw

After checking the Yaml file for certificate, I found letsencrypt-key-cloak-tls is created in below format

k get secrets | grep -i letsencrypt-key

letsencrypt-key-cloak-staging                                Opaque                                1      94m
letsencrypt-key-cloak-tls-p78nw                              Opaque                                1      85m

can you please suggest anything wrong here,

17

I don't get the "404" error.
Because it seems like you can get a staging cert.

2 Likes
18

Shall i have to mention letsencrypt-kc-prod in certificate yml file?

cat kc_prod_issuer.yml
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
annotations:
name: letsencrypt-kc-prod
namespace: default
spec:
acme:
email: ""
preferredChain: ""
privateKeySecretRef:
name: letsencrypt-prodkc-preview2
server: https://acme-v02.api.letsencrypt.org/directory
solvers:
- http01:
ingress:
class: nginx

19

Could you please suggest?

20

Unfortunately, I'm not familiar with cert-manager.

2 Likes