HTTPS encryption is not active for my domain. My Order certificates is not completed


#1

I am working with cert-manager in my kubernetes cluster, in order to get certificates signed by let’sencrypt CA to my service application inside my cluster.

I am performing the following steps in the order presented. I’ve wanted provide the most details as a possible of my process in order to understand the behavior presented.

Install the CustomResourceDefinition resources separately

⟩ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
[I] 

Label the cert-manager namespace to disable resource validation

⟩ kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
namespace/kube-system labeled
[I] 

Install the cert-manager Helm chart

⟩ helm install \
    --name cert-manager \
    --namespace kube-system \
    --version v0.7.0 \
    jetstack/cert-manager
  • I’ve confirmed the steps in this guide, in order to avoid possible problems, and all steps are ok …

Creating my ingress

⟩ kubectl get pod,svc,deploy,replicaset -n kong | grep kong-ingress-controller
pod/kong-ingress-controller-667b4748d4-ccj8z   2/2     Running   14         95m
service/kong-ingress-controller   NodePort       10.0.48.131   <none>          8001:32257/TCP               3d19h

deployment.extensions/kong-ingress-controller   1         1         1            1           3d19h

replicaset.extensions/kong-ingress-controller-667b4748d4   1         1         1       3d19h

This means that my external IP addres is given by kong-proxy and is 52.166.60.158

⟩ kubectl get svc -n kong | grep kong-proxy
kong-proxy                LoadBalancer   10.0.153.8    52.166.60.158   80:31577/TCP,443:32323/TCP   3d21h
  • I’ve created the ingress for first time of this way:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kong-ingress-zcrm365
  # namespace: default
  annotations:
    # kubernetes.io/ingress.class: "nginx" # Don't include it in order to use kong-ingress-controller
    # add an annotation indicating the issuer to use.
spec:
  rules: 
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
        - path: "/"
          backend:
            serviceName: zcrm365dev
            servicePort: 80
      #- backend:
      #    serviceName: zcrm365dev
      #    servicePort: 80
      #  path: /
        
  tls: # < placing a host in the TLS config will indicate a cert should be created
  - hosts: 
    - test1kongletsencrypt.possibilit.nl
    secretName:  cert-manager-webhook-webhook-tls 
    # for the moment I've included this secret which was created by cert-manager installation
  • Apply it.
⟩ kubectl apply -f 03-zcrm365-ingress.yaml 
ingress.extensions/kong-ingress-zcrm365 created
[I] 
  • And my ingress is taking the kong-ingress-controller
⟩ kubectl describe ingress  kong-ingress-zcrm365 
Name:             kong-ingress-zcrm365
Namespace:        default
Address:          52.166.60.158
Default backend:  default-http-backend:80 (<none>)
TLS:
  cert-manager-webhook-webhook-tls terminates test1kongletsencrypt.possibilit.nl
Rules:
  Host                                Path  Backends
  ----                                ----  --------
  test1kongletsencrypt.possibilit.nl  
                                      /   zcrm365dev:80 (<none>)
Annotations:
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"cert-manager-webhook-webhook-tls"}]}}

Events:
  Type    Reason  Age    From                     Message
  ----    ------  ----   ----                     -------
  Normal  CREATE  3m30s  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
  Normal  UPDATE  3m28s  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
[I] 

Creating a ClusterIssuer

  • I am going to create a ClusterIssuer, I’d can create a Issuer, but I’ve started with a ClusterIssuer. What is the best alternative? This depen of our deployment and requirements to future, mostly in the namespace situations
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
 name: letsencrypt-staging
spec:
 acme:
   # The ACME server URL
   server: https://acme-staging-v02.api.letsencrypt.org/directory
   # Email address used for ACME registration
   email: b.garcia@possibilit.nl
   # Name of a secret used to store the ACME account private key
   privateKeySecretRef:
     name: letsencrypt-staging
   # Enable the HTTP-01 challenge provider
   http01: {}
  • Apply it
⟩ kubectl apply -f 01-lets-encrypt-issuer-staging.yaml 
clusterissuer.certmanager.k8s.io/letsencrypt-staging created
[I] 
  • This ClusterIssuer was registered on ACME letsencrypt server
⟩ kubectl describe clusterissuers letsencrypt-staging
Name:         letsencrypt-staging
Namespace:    
Labels:       <none>
Annotations:  kubectl.kubernetes.io/last-applied-configuration:
                {"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-staging"},"spec":{"acm...
API Version:  certmanager.k8s.io/v1alpha1
Kind:         ClusterIssuer
Metadata:
  Creation Timestamp:  2019-03-15T11:38:03Z
  Generation:          1
  Resource Version:    623999
  Self Link:           /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-staging
  UID:                 cb48b391-4716-11e9-a113-e27267a7d354
Spec:
  Acme:
    Email:  b.garcia@possibilit.nl
    Http 01:
    Private Key Secret Ref:
      Name:  letsencrypt-staging
    Server:  https://acme-staging-v02.api.letsencrypt.org/directory
Status:
  Acme:
    Uri:  https://acme-staging-v02.api.letsencrypt.org/acme/acct/8579841
  Conditions:
    Last Transition Time:  2019-03-15T11:38:05Z
    Message:               The ACME account was registered with the ACME server
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>
[I] 

Modifying my ingress resource created previously

  • Now that I’ve created our Let’s Encrypt staging ClusterIssuer, I am ready to modify the Ingress Resource we created above and enable TLS encryption for the test1kongletsencrypt.possibilit.nl paths adding the following

I am going to add certmanager.k8s.io/cluster-issuer: letsencrypt-staging annotation and use the secret created with the letsencrypt-staging ClusterIssuer named letsencrypt-staging

Our ingress has been stayed of this way:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: kong-ingress-zcrm365
  #namespace: default
  annotations:
    # kubernetes.io/ingress.class: "nginx" #new
    # certmanager.k8s.io/acme-challenge-type: http01

    # add an annotation indicating the issuer to use.
    certmanager.k8s.io/cluster-issuer: letsencrypt-staging
        
spec:
  rules: 
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
        - path: "/"
          backend:
            serviceName: zcrm365dev
            servicePort: 80    
  tls: 
  - hosts: 
    - test1kongletsencrypt.possibilit.nl
    secretName:  letsencrypt-staging # I've added this secret of letsencrypt cluster issuer
  • Apply it
⟩ kubectl apply -f 03-zcrm365-ingress.yaml 
ingress.extensions/kong-ingress-zcrm365 configured
[I]

This process update on the ingress, create one ingres named cm-acme-http-solver-jr4fg

⟩ kubectl get ingress 
NAME                        HOSTS                                ADDRESS         PORTS     AGE
cm-acme-http-solver-jr4fg   test1kongletsencrypt.possibilit.nl                   80        33s
kong-ingress-zcrm365        test1kongletsencrypt.possibilit.nl   52.166.60.158   80, 443   56m
[I] 

The detail of cm-acme-http-solver-jr4fg ingress is:

⟩ kubectl get ingress cm-acme-http-solver-jr4fg -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
  creationTimestamp: "2019-03-15T12:10:57Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    certmanager.k8s.io/acme-http-domain: "4095675862"
    certmanager.k8s.io/acme-http-token: "657526223"
  name: cm-acme-http-solver-jr4fg
  namespace: default
  ownerReferences:
  - apiVersion: certmanager.k8s.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: letsencrypt-staging-2613163196-0
    uid: 638f1701-471b-11e9-a113-e27267a7d354
  resourceVersion: "628284"
  selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/cm-acme-http-solver-jr4fg
  uid: 640ef483-471b-11e9-a113-e27267a7d354
spec:
  rules:
  - host: test1kongletsencrypt.possibilit.nl
    http:
      paths:
      - backend:
          serviceName: cm-acme-http-solver-svmvw
          servicePort: 8089
        path: /.well-known/acme-challenge/W7-9-KuPao_jg6EF5E2FXitFs8shOEsY5PlT9EEvNxE
status:
  loadBalancer:
    ingress:
    - ip: 52.166.60.158 
  • And the detail of our kong-ingress-zcrm365 resource ingress is:
⟩ kubectl describe  ingress kong-ingress-zcrm365
Name:             kong-ingress-zcrm365
Namespace:        default
Address:          52.166.60.158
Default backend:  default-http-backend:80 (<none>)
TLS:
  letsencrypt-staging terminates test1kongletsencrypt.possibilit.nl
Rules:
  Host                                Path  Backends
  ----                                ----  --------
  test1kongletsencrypt.possibilit.nl  
                                      /   zcrm365dev:80 (<none>)
Annotations:
  certmanager.k8s.io/cluster-issuer:                 letsencrypt-staging
  kubectl.kubernetes.io/last-applied-configuration:  {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/cluster-issuer":"letsencrypt-staging"},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"letsencrypt-staging"}]}}

Events:
  Type    Reason             Age                  From                     Message
  ----    ------             ----                 ----                     -------
  Normal  CREATE             60m                  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
  Normal  UPDATE             4m25s (x2 over 60m)  kong-ingress-controller  Ingress default/kong-ingress-zcrm365
  Normal  CreateCertificate  4m25s                cert-manager             Successfully created Certificate "letsencrypt-staging"
[I] 

We can see that our ingress even using the kong-ingress-controller and the letsencrypt-staging certificate has been created in the default namespace:

⟩ kubectl get certificates 
NAME
letsencrypt-staging
[I]

The letsencypt-staging certificate have the following detail

⟩ kubectl describe certificate letsencrypt-staging
Name:         letsencrypt-staging
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  certmanager.k8s.io/v1alpha1
Kind:         Certificate
Metadata:
  Creation Timestamp:  2019-03-15T12:10:55Z
  Generation:          1
  Owner References:
    API Version:           extensions/v1beta1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Ingress
    Name:                  kong-ingress-zcrm365
    UID:                   8643558f-4713-11e9-a113-e27267a7d354
  Resource Version:        628164
  Self Link:               /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/letsencrypt-staging
  UID:                     62b3a31e-471b-11e9-a113-e27267a7d354
Spec:
  Acme:
    Config:
      Domains:
        test1kongletsencrypt.possibilit.nl
      Http 01:
  Dns Names:
    test1kongletsencrypt.possibilit.nl
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-staging
  Secret Name:  letsencrypt-staging
Status:
  Conditions:
    Last Transition Time:  2019-03-15T12:10:55Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age    From          Message
  ----    ------              ----   ----          -------
  Normal  Generated           7m24s  cert-manager  Generated new private key
  Normal  GenerateSelfSigned  7m24s  cert-manager  Generated temporary self signed certificate
  Normal  OrderCreated        7m23s  cert-manager  Created Order resource "letsencrypt-staging-2613163196"
[I] 
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)

I can see that my order issue is not completed, only was created in the OrderCreated event, and this order already have 7 minutes since I’ve created this certificate and the order was not completed and by that reason the certificate is not issued successfully

Another thing that happens to me, is that the letsencrypt-staging secret created by the letsencrypt-staging cluster Issuer and their respective certificate, only have the tls.key:

⟩ kubectl describe secrets letsencrypt-staging -n kube-system
Name:         letsencrypt-staging
Namespace:    kube-system
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
tls.key:  1675 bytes
[I] 

According to I understand, is that if the letsencrypt certificate complete the order and the certificate is issued, in the letsencrypt-staging secret I would have one tls.crt key and maybe my letsencrypt-staging will be of tls type and not Opaque?

  • When I see the logs of my cert-manager pod I get the following output, I think that the http challenge is not executed:
I0315 12:10:56.109919       1 controller.go:168] certificates controller: Finished processing work item "default/letsencrypt-staging"
I0315 12:10:56.910004       1 logger.go:73] Calling GetAuthorization
I0315 12:10:57.135213       1 logger.go:93] Calling HTTP01ChallengeResponse
I0315 12:10:57.150924       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.151110       1 controller.go:162] certificates controller: syncing item 'default/letsencrypt-staging'
I0315 12:10:57.151113       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.151521       1 sync.go:274] Need to create 1 challenges
I0315 12:10:57.151891       1 issue.go:160] Order default/letsencrypt-staging-2613163196 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0315 12:10:57.152022       1 controller.go:168] certificates controller: Finished processing work item "default/letsencrypt-staging"
I0315 12:10:57.160718       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.160804       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.160870       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.160739       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.161051       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:57.161214       1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.161263       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.161287       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.562383       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.562738       1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.562755       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.562906       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.563018       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.563231       1 logger.go:68] Calling GetChallenge
I0315 12:10:57.833421       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:57.833648       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.833755       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.833820       1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.833828       1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.833838       1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.833858       1 logger.go:103] Calling Discover
I0315 12:10:57.856136       1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.923080       1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.989596       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:57.989682       1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:58.014803       1 controller.go:178] ingress-shim controller: syncing item 'default/cm-acme-http-solver-jr4fg'
I0315 12:10:58.014842       1 sync.go:64] Not syncing ingress default/cm-acme-http-solver-jr4fg as it does not contain necessary annotations
I0315 12:10:58.014846       1 controller.go:184] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-jr4fg"
I0315 12:10:58.015447       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:58.033431       1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 12:10:58.079504       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:58.079616       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:58.079569       1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'

I get this message No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0"
According to this, I decide add the certmanager.k8s.io/acme-challenge-type: http01 annotation to my kong-ingress-zcrm365 ingress but nothing happened … my ingress is updated, but nothing more.

All this process confirms that the TLS certificate was not successfully issued and HTTPS encryption is not active for my domains test1kongletsencrypt.possibilit.nl configured.

This make that my letsencrypt-staging certificate have a Status:False, and the order created event does not advance to completed to be issued.



  Conditions:
    Last Transition Time:  2019-03-15T12:10:55Z
    Message:               Certificate issuance in progress. Temporary certificate issued.
    Reason:                TemporaryCertificate
    Status:                False
    Type:                  Ready
Events:
  Type    Reason              Age                From          Message
  ----    ------              ----               ----          -------
  Normal  Generated           51m                cert-manager  Generated new private key
  Normal  GenerateSelfSigned  51m                cert-manager  Generated temporary self signed certificate
  Normal  Cleanup             5m42s              cert-manager  Deleting old Order resource "letsencrypt-staging-2613163196"
  Normal  OrderCreated        5m42s              cert-manager  Created Order resource "letsencrypt-staging-2965106631"
  Normal  OrderCreated        39s (x2 over 51m)  cert-manager  Created Order resource "letsencrypt-staging-2613163196"
  Normal  Cleanup             39s                cert-manager  Deleting old Order resource "letsencrypt-staging-2965106631"
[I] 
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)

How to can I to my certificate to be signed and succesfully issued by letsencrypt CA and active the https encryption active?
What is happening with these logs messages?

kubectl logs -n kube-system cert-manager-6f68b58796-q7txg

0315 13:06:11.027204       1 logger.go:103] Calling Discover
I0315 13:06:11.032299       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 13:06:11.046081       1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 13:06:11.046109       1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 13:06:21.046242       1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'

I’ve heared that letsencrypt-staging environment only have test certificates and these are a kind of ‘fake certificates’ and maybe some clients like my chrome/firefox browser doesn’t trust certificate issuer …

Is this a reason to I cannot enable https encryption on my domain?
In affirmative case, should I change from staging environment to production environment?

In this question some people talk about that but they emphasize:

that the staging environment should be used just to test that your client is working fine and can generate the challenges, certificates

In my case the http challenge is not generated still in staging environment. :frowning:


#2

Hi @bgarcial,

You might have quicker results asking your question in the Slack channel Jetstack mentions in the cert-manager README: https://github.com/jetstack/cert-manager#troubleshooting

The quickest way to ask a question is to first post on our Slack channel (#cert-manager) on the Kubernetes Slack. There are a lot of community members in this channel, and you can often get an answer to your question straight away!


#3

Hi @cpu Yes, I understand, I did want to put the question here because it’s related with the letsencrypt process too. I was thinking that maybe someone wants support me. Of anyway I understand that you says.


#4

No problem! It’s totally OK to post here for support as well :slight_smile: I just wanted to make sure you also knew about the Slack option.


#5

Thanks, I’ve shared also my question there.


#6

I have been exploring some cert-manager issues and I found this https://github.com/Kong/kubernetes-ingress-controller/issues/162 and https://github.com/jetstack/cert-manager/issues/958 but I am not sure if maybe could to be related to my situation.
Is possible that kong and cert-manager does not work together?


#7

I’ve applied this acme-kong-kube-helper github.com/ollystephens/acme-kong-kube-helper in order to solve the http01 validation problem in staging environment, and all it’s works. When the http01 validation to be performed, is necessary use the letsencrypt production environment to get the https encryption

The problem is mainly that cert-manager have some problems to work with other ingress controller different to nginx. This helper is a temporal solution. Currently cert-manager work in this feature to solve this https://github.com/jetstack/cert-manager/issues/1097