I am working with cert-manager in my kubernetes cluster, in order to get certificates signed by let'sencrypt CA to my service application inside my cluster.
I am performing the following steps in the order presented. I've wanted provide the most details as a possible of my process in order to understand the behavior presented.
Install the CustomResourceDefinition resources separately
⟩ kubectl apply -f https://raw.githubusercontent.com/jetstack/cert-manager/release-0.7/deploy/manifests/00-crds.yaml
customresourcedefinition.apiextensions.k8s.io/certificates.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/challenges.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/clusterissuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/issuers.certmanager.k8s.io created
customresourcedefinition.apiextensions.k8s.io/orders.certmanager.k8s.io created
[I]
Label the cert-manager namespace to disable resource validation
⟩ kubectl label namespace kube-system certmanager.k8s.io/disable-validation=true
namespace/kube-system labeled
[I]
Install the cert-manager Helm chart
⟩ helm install \
--name cert-manager \
--namespace kube-system \
--version v0.7.0 \
jetstack/cert-manager
- I've confirmed the steps in this guide, in order to avoid possible problems, and all steps are ok ...
Creating my ingress
- I am using kong-ingress-controller to manage the ingress process.
⟩ kubectl get pod,svc,deploy,replicaset -n kong | grep kong-ingress-controller
pod/kong-ingress-controller-667b4748d4-ccj8z 2/2 Running 14 95m
service/kong-ingress-controller NodePort 10.0.48.131 <none> 8001:32257/TCP 3d19h
deployment.extensions/kong-ingress-controller 1 1 1 1 3d19h
replicaset.extensions/kong-ingress-controller-667b4748d4 1 1 1 3d19h
This means that my external IP addres is given by kong-proxy and is 52.166.60.158
⟩ kubectl get svc -n kong | grep kong-proxy
kong-proxy LoadBalancer 10.0.153.8 52.166.60.158 80:31577/TCP,443:32323/TCP 3d21h
- I've created the ingress for first time of this way:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kong-ingress-zcrm365
# namespace: default
annotations:
# kubernetes.io/ingress.class: "nginx" # Don't include it in order to use kong-ingress-controller
# add an annotation indicating the issuer to use.
spec:
rules:
- host: test1kongletsencrypt.possibilit.nl
http:
paths:
- path: "/"
backend:
serviceName: zcrm365dev
servicePort: 80
#- backend:
# serviceName: zcrm365dev
# servicePort: 80
# path: /
tls: # < placing a host in the TLS config will indicate a cert should be created
- hosts:
- test1kongletsencrypt.possibilit.nl
secretName: cert-manager-webhook-webhook-tls
# for the moment I've included this secret which was created by cert-manager installation
- Apply it.
⟩ kubectl apply -f 03-zcrm365-ingress.yaml
ingress.extensions/kong-ingress-zcrm365 created
[I]
- And my ingress is taking the kong-ingress-controller
⟩ kubectl describe ingress kong-ingress-zcrm365
Name: kong-ingress-zcrm365
Namespace: default
Address: 52.166.60.158
Default backend: default-http-backend:80 (<none>)
TLS:
cert-manager-webhook-webhook-tls terminates test1kongletsencrypt.possibilit.nl
Rules:
Host Path Backends
---- ---- --------
test1kongletsencrypt.possibilit.nl
/ zcrm365dev:80 (<none>)
Annotations:
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"cert-manager-webhook-webhook-tls"}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 3m30s kong-ingress-controller Ingress default/kong-ingress-zcrm365
Normal UPDATE 3m28s kong-ingress-controller Ingress default/kong-ingress-zcrm365
[I]
Creating a ClusterIssuer
- I am going to create a
ClusterIssuer, I'd can create a Issuer, but I've started with a ClusterIssuer. What is the best alternative? This depen of our deployment and requirements to future, mostly in the namespace situations
apiVersion: certmanager.k8s.io/v1alpha1
kind: ClusterIssuer
metadata:
name: letsencrypt-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: b.garcia@possibilit.nl
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
http01: {}
- Apply it
⟩ kubectl apply -f 01-lets-encrypt-issuer-staging.yaml
clusterissuer.certmanager.k8s.io/letsencrypt-staging created
[I]
- This ClusterIssuer was registered on ACME letsencrypt server
⟩ kubectl describe clusterissuers letsencrypt-staging
Name: letsencrypt-staging
Namespace:
Labels: <none>
Annotations: kubectl.kubernetes.io/last-applied-configuration:
{"apiVersion":"certmanager.k8s.io/v1alpha1","kind":"ClusterIssuer","metadata":{"annotations":{},"name":"letsencrypt-staging"},"spec":{"acm...
API Version: certmanager.k8s.io/v1alpha1
Kind: ClusterIssuer
Metadata:
Creation Timestamp: 2019-03-15T11:38:03Z
Generation: 1
Resource Version: 623999
Self Link: /apis/certmanager.k8s.io/v1alpha1/clusterissuers/letsencrypt-staging
UID: cb48b391-4716-11e9-a113-e27267a7d354
Spec:
Acme:
Email: b.garcia@possibilit.nl
Http 01:
Private Key Secret Ref:
Name: letsencrypt-staging
Server: https://acme-staging-v02.api.letsencrypt.org/directory
Status:
Acme:
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/8579841
Conditions:
Last Transition Time: 2019-03-15T11:38:05Z
Message: The ACME account was registered with the ACME server
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
[I]
Modifying my ingress resource created previously
- Now that I've created our Let's Encrypt staging ClusterIssuer, I am ready to modify the Ingress Resource we created above and enable TLS encryption for the test1kongletsencrypt.possibilit.nl paths adding the following
I am going to add certmanager.k8s.io/cluster-issuer: letsencrypt-staging annotation and use the secret created with the letsencrypt-staging ClusterIssuer named letsencrypt-staging
Our ingress has been stayed of this way:
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: kong-ingress-zcrm365
#namespace: default
annotations:
# kubernetes.io/ingress.class: "nginx" #new
# certmanager.k8s.io/acme-challenge-type: http01
# add an annotation indicating the issuer to use.
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
spec:
rules:
- host: test1kongletsencrypt.possibilit.nl
http:
paths:
- path: "/"
backend:
serviceName: zcrm365dev
servicePort: 80
tls:
- hosts:
- test1kongletsencrypt.possibilit.nl
secretName: letsencrypt-staging # I've added this secret of letsencrypt cluster issuer
- Apply it
⟩ kubectl apply -f 03-zcrm365-ingress.yaml
ingress.extensions/kong-ingress-zcrm365 configured
[I]
This process update on the ingress, create one ingres named cm-acme-http-solver-jr4fg
⟩ kubectl get ingress
NAME HOSTS ADDRESS PORTS AGE
cm-acme-http-solver-jr4fg test1kongletsencrypt.possibilit.nl 80 33s
kong-ingress-zcrm365 test1kongletsencrypt.possibilit.nl 52.166.60.158 80, 443 56m
[I]
The detail of cm-acme-http-solver-jr4fg ingress is:
⟩ kubectl get ingress cm-acme-http-solver-jr4fg -o yaml
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0
creationTimestamp: "2019-03-15T12:10:57Z"
generateName: cm-acme-http-solver-
generation: 1
labels:
certmanager.k8s.io/acme-http-domain: "4095675862"
certmanager.k8s.io/acme-http-token: "657526223"
name: cm-acme-http-solver-jr4fg
namespace: default
ownerReferences:
- apiVersion: certmanager.k8s.io/v1alpha1
blockOwnerDeletion: true
controller: true
kind: Challenge
name: letsencrypt-staging-2613163196-0
uid: 638f1701-471b-11e9-a113-e27267a7d354
resourceVersion: "628284"
selfLink: /apis/extensions/v1beta1/namespaces/default/ingresses/cm-acme-http-solver-jr4fg
uid: 640ef483-471b-11e9-a113-e27267a7d354
spec:
rules:
- host: test1kongletsencrypt.possibilit.nl
http:
paths:
- backend:
serviceName: cm-acme-http-solver-svmvw
servicePort: 8089
path: /.well-known/acme-challenge/W7-9-KuPao_jg6EF5E2FXitFs8shOEsY5PlT9EEvNxE
status:
loadBalancer:
ingress:
- ip: 52.166.60.158
- And the detail of our
kong-ingress-zcrm365resource ingress is:
⟩ kubectl describe ingress kong-ingress-zcrm365
Name: kong-ingress-zcrm365
Namespace: default
Address: 52.166.60.158
Default backend: default-http-backend:80 (<none>)
TLS:
letsencrypt-staging terminates test1kongletsencrypt.possibilit.nl
Rules:
Host Path Backends
---- ---- --------
test1kongletsencrypt.possibilit.nl
/ zcrm365dev:80 (<none>)
Annotations:
certmanager.k8s.io/cluster-issuer: letsencrypt-staging
kubectl.kubernetes.io/last-applied-configuration: {"apiVersion":"extensions/v1beta1","kind":"Ingress","metadata":{"annotations":{"certmanager.k8s.io/cluster-issuer":"letsencrypt-staging"},"name":"kong-ingress-zcrm365","namespace":"default"},"spec":{"rules":[{"host":"test1kongletsencrypt.possibilit.nl","http":{"paths":[{"backend":{"serviceName":"zcrm365dev","servicePort":80},"path":"/"}]}}],"tls":[{"hosts":["test1kongletsencrypt.possibilit.nl"],"secretName":"letsencrypt-staging"}]}}
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal CREATE 60m kong-ingress-controller Ingress default/kong-ingress-zcrm365
Normal UPDATE 4m25s (x2 over 60m) kong-ingress-controller Ingress default/kong-ingress-zcrm365
Normal CreateCertificate 4m25s cert-manager Successfully created Certificate "letsencrypt-staging"
[I]
We can see that our ingress even using the kong-ingress-controller and the letsencrypt-staging certificate has been created in the default namespace:
⟩ kubectl get certificates
NAME
letsencrypt-staging
[I]
The letsencypt-staging certificate have the following detail
⟩ kubectl describe certificate letsencrypt-staging
Name: letsencrypt-staging
Namespace: default
Labels: <none>
Annotations: <none>
API Version: certmanager.k8s.io/v1alpha1
Kind: Certificate
Metadata:
Creation Timestamp: 2019-03-15T12:10:55Z
Generation: 1
Owner References:
API Version: extensions/v1beta1
Block Owner Deletion: true
Controller: true
Kind: Ingress
Name: kong-ingress-zcrm365
UID: 8643558f-4713-11e9-a113-e27267a7d354
Resource Version: 628164
Self Link: /apis/certmanager.k8s.io/v1alpha1/namespaces/default/certificates/letsencrypt-staging
UID: 62b3a31e-471b-11e9-a113-e27267a7d354
Spec:
Acme:
Config:
Domains:
test1kongletsencrypt.possibilit.nl
Http 01:
Dns Names:
test1kongletsencrypt.possibilit.nl
Issuer Ref:
Kind: ClusterIssuer
Name: letsencrypt-staging
Secret Name: letsencrypt-staging
Status:
Conditions:
Last Transition Time: 2019-03-15T12:10:55Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 7m24s cert-manager Generated new private key
Normal GenerateSelfSigned 7m24s cert-manager Generated temporary self signed certificate
Normal OrderCreated 7m23s cert-manager Created Order resource "letsencrypt-staging-2613163196"
[I]
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)
I can see that my order issue is not completed, only was created in the OrderCreated event, and this order already have 7 minutes since I've created this certificate and the order was not completed and by that reason the certificate is not issued successfully
Another thing that happens to me, is that the letsencrypt-staging secret created by the letsencrypt-staging cluster Issuer and their respective certificate, only have the tls.key:
⟩ kubectl describe secrets letsencrypt-staging -n kube-system
Name: letsencrypt-staging
Namespace: kube-system
Labels: <none>
Annotations: <none>
Type: Opaque
Data
====
tls.key: 1675 bytes
[I]
According to I understand, is that if the letsencrypt certificate complete the order and the certificate is issued, in the letsencrypt-staging secret I would have one tls.crt key and maybe my letsencrypt-staging will be of tls type and not Opaque?
- When I see the logs of my cert-manager pod I get the following output, I think that the http challenge is not executed:
I0315 12:10:56.109919 1 controller.go:168] certificates controller: Finished processing work item "default/letsencrypt-staging"
I0315 12:10:56.910004 1 logger.go:73] Calling GetAuthorization
I0315 12:10:57.135213 1 logger.go:93] Calling HTTP01ChallengeResponse
I0315 12:10:57.150924 1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.151110 1 controller.go:162] certificates controller: syncing item 'default/letsencrypt-staging'
I0315 12:10:57.151113 1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.151521 1 sync.go:274] Need to create 1 challenges
I0315 12:10:57.151891 1 issue.go:160] Order default/letsencrypt-staging-2613163196 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0315 12:10:57.152022 1 controller.go:168] certificates controller: Finished processing work item "default/letsencrypt-staging"
I0315 12:10:57.160718 1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.160804 1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.160870 1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.160739 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.161051 1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:57.161214 1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.161263 1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.161287 1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.562383 1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.562738 1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.562755 1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.562906 1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.563018 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.563231 1 logger.go:68] Calling GetChallenge
I0315 12:10:57.833421 1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:57.833648 1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I0315 12:10:57.833755 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:57.833820 1 sync.go:274] Need to create 0 challenges
I0315 12:10:57.833828 1 sync.go:323] Waiting for all challenges for order "letsencrypt-staging-2613163196" to enter 'valid' state
I0315 12:10:57.833838 1 controller.go:190] orders controller: Finished processing work item "default/letsencrypt-staging-2613163196"
I0315 12:10:57.833858 1 logger.go:103] Calling Discover
I0315 12:10:57.856136 1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.923080 1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:57.989596 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:57.989682 1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/letsencrypt-staging-2613163196-0". One will be created.
I0315 12:10:58.014803 1 controller.go:178] ingress-shim controller: syncing item 'default/cm-acme-http-solver-jr4fg'
I0315 12:10:58.014842 1 sync.go:64] Not syncing ingress default/cm-acme-http-solver-jr4fg as it does not contain necessary annotations
I0315 12:10:58.014846 1 controller.go:184] ingress-shim controller: Finished processing work item "default/cm-acme-http-solver-jr4fg"
I0315 12:10:58.015447 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 12:10:58.033431 1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 12:10:58.079504 1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 12:10:58.079616 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I0315 12:10:58.079569 1 controller.go:184] orders controller: syncing item 'default/letsencrypt-staging-2613163196'
I get this message No existing HTTP01 challenge solver pod found for Certificate "default/letsencrypt-staging-2613163196-0"
According to this, I decide add the certmanager.k8s.io/acme-challenge-type: http01 annotation to my kong-ingress-zcrm365 ingress but nothing happened ... my ingress is updated, but nothing more.
All this process confirms that the TLS certificate was not successfully issued and HTTPS encryption is not active for my domains test1kongletsencrypt.possibilit.nl configured.
This make that my letsencrypt-staging certificate have a Status:False, and the order created event does not advance to completed to be issued.
Conditions:
Last Transition Time: 2019-03-15T12:10:55Z
Message: Certificate issuance in progress. Temporary certificate issued.
Reason: TemporaryCertificate
Status: False
Type: Ready
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Generated 51m cert-manager Generated new private key
Normal GenerateSelfSigned 51m cert-manager Generated temporary self signed certificate
Normal Cleanup 5m42s cert-manager Deleting old Order resource "letsencrypt-staging-2613163196"
Normal OrderCreated 5m42s cert-manager Created Order resource "letsencrypt-staging-2965106631"
Normal OrderCreated 39s (x2 over 51m) cert-manager Created Order resource "letsencrypt-staging-2613163196"
Normal Cleanup 39s cert-manager Deleting old Order resource "letsencrypt-staging-2965106631"
[I]
~/workspace/ZCRM365/Deployments/Kubernetes/cert-manager · (Deployments±)
How to can I to my certificate to be signed and succesfully issued by letsencrypt CA and active the https encryption active?
What is happening with these logs messages?
kubectl logs -n kube-system cert-manager-6f68b58796-q7txg
0315 13:06:11.027204 1 logger.go:103] Calling Discover
I0315 13:06:11.032299 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=4095675862,certmanager.k8s.io/acme-http-token=657526223
I0315 13:06:11.046081 1 sync.go:173] propagation check failed: wrong status code '404', expected '200'
I0315 13:06:11.046109 1 controller.go:212] challenges controller: Finished processing work item "default/letsencrypt-staging-2613163196-0"
I0315 13:06:21.046242 1 controller.go:206] challenges controller: syncing item 'default/letsencrypt-staging-2613163196-0'
I've heared that letsencrypt-staging environment only have test certificates and these are a kind of 'fake certificates' and maybe some clients like my chrome/firefox browser doesn’t trust certificate issuer ...
Is this a reason to I cannot enable https encryption on my domain?
In affirmative case, should I change from staging environment to production environment?
In this question some people talk about that but they emphasize:
that the staging environment should be used just to test that your client is working fine and can generate the challenges, certificates
In my case the http challenge is not generated still in staging environment. 
I just wanted to make sure you also knew about the Slack option.