Problem with cert-manager and first time install on kubernetes cluster

My domain is: www.billapaji.com
I ran this command : check on website
It produced this output:

My web server is (include version): nginx
The operating system my web server runs on is (include version): ubuntu-server 21
My hosting provider, if applicable, is: bigrock.in
I can login to a root shell on my machine (yes or no, or I don't know): Yes
I'm using a control panel to manage my site (no, or provide the name and version of the control panel): No
The version of my client is:"using cert-manager"

Unrelated to your solution, but Ubuntu 21 doesn't have Long Term Support.

Did the cert-manager previously work for you?
Did anything IP/DNS related change since the last renewal?
[have you recently modified your /etc/hosts file?]

What shows?:
nginx -T | grep -i ssl_cert

4 Likes

nginx

Please copy and paste the entire command.
[that's a capital T]

If you did use a capital T, then nginx isn't using any certificate at all...
Which means that your answer may be incorrect:

OR
You did not run the command within the correct VM.

3 Likes

I am using two physical machines to make cluster.

Why doesn't the nginx output show the lines that use ssl?

3 Likes

Because I have re-installed the operating system of my server.

Have you configured a ClusterIssuer per the cert-manager tutorials? Why are you apparently not using an ingress controller (e.g. ingress-nginx)?

3 Likes

I am using the issuer instead of ClusterIssuer and stuck with challenge http01 pending status "
Reason: Waiting for HTTP-01 challenge propagation: failed to perform self check GET request 'http://www.billapaji.com/.well-known/acme-challenge/047jzunW2457wEc2ss99huivwj7VwQgmZ0gAEmxaXPc': Get "http://www.billapaji.com/.well-known/acme-challenge/047jzunW2457wEc2ss99huivwj7VwQgmZ0gAEmxaXPc": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"

Please post the output of kubectl for the logs of cert-manager. That will probably give a deeper indication of what's happening.

3 Likes

That indicates that your nginx isn't responding (or maybe isn't able to respond) to the request for the challenge file. Considering that it's cert-manager making that HEAD request, it seems that your server isn't able to reach itself.

3 Likes
I0123 12:16:42.672621       1 start.go:75] cert-manager "msg"="starting controller" "git-commit"="2a0ef53b06e183356d922cd58af2510d8885bef5" "version"="v1.11.0"
I0123 12:16:42.672737       1 controller.go:242] cert-manager/controller/build-context "msg"="configured acme dns01 nameservers" "nameservers"=["10.96.0.10:53"]
W0123 12:16:42.677932       1 client_config.go:618] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0123 12:16:42.696308       1 controller.go:70] cert-manager/controller "msg"="enabled controllers: [certificaterequests-approver certificaterequests-issuer-acme certificaterequests-issuer-ca certificaterequests-issuer-selfsigned certificaterequests-issuer-vault certificaterequests-issuer-venafi certificates-issuing certificates-key-manager certificates-metrics certificates-readiness certificates-request-manager certificates-revision-manager certificates-trigger challenges clusterissuers ingress-shim issuers orders]" 
I0123 12:16:42.699005       1 controller.go:91] cert-manager/controller "msg"="starting metrics server" "address"={"IP":"::","Port":9402,"Zone":""}
I0123 12:16:42.699085       1 controller.go:134] cert-manager/controller "msg"="starting leader election" 
I0123 12:16:42.700857       1 leaderelection.go:248] attempting to acquire leader lease kube-system/cert-manager-controller...
I0123 12:16:42.823203       1 leaderelection.go:258] successfully acquired lease kube-system/cert-manager-controller
I0123 12:16:42.826108       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="gateway-shim"
I0123 12:16:42.826701       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-acme"
I0123 12:16:42.827379       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="orders"
I0123 12:16:42.827442       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-readiness"
I0123 12:16:42.827972       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-selfsigned"
I0123 12:16:42.830603       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-vault"
I0123 12:16:42.838708       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-key-manager"
I0123 12:16:42.839360       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-revision-manager"
I0123 12:16:42.839395       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-trigger"
I0123 12:16:42.841142       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="issuers"
I0123 12:16:42.843223       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-ca"
I0123 12:16:42.843742       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-selfsigned"
I0123 12:16:42.843606       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-ca"
I0123 12:16:42.843622       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="challenges"
I0123 12:16:42.843673       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-issuing"
I0123 12:16:42.843714       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-request-manager"
I0123 12:16:42.846128       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="ingress-shim"
I0123 12:16:42.846233       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-acme"
I0123 12:16:42.846310       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-approver"
I0123 12:16:42.846395       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificaterequests-issuer-venafi"
I0123 12:16:42.846596       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-vault"
I0123 12:16:42.846686       1 controller.go:182] cert-manager/controller "msg"="not starting controller as it's disabled" "controller"="certificatesigningrequests-issuer-venafi"
I0123 12:16:42.846654       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="certificates-metrics"
I0123 12:16:42.849859       1 controller.go:205] cert-manager/controller "msg"="starting controller" "controller"="clusterissuers"
I0123 12:16:42.957550       1 setup.go:204] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging-new" "resource_namespace"="" "resource_version"="v1"
I0123 12:16:47.862619       1 setup.go:204] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging-new" "resource_namespace"="" "resource_version"="v1"
I0123 12:33:57.285468       1 setup.go:221] cert-manager/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0123 12:34:02.747875       1 setup.go:311] cert-manager/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0123 12:34:02.748158       1 conditions.go:96] Setting lastTransitionTime for Issuer "letsencrypt-staging" condition "Ready" to 2023-01-23 12:34:02.747906145 +0000 UTC m=+1041.021158789
I0123 12:34:02.788182       1 setup.go:204] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-staging" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-staging" "resource_namespace"="" "resource_version"="v1"
I0123 12:34:32.937039       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Ready" to 2023-01-23 12:34:32.93702619 +0000 UTC m=+1071.210278838
E0123 12:36:57.089309       1 controller.go:145] cert-manager/clusterissuers "msg"="clusterissuer in work queue no longer exists" "error"="clusterissuer.cert-manager.io \"letsencrypt-staging-new\" not found"
I0123 12:44:11.896998       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Ready" to 2023-01-23 12:44:11.89698546 +0000 UTC m=+1650.170238105
E0123 12:46:48.830236       1 controller.go:145] cert-manager/clusterissuers "msg"="clusterissuer in work queue no longer exists" "error"="clusterissuer.cert-manager.io \"letsencrypt-staging\" not found"
I0123 12:51:22.282437       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Ready" to 2023-01-23 12:51:22.28242596 +0000 UTC m=+2080.555678607
I0123 12:58:50.204580       1 setup.go:221] cert-manager/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
I0123 12:59:10.736317       1 setup.go:311] cert-manager/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
I0123 12:59:10.736548       1 conditions.go:96] Setting lastTransitionTime for Issuer "letsencrypt-prod" condition "Ready" to 2023-01-23 12:59:10.736497233 +0000 UTC m=+2549.009749908
I0123 12:59:10.785868       1 setup.go:204] cert-manager/clusterissuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-prod" "resource_namespace"="" "resource_version"="v1"
I0123 13:00:38.900339       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Ready" to 2023-01-23 13:00:38.900326539 +0000 UTC m=+2637.173579192
E0123 13:04:06.255535       1 controller.go:98] ingress 'default/billapaji' in work queue no longer exists
E0123 13:04:06.319102       1 controller.go:98] ingress 'default/billapaji' in work queue no longer exists
I0123 13:06:03.458538       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Ready" to 2023-01-23 13:06:03.458528653 +0000 UTC m=+2961.731781283
E0123 13:14:16.058816       1 controller.go:145] cert-manager/clusterissuers "msg"="clusterissuer in work queue no longer exists" "error"="clusterissuer.cert-manager.io \"letsencrypt-prod\" not found"
I0123 13:14:42.295414       1 setup.go:111] cert-manager/issuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="default" "resource_kind"="Issuer" "resource_name"="letsencrypt-prod" "resource_namespace"="default" "resource_version"="v1"
I0123 13:14:42.892432       1 setup.go:221] cert-manager/issuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="default" "resource_kind"="Issuer" "resource_name"="letsencrypt-prod" "resource_namespace"="default" "resource_version"="v1"
I0123 13:14:47.961281       1 setup.go:311] cert-manager/issuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="default" "resource_kind"="Issuer" "resource_name"="letsencrypt-prod" "resource_namespace"="default" "resource_version"="v1"
I0123 13:14:47.961319       1 conditions.go:96] Setting lastTransitionTime for Issuer "letsencrypt-prod" condition "Ready" to 2023-01-23 13:14:47.961310417 +0000 UTC m=+3486.234563064
I0123 13:14:48.131959       1 setup.go:204] cert-manager/issuers "msg"="skipping re-verifying ACME account as cached registration details look sufficient" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-prod" "related_resource_namespace"="default" "resource_kind"="Issuer" "resource_name"="letsencrypt-prod" "resource_namespace"="default" "resource_version"="v1"
I0123 13:15:37.164985       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Ready" to 2023-01-23 13:15:37.164973945 +0000 UTC m=+3535.438226592
I0123 13:17:39.440815       1 trigger_controller.go:200] cert-manager/certificates-trigger "msg"="Certificate must be re-issued" "key"="default/quickstart-example-tls" "message"="Issuing certificate as Secret was previously issued by ClusterIssuer.cert-manager.io/letsencrypt-prod" "reason"="IncorrectIssuer"
I0123 13:17:39.441161       1 conditions.go:203] Setting lastTransitionTime for Certificate "quickstart-example-tls" condition "Issuing" to 2023-01-23 13:17:39.441149865 +0000 UTC m=+3657.714402528
I0123 13:17:39.499257       1 controller.go:162] cert-manager/certificates-readiness "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"quickstart-example-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/quickstart-example-tls"
I0123 13:17:39.615977       1 controller.go:162] cert-manager/certificates-key-manager "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"quickstart-example-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/quickstart-example-tls"
I0123 13:17:39.783731       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "quickstart-example-tls-llg7g" condition "Approved" to 2023-01-23 13:17:39.783719025 +0000 UTC m=+3658.056971685
I0123 13:17:39.941804       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "quickstart-example-tls-llg7g" condition "Ready" to 2023-01-23 13:17:39.94178524 +0000 UTC m=+3658.215037887
I0123 13:17:40.002116       1 conditions.go:263] Setting lastTransitionTime for CertificateRequest "quickstart-example-tls-llg7g" condition "Ready" to 2023-01-23 13:17:40.002100989 +0000 UTC m=+3658.275353656
I0123 13:17:40.035480       1 controller.go:162] cert-manager/certificaterequests-issuer-acme "msg"="re-queuing item due to optimistic locking on resource" "error"="Operation cannot be fulfilled on certificaterequests.cert-manager.io \"quickstart-example-tls-llg7g\": the object has been modified; please apply your changes to the latest version and try again" "key"="default/quickstart-example-tls-llg7g"
I0123 13:18:04.511454       1 pod.go:71] cert-manager/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="billapaji.com" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-4118827627" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:04.636319       1 pod.go:71] cert-manager/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="www.billapaji.com" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-1871341451" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:04.831683       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="billapaji.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-6v4zr" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-4118827627" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:04.831932       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="billapaji.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vpnd4" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-4118827627" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:05.035991       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.billapaji.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-87jz2" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-1871341451" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:05.036080       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.billapaji.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-ppvwz" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-1871341451" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0123 13:18:14.832277       1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://billapaji.com/.well-known/acme-challenge/-Q_7-yew0CcKFKbiQC-t5TQdLQ-W3zuB_2KfcwpMEPo': Get \"http://billapaji.com/.well-known/acme-challenge/-Q_7-yew0CcKFKbiQC-t5TQdLQ-W3zuB_2KfcwpMEPo\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="billapaji.com" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-4118827627" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:14.862497       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="billapaji.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-6v4zr" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-4118827627" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:14.862602       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="billapaji.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-vpnd4" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-4118827627" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0123 13:18:15.036888       1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="failed to perform self check GET request 'http://www.billapaji.com/.well-known/acme-challenge/047jzunW2457wEc2ss99huivwj7VwQgmZ0gAEmxaXPc': Get \"http://www.billapaji.com/.well-known/acme-challenge/047jzunW2457wEc2ss99huivwj7VwQgmZ0gAEmxaXPc\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" "dnsName"="www.billapaji.com" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-1871341451" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:15.080968       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="www.billapaji.com" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-87jz2" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-1871341451" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0123 13:18:15.081289       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="www.billapaji.com" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-ppvwz" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="quickstart-example-tls-llg7g-1842627701-1871341451" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"

But why? I am able to access it from the internet "www.billapaji.com"

Does your server have working outbound internet access? Where are the requests originating from your server being routed? It's possible they're being dropped by a firewall/router or that nginx isn't responding to them for the challenge files, but is responding to regular traffic. Also (and possibly unrelated), I've noticed that clusterissuer is mentioned hundreds of times in your log with "incorrect issuer" appearing. You might want to check your configuration against the tutorials to which I linked above.

3 Likes

Recently I have changed my timezone in my os. Do I need to setup it up again to have a new certificate?

Yes, I have outbound internet access. I am using my static public IP address to access my server from the internet and am able to access my website from any device with the HTTP request.

progress...
[slow but moving forward]

3 Likes

Yes, but still no solution

Well...
What all has changed (since it last worked)?
Did you upgrade the server OR blew it away and started a new one?

3 Likes

Still on the same level where you leave me.