nvcnvn
December 22, 2021, 5:45am
1
My domain is:admin.prod.ga.manabie.io
Check your redirects http - https, your preferred version (www vs. non-www), certificates, connections and your html-content. A ranking system shows, if your domain is A+ (no errors + preload), has errors (https - http) or loops.
I ran this command:kubectl cert-manager renew admin-prod-ga-gateway
It produced this output: Warning Failed 8s (x7 over 57m) cert-manager The certificate request has failed to complete and will be retried: Failed to wait for order resource "admin-prod-ga-gateway-mbfnd-3584482769" to become ready: order is in "errored" state: Failed to finalize Order: 403 urn:ietf:params:acme:error:caa: Error finalizing order :: While processing CAA for admin.prod.ga.manabie.io: DNS problem: SERVFAIL looking up CAA for prod.ga.manabie.io - the domain's nameservers may be malfunctioning
cert-manager v1.4.0
We're using Google Cloud DNS with DNSSec disabled.
We can just create the new cert with http-01 successfully with exactly the same infomation
1 Like
MikeMcQ
December 23, 2021, 3:21pm
2
It looks like you got a new certificate yesterday . Do you still need help?
I am not familiar with your kubectl cert-manager. I am also not a DNS expert. But, this website often shows problems that helps others fix DNS problems. See the error using UDP to contact the DNS server. Perhaps this was part of your original problem?https://dnsviz.net/d/admin.prod.ga.manabie.io/dnssec/
3 Likes
nvcnvn
December 24, 2021, 12:41am
3
Actually I can just create the new cert normally.
opened 05:44PM - 20 Jan 21 UTC
closed 06:01AM - 18 Aug 21 UTC
kind/bug
priority/important-soon
area/acme
triage/needs-information
<!--
Bugs should be filed for issues encountered whilst operating cert-manager.…
You should first attempt to resolve your issues through the community support
channels, e.g. Slack, in order to rule out individual configuration errors.
Please provide as much detail as possible.
-->
**Describe the bug**:
<!--
A clear and concise description of what the bug is.
Tip: you can use
```
<code here>
```
for code blocks of your kubectl output or YAML files.
-->
We are using cert-manager + letsencrypt staging in our test pipeline. The pipeline only runs a few times a week, so it's well under the limits of letsencrypt staging. Our certificate sometimes fails to issue properly. We know that our config is OK since it usually works fine. However in the last week we had two failures caused by cert-manager failing to issue the certificate. We did not encounter this problem before this week.
Our certificate has this spec:
```
spec:
dnsNames:
- '*.test-khhyml7mh2jkg732.loci.ubi.com'
issuerRef:
kind: ClusterIssuer
name: letsencrypt-test
secretName: wildcard-cert-tls
```
Our cluster issuer has this spec:
```
spec:
acme:
email: <redacted>@ubisoft.com
preferredChain: ""
privateKeySecretRef:
name: letsencrypt-test-issuer-account-key
server: https://acme-staging-v02.api.letsencrypt.org/directory
solvers:
- dns01:
cloudDNS:
project: bob-dbaas-dev
serviceAccountSecretRef:
key: credentials.json
name: cert-manager-google-secret
```
The cert manager logs are below:
```
I0120 16:14:44.476975 1 conditions.go:173] Setting lastTransitionTime for Certificate "wildcard-cert" condition "Issuing" to 2021-01-20 16:14:44.476965945 +0000 UTC m=+13.944761228
I0120 16:14:44.477078 1 conditions.go:173] Setting lastTransitionTime for Certificate "wildcard-cert" condition "Ready" to 2021-01-20 16:14:44.477064448 +0000 UTC m=+13.944859754
E0120 16:14:44.575829 1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"wildcard-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="mongo-operator/wildcard-cert"
I0120 16:14:44.575909 1 conditions.go:173] Setting lastTransitionTime for Certificate "wildcard-cert" condition "Issuing" to 2021-01-20 16:14:44.575903168 +0000 UTC m=+14.043698423
I0120 16:14:44.580373 1 setup.go:90] cert-manager/controller/clusterissuers "msg"="generating acme account private key" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
I0120 16:14:44.813391 1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "wildcard-cert-mjj2b" condition "Ready" to 2021-01-20 16:14:44.813382149 +0000 UTC m=+14.281177396
I0120 16:14:44.922381 1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
I0120 16:14:45.391446 1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
I0120 16:14:45.391473 1 conditions.go:92] Setting lastTransitionTime for Issuer "letsencrypt-test" condition "Ready" to 2021-01-20 16:14:45.391467192 +0000 UTC m=+14.859262437
I0120 16:14:45.526098 1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
E0120 16:14:45.620396 1 controller.go:158] cert-manager/controller/orders "msg"="re-queuing item due to error processing" "error"="ACME client for issuer not initialised/available" "key"="mongo-operator/wildcard-cert-mjj2b-2407777396"
I0120 16:14:46.048213 1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
I0120 16:14:49.922449 1 setup.go:178] cert-manager/controller/clusterissuers "msg"="ACME server URL host and ACME private key registration host differ. Re-checking ACME account registration" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
I0120 16:14:50.388598 1 setup.go:270] cert-manager/controller/clusterissuers "msg"="verified existing registration with ACME server" "related_resource_kind"="Secret" "related_resource_name"="letsencrypt-test-issuer-account-key" "related_resource_namespace"="cert-manager" "resource_kind"="ClusterIssuer" "resource_name"="letsencrypt-test" "resource_namespace"="" "resource_version"="v1"
E0120 16:14:53.757052 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:14:53.791518 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:15:03.762095 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:15:13.767389 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:15:23.789744 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:15:33.794373 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:15:43.799491 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:15:53.835277 1 sync.go:182] cert-manager/controller/challenges "msg"="propagation check failed" "error"="DNS record for \"test-khhyml7mh2jkg732.loci.ubi.com\" not yet propagated" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
E0120 16:17:34.960129 1 sync.go:354] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for test-khhyml7mh2jkg732.loci.ubi.com: 400 urn:ietf:params:acme:error:dns: During secondary validation: DNS problem: NXDOMAIN looking up TXT for _acme-challenge.test-khhyml7mh2jkg732.loci.ubi.com - check that a DNS record exists for this domain" "dnsName"="test-khhyml7mh2jkg732.loci.ubi.com" "resource_kind"="Challenge" "resource_name"="wildcard-cert-mjj2b-2407777396-2216626796" "resource_namespace"="mongo-operator" "resource_version"="v1" "type"="DNS-01"
I0120 16:17:35.135532 1 conditions.go:162] Found status change for Certificate "wildcard-cert" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2021-01-20 16:17:35.135521875 +0000 UTC m=+184.603317157
E0120 16:17:35.174806 1 controller.go:158] cert-manager/controller/CertificateReadiness "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"wildcard-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="mongo-operator/wildcard-cert"
I0120 16:17:35.176107 1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="mongo-operator/wildcard-cert" "retry_after"="2021-01-20T17:17:35Z"
I0120 16:17:35.233041 1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="mongo-operator/wildcard-cert" "retry_after"="2021-01-20T17:17:35Z"
E0120 16:17:35.247545 1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"wildcard-cert\": the object has been modified; please apply your changes to the latest version and try again" "key"="mongo-operator/wildcard-cert"
```
Also, status on certificate
```
- lastTransitionTime: "2021-01-20T17:17:35Z"
message: 'The certificate request has failed to complete and will be retried:
Failed to wait for order resource "wildcard-cert-mjj2b-2407777396" to become
ready: order is in "invalid" state: '
reason: Failed
status: "False"
type: Issuing
```
**Expected behaviour**:
Certificate should issue properly
**Steps to reproduce the bug**:
<!--Steps to reproduce the bug should be clear and easily reproducible to help people
gain an understanding of the problem.-->
Setup a cluster issuer with letsencrypt staging, ask for a certificate.
**Environment details:**
- Kubernetes version: 1.16
- Cloud-provider/provisioner: GKE
- cert-manager version: 1.1.0
- Install method: helm
/kind bug
2 Likes
system
January 23, 2022, 12:42am
4
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.