Kubernetes acme http01 returns urn:ietf:params:acme:error:dns despite correct A record

Help
1

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel): no.

My hosting provider, if applicable, is: cloudflare + namecheap

Kubernetes nodes OS/arch: ubuntu / arm64

Kubernetes "flavor": k3s

I'm using kubernetes with an nginx ingress + cert-manager helm chart (latest version 1.0.4).

I'm currently trying to use ACME verification to issue the certificate , but the challenge becomes invalid with the warning: "Accepting challenge authorization failed: acme: authorization error for : 400 urn:ietf:params:acme:error:dns: No valid IP addresses found for ".

I've read pretty much all the previous occurrences in this forum regarding the same error and all seemed to point to wrong DNS configuration. I confirmed that I have an A record configured for this name and a dns lookup works:

nslookup <redacted>
Server:		10.10.16.1
Address:	10.10.16.1#53

Non-authoritative answer:
Name:	<redacted>
Address: 100.64.201.33

Full logs of cert-manager pod:

I1111 06:45:52.179239       1 conditions.go:173] Setting lastTransitionTime for Certificate "vault-ingress-tls" condition "Ready" to 2020-11-11 06:45:52.179215977 +0000 UTC m=+57652.127550743
I1111 06:45:52.179300       1 conditions.go:173] Setting lastTransitionTime for Certificate "vault-ingress-tls" condition "Issuing" to 2020-11-11 06:45:52.179287605 +0000 UTC m=+57652.127622413
E1111 06:45:52.438138       1 controller.go:158] cert-manager/controller/CertificateTrigger "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"vault-ingress-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="tools/vault-ingress-tls"
I1111 06:45:52.438317       1 conditions.go:173] Setting lastTransitionTime for Certificate "vault-ingress-tls" condition "Issuing" to 2020-11-11 06:45:52.438300988 +0000 UTC m=+57652.386635796
E1111 06:45:53.516403       1 controller.go:158] cert-manager/controller/CertificateKeyManager "msg"="re-queuing item  due to error processing" "error"="Operation cannot be fulfilled on certificates.cert-manager.io \"vault-ingress-tls\": the object has been modified; please apply your changes to the latest version and try again" "key"="tools/vault-ingress-tls"
I1111 06:45:53.615072       1 conditions.go:233] Setting lastTransitionTime for CertificateRequest "vault-ingress-tls-g8rw9" condition "Ready" to 2020-11-11 06:45:53.615046531 +0000 UTC m=+57653.563381214
I1111 06:45:58.503365       1 pod.go:70] cert-manager/controller/challenges/http01/ensurePod "msg"="creating HTTP01 challenge solver pod" "dnsName"="<redacted>" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:45:59.848152       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="<redacted>" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-47929" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:45:59.849154       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="<redacted>" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-9fc5f" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:45:59.850220       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="<redacted>" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-q7dff" "related_resource_namespace"="tools" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
E1111 06:45:59.916376       1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="<redacted>" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:00.082382       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="<redacted>" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-47929" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:00.082654       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="<redacted>" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-9fc5f" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:00.082901       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="<redacted>" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-q7dff" "related_resource_namespace"="tools" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
E1111 06:46:00.118466       1 sync.go:183] cert-manager/controller/challenges "msg"="propagation check failed" "error"="wrong status code '503', expected '200'" "dnsName"="<redacted>" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:09.917194       1 pod.go:58] cert-manager/controller/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="<redacted>" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-47929" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:09.917476       1 service.go:43] cert-manager/controller/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="<redacted>" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-9fc5f" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:09.917883       1 ingress.go:91] cert-manager/controller/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="<redacted>" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-q7dff" "related_resource_namespace"="tools" "related_resource_version"="v1beta1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
E1111 06:46:20.460226       1 sync.go:356] cert-manager/controller/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for <redacted>: 400 urn:ietf:params:acme:error:dns: No valid IP addresses found for <redacted>" "dnsName"="<redacted>" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:20.503582       1 pod.go:118] cert-manager/controller/challenges/cleanupPods "msg"="deleting pod resource" "dnsName"="<redacted>" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-47929" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
I1111 06:46:20.523911       1 pod.go:126] cert-manager/controller/challenges/cleanupPods "msg"="successfully deleted pod resource" "dnsName"="<redacted>" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-47929" "related_resource_namespace"="tools" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="vault-ingress-tls-g8rw9-2834219037-2762095063" "resource_namespace"="tools" "resource_version"="v1" "type"="HTTP-01"
E1111 06:46:20.974006       1 controller.go:156] ingress 'tools/cm-acme-http-solver-q7dff' in work queue no longer exists
I1111 06:46:20.988585       1 conditions.go:162] Found status change for Certificate "vault-ingress-tls" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2020-11-11 06:46:20.98855727 +0000 UTC m=+57680.936891995
I1111 06:46:21.124192       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="tools/vault-ingress-tls" "retry_after"="2020-11-11T07:46:20Z"
I1111 06:46:21.267054       1 trigger_controller.go:162] cert-manager/controller/CertificateTrigger "msg"="Not re-issuing certificate as an attempt has been made in the last hour" "key"="tools/vault-ingress-tls" "retry_after"="2020-11-11T07:46:20Z"

I'm using the following ClusterIssuer:

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-issuer
spec:
  acme:
    # You must replace this email address with your own.
    # Let's Encrypt will use this to contact you about expiring
    # certificates, and issues related to your account.
    email: <redacted>
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      # Secret resource that will be used to store the account's private key.
      name: letsencrypt-issue
    # Add a single challenge solver, HTTP01 using nginx
    solvers:
    - http01:
        ingress:
          class: nginx

Thanks in advance!

1 Like
2

100.64.201.33 is a CGNAT address, which is not publicly routable.

You cannot host a server on a CGNAT internet connection.

You might be able to ask your ISP for a real IP address.

5 Likes
3

You took a minute to solve my day-long debugging. When you know you know. Thank you!

In case anyone is interested, this is bypassable (at least in my case) by using ipv6.

I configured both A and AAAA records for the domain and everything worked (letsencrypt uses ipv6 preferentially if an AAAA record is configured). This is because ISPs will most likely issue you a unique ipv6 address despite using a CGNAT ipv4.

The ipv4 address was kept for compatibility with some apps. Modern browsers will happily resolve your domain via ipv6, but some apps won't. K3s doesn't support dual stack yet, so ipv4 resolution was still required.

I'm extremely happy that this has worked and I thank you very much @_az for your amazing help.

4 Likes
4

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.