Acme: authorization 403 error when setting up Ingress on Minikube

Help
1

I have an API that is running in Minikube and I have set it up to be exposed on an https domain. After setting up the Ingress object and associated Cert-Manager/LetsEncrypt configs the certificate is appearing as not ready :

kubectl get certificate
NAME             READY   SECRET           AGE
memdump-secret   False   memdump-secret   11m

The 403 authorization error :

kubectl logs -l app=cert-manager,app.kubernetes.io/component=controller -n ingress-nginx >> certmgr.txt
I0412 11:17:27.690407       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="vaultserver.xyz" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-xxx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="memdump-secret-wjcn7-xxx-xxx" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0412 11:17:27.721978       1 sync.go:190] cert-manager/challenges "msg"="propagation check failed" "error"="wrong status code '502', expected '200'" "dnsName"="vaultserver.xyz" "resource_kind"="Challenge" "resource_name"="memdump-secret-wjcn7-xxx-xxx" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0412 11:17:37.682527       1 pod.go:59] cert-manager/challenges/http01/selfCheck/http01/ensurePod "msg"="found one existing HTTP01 solver pod" "dnsName"="vaultserver.xyz" "related_resource_kind"="Pod" "related_resource_name"="cm-acme-http-solver-xxx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="memdump-secret-wjcn7-xxx-xxx" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0412 11:17:37.682708       1 service.go:43] cert-manager/challenges/http01/selfCheck/http01/ensureService "msg"="found one existing HTTP01 solver Service for challenge resource" "dnsName"="vaultserver.xyz" "related_resource_kind"="Service" "related_resource_name"="cm-acme-http-solver-2ccnn" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="memdump-secret-wjcn7-xxx-xxx" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
I0412 11:17:37.682830       1 ingress.go:99] cert-manager/challenges/http01/selfCheck/http01/ensureIngress "msg"="found one existing HTTP01 solver ingress" "dnsName"="vaultserver.xyz" "related_resource_kind"="Ingress" "related_resource_name"="cm-acme-http-solver-xxx" "related_resource_namespace"="default" "related_resource_version"="v1" "resource_kind"="Challenge" "resource_name"="memdump-secret-wjcn7-xxx-xxx" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0412 11:17:48.843592       1 sync.go:379] cert-manager/challenges/acceptChallenge "msg"="error waiting for authorization" "error"="acme: authorization error for vaultserver.xyz: 403 urn:ietf:params:acme:error:unauthorized: 146.xxx.xxx.xxx: Invalid response from https://vaultserver.xyz/.well-known/acme-challenge/iiwVcwn0ZxIViugq_u-PeDFeMot1vlgGh0l74WlnZws: 502" "dnsName"="vaultserver.xyz" "resource_kind"="Challenge" "resource_name"="memdump-secret-wjcn7-xxx-xxx" "resource_namespace"="default" "resource_version"="v1" "type"="HTTP-01"
E0412 11:17:48.969335       1 controller.go:98] ingress 'default/cm-acme-http-solver-xxx' in work queue no longer exists
I0412 11:17:49.095940       1 conditions.go:192] Found status change for Certificate "memdump-secret" condition "Issuing": "True" -> "False"; setting lastTransitionTime to 2023-04-12 11:17:49.095929145 +0000 UTC m=+12844.737001850
I0412 11:17:49.116149       1 trigger_controller.go:179] cert-manager/certificates-trigger "msg"="Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2023-04-12 15:17:49.000001708 +0000 UTC m=+27244.641074395" "key"="default/memdump-secret"
I0412 11:17:49.151440       1 trigger_controller.go:179] cert-manager/certificates-trigger "msg"="Backing off from issuance due to previously failed issuance(s). Issuance will next be attempted at 2023-04-12 15:17:49.000002098 +0000 UTC m=+27244.641074785" "key"="default/memdump-secret"

This how my ClusterIssuer is setup :

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: myemail@gmail.com
    privateKeySecretRef:
      name: letsencrypt
    solvers:
    - http01:
        ingress:
          class: nginx
          podTemplate:
            spec:
              nodeSelector:
                "kubernetes.io/os": linux

And the Ingress object :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: memdump-ingress
  namespace: default
  annotations:
    cert-manager.io/cluster-issuer: letsencrypt
    kubernetes.io/ingress.class: "nginx"
spec:
  tls:
    - hosts:
      - vaultserver.xyz
      secretName: memdump-secret
  rules:
    - host: vaultserver.xyz
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: notesapi-clusterip-svc
                port:
                  number: 7799

I have done the following checks :

  1. Although not strictly necessary I have the domain entry mapped in /etc/hosts :

127.0.1.1 ubuntu-devops ubuntu-devops
127.0.0.1 localhost 192.xxx.xx.x jenkinserver.website vaultserver.xyz

  1. Curl the domain. This is what I am getting :curl -kv https://vaultserver.xyz
  • Trying 192.xxx.xx.x:443...
  • TCP_NODELAY set
  • Connected to vaultserver.xyz (192.xxx.xx.x) port 443 (#0)
  • ALPN, offering h2
  • ALPN, offering http/1.1
  • successfully set certificate verify locations:
  • CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs
  • TLSv1.3 (OUT), TLS handshake, Client hello (1):
  • TLSv1.3 (IN), TLS handshake, Server hello (2):
  • TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
  • TLSv1.3 (IN), TLS handshake, Certificate (11):
  • TLSv1.3 (IN), TLS handshake, CERT verify (15):
  • TLSv1.3 (IN), TLS handshake, Finished (20):
  • TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
  • TLSv1.3 (OUT), TLS handshake, Finished (20):
  • SSL connection using TLSv1.3 / TLS_AES_256_GCM_SHA384
  • ALPN, server accepted to use h2
  • Server certificate:
  • subject: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  • start date: Apr 12 07:43:48 2023 GMT
  • expire date: Apr 11 07:43:48 2024 GMT
  • issuer: O=Acme Co; CN=Kubernetes Ingress Controller Fake Certificate
  • SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
  • Using HTTP2, server supports multi-use
  • Connection state changed (HTTP/2 confirmed)
  • Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
  • Using Stream ID: 1 (easy handle 0x559c867218f0)

GET / HTTP/2 Host: vaultserver.xyz ...

  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • TLSv1.3 (IN), TLS handshake, Newsession Ticket (4):
  • old SSL session ID is stale, removing
  • Connection state changed (MAX_CONCURRENT_STREAMS == 128)! < HTTP/2 502 ...

502 Bad Gateway ...

3.Dig my domain - and the correct IP is being returned :

dig vaultserver.xyz
; <<>> DiG 9.16.1-Ubuntu <<>> vaultserver.xyz
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 25711
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 65494
;; QUESTION SECTION:
;vaultserver.xyz.               IN      A
;; ANSWER SECTION:
vaultserver.xyz.        0       IN      A       192.xxx.xx.x
;; Query time: 0 msec
;; SERVER: 127.0.0.53#53(127.0.0.53)
;; WHEN: Wed Apr 12 11:25:04 UTC 2023
;; MSG SIZE  rcvd: 60

All components are also up :

kubectl -n ingress-nginx get pods
NAME                                       READY   STATUS      RESTARTS      AGE
cert-manager-xxx              1/1     Running     2 (27m ago)   7d19h
cert-manager-cainjector-xxx   1/1     Running     1             7d19h
cert-manager-webhook-xxx       1/1     Running     2             7d19h
ingress-nginx-admission-create-xxx      0/1     Completed   0             17d
ingress-nginx-admission-patch-xxx       0/1     Completed   1             17d
ingress-nginx-controller-xxx  1/1     Running     2 (27m ago)   17d

What am I missing ?

2

Your entire site is down and returning 502 bad gateway-errors. As is also the case for the validation attempt:

(Also, why would you obfuscate the IP address where everybody can just simply use DNS to resolve your hostname to that same IP address?)

More than the above I cannot say, as I don't have any experience with Ingress/Minikube et cetera.

3 Likes
3

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.