403 GoogleCloud API failed

Help
1

My domain is:
zero-sum-seattle.net

I ran this command:
kubectl describe challenge -n
It produced this output:
Warning PresentError 8m27s (x12 over 110m) cert-manager-challenges Error presenting challenge: GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden

My web server is (include version):
nginx-ingress (GKE pod)

The operating system my web server runs on is (include version):
uh whatever nginx-ingress is

My hosting provider, if applicable, is:
GCP

I can login to a root shell on my machine (yes or no, or I don't know):
Yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):
eh I'm getting there :slight_smile: I have access to my terraform and GCP console

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot):
cert-manager-controller:v1.9.1

Hey guys! I am having an odd issue and have wasted so much time trying to figure this out..
I am currently using let's encrypt/cert-manager to handle the provisioning TLS/SSL certificates for my domain zero-sum-seattle.net. I love the product BTW, been using it for years!

I am setting up my own GCP project for a python project I am using. It's running a GKE cluster and what not. I have set up and used cert-manager to provision two certificates from a clusterissuer already for this domain. The issue I am running into started when I tried to deploy redmine to manage tickets for my little OS group. At first I thought it was an issue with the redmine helm chart, but it turned out to be false. I know this because I just created my own certificate manifest and tried to create a certificate that way, and also received a 403. However, I changed the manifest to use my nginx-ingress A record, and then the certificate provisoned away, so I thought to try my staging issuer for redmine and same error. I have WLI setup and a service account with dns.admin. I can link to my terraform.

I don't know if I'm overlooking some quota or rate limit issue with let's encrypt, but I thought that would be solved with switching to staging.

Here are some cert-manager pod logs:
Success!

I0914 21:28:01.728920       1 sync.go:257] cert-manager/orders "msg"="build set of domains for Order" "domains"=["nginx-ingress.zero-sum-seattle.net"] "resource_kind"="Order" "resource_name"="redmine-cert-zgj8j-2411455687" "resource_namespace"="jenkins" "resource_version"="v1"
I0914 21:28:01.729046       1 sync.go:260] cert-manager/orders "msg"="build set of IPs for Order" "domains"=["nginx-ingress.zero-sum-seattle.net"] "resource_kind"="Order" "resource_name"="redmine-cert-zgj8j-2411455687" "resource_namespace"="jenkins" "resource_version"="v1"
I0914 21:28:02.068073       1 logs.go:177] cert-manager/controller "msg"="Event(v1.ObjectReference{Kind:\"Order\", Namespace:\"jenkins\", Name:\"redmine-cert-zgj8j-2411455687\", UID:\"a21dcdba-138b-4520-80e0-f2350db0978f\", APIVersion:\"acme.cert-manager.io/v1\", ResourceVersion:\"1846068\", FieldPath:\"\"}): type: 'Normal' reason: 'Created' Created Challenge resource \"redmine-cert-zgj8j-2411455687-721461229\" for domain \"nginx-ingress.zero-sum-seattle.net\""
I0914 21:28:02.179396       1 sync.go:682] cert-manager/orders "msg"="Retrieved ACME order from server" "raw_data"={"URI":"","Status":"pending","Expires":"2022-09-21T21:28:01Z","Identifiers":[{"Type":"dns","Value":"nginx-ingress.zero-sum-seattle.net"}],"NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","AuthzURLs":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/153414393827"],"FinalizeURL":"https://acme-v02.api.letsencrypt.org/acme/finalize/729354887/125525301747","CertURL":"","Error":null} "resource_kind"="Order" "resource_name"="redmine-cert-zgj8j-2411455687" "resource_namespace"="jenkins" "resource_version"="v1"
I0914 21:28:02.283266       1 sync.go:682] cert-manager/orders "msg"="Retrieved ACME order from server" "raw_data"={"URI":"","Status":"pending","Expires":"2022-09-21T21:28:01Z","Identifiers":[{"Type":"dns","Value":"nginx-ingress.zero-sum-seattle.net"}],"NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","AuthzURLs":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/153414393827"],"FinalizeURL":"https://acme-v02.api.letsencrypt.org/acme/finalize/729354887/125525301747","CertURL":"","Error":null} "resource_kind"="Order" "resource_name"="redmine-cert-zgj8j-2411455687" "resource_namespace"="jenkins" "resource_version"="v1"
I0914 21:28:02.350366       1 dns.go:219] cert-manager/challenges/Present/solverForChallenge "msg"="preparing to create CloudDNS provider" "dnsName"="nginx-ingress.zero-sum-seattle.net" "domain"="nginx-ingress.zero-sum-seattle.net" "resource_kind"="Challenge" "resource_name"="redmine-cert-zgj8j-2411455687-721461229" "resource_namespace"="jenkins" "resource_version"="v1" "type"="DNS-01"
I0914 21:28:02.350563       1 dns.go:102] cert-manager/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="nginx-ingress.zero-sum-seattle.net" "domain"="nginx-ingress.zero-sum-seattle.net" "resource_kind"="Challenge" "resource_name"="redmine-cert-zgj8j-2411455687-721461229" "resource_namespace"="jenkins" "resource_version"="v1" "type"="DNS-01"
I0914 21:28:02.393100       1 sync.go:682] cert-manager/orders "msg"="Retrieved ACME order from server" "raw_data"={"URI":"","Status":"pending","Expires":"2022-09-21T21:28:01Z","Identifiers":[{"Type":"dns","Value":"nginx-ingress.zero-sum-seattle.net"}],"NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","AuthzURLs":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/153414393827"],"FinalizeURL":"https://acme-v02.api.letsencrypt.org/acme/finalize/729354887/125525301747","CertURL":"","Error":null} "resource_kind"="Order" "resource_name"="redmine-cert-zgj8j-2411455687" "resource_namespace"="jenkins" "resource_version"="v1"
I0914 21:28:02.433655       1 wait.go:383] Returning discovered zone record "zero-sum-seattle.net." for fqdn "_acme-challenge.nginx-ingress.zero-sum-seattle.net."
I0914 21:28:02.487401       1 sync.go:682] cert-manager/orders "msg"="Retrieved ACME order from server" "raw_data"={"URI":"","Status":"pending","Expires":"2022-09-21T21:28:01Z","Identifiers":[{"Type":"dns","Value":"nginx-ingress.zero-sum-seattle.net"}],"NotBefore":"0001-01-01T00:00:00Z","NotAfter":"0001-01-01T00:00:00Z","AuthzURLs":["https://acme-v02.api.letsencrypt.org/acme/authz-v3/153414393827"],"FinalizeURL":"https://acme-v02.api.letsencrypt.org/acme/finalize/729354887/125525301747","CertURL":"","Error":null} "resource_kind"="Order" "resource_name"="redmine-cert-zgj8j-2411455687" "resource_namespace"="jenkins" "resource_version"="v1"
I0914 21:28:02.541517       1 dns.go:219] cert-manager/challenges/Present/solverForChallenge "msg"="preparing to create CloudDNS provider" "dnsName"="nginx-ingress.zero-sum-seattle.net" "domain"="nginx-ingress.zero-sum-seattle.net" "resource_kind"="Challenge" "resource_name"="redmine-cert-zgj8j-2411455687-721461229" "resource_namespace"="jenkins" "resource_version"="v1" "type"="DNS-01"
I0914 21:28:02.541659       1 dns.go:102] cert-manager/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="nginx-ingress.zero-sum-seattle.net" "domain"="nginx-ingress.zero-sum-seattle.net" "resource_kind"="Challenge" "resource_name"="redmine-cert-zgj8j-2411455687-721461229" "resource_namespace"="jenkins" "resource_version"="v1" "type"="DNS-01"

Failed:

I0915 00:48:41.857793       1 leaderelection.go:278] successfully renewed lease kube-system/cert-manager-controller
I0915 00:48:42.039910       1 controller.go:153] cert-manager/challenges "msg"="syncing item" "key"="redmine/redmine-tls-pqjh7-1084067888-1050907113"
I0915 00:48:42.040018       1 dns.go:219] cert-manager/challenges/Present/solverForChallenge "msg"="preparing to create CloudDNS provider" "dnsName"="rm.zero-sum-seattle.net" "domain"="rm.zero-sum-seattle.net" "resource_kind"="Challenge" "resource_name"="redmine-tls-pqjh7-1084067888-1050907113" "resource_namespace"="redmine" "resource_version"="v1" "type"="DNS-01"
I0915 00:48:42.040087       1 dns.go:102] cert-manager/challenges/Present "msg"="presenting DNS01 challenge for domain" "dnsName"="rm.zero-sum-seattle.net" "domain"="rm.zero-sum-seattle.net" "resource_kind"="Challenge" "resource_name"="redmine-tls-pqjh7-1084067888-1050907113" "resource_namespace"="redmine" "resource_version"="v1" "type"="DNS-01"
I0915 00:48:42.040095       1 wait.go:329] Returning cached zone record "zero-sum-seattle.net." for fqdn "_acme-challenge.rm.zero-sum-seattle.net."
E0915 00:48:42.117506       1 controller.go:166] cert-manager/challenges "msg"="re-queuing item due to error processing" "error"="GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden" "key"="redmine/redmine-tls-pqjh7-1084067888-1050907113"
I0915 00:48:42.117654       1 logs.go:177] cert-manager/controller "msg"="Event(v1.ObjectReference{Kind:\"Challenge\", Namespace:\"redmine\", Name:\"redmine-tls-pqjh7-1084067888-1050907113\", UID:\"fd860839-99e3-497b-bdf3-d8fb2016884b\", APIVersion:\"acme.cert-manager.io/v1\", ResourceVersion:\"1946468\", FieldPath:\"\"}): type: 'Warning' reason: 'PresentError' Error presenting challenge: GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden"

apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: redmine-tls
  namespace: redmine
spec:
  dnsNames:
    - rm.zero-sum-seattle.net
  secretName: rm.zero-sum-seattle.net
  issuerRef:
    kind: ClusterIssuer
    name: letsencrypt-prod

apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    server: https://acme-v02.api.letsencrypt.org/directory
    email: spahmatthew@gmail.com
    privateKeySecretRef:
      name: letsencrypt-prod
    solvers:
      - selector: {}
        dns01:
          cloudDNS:
            project: long-flame-659


Did I possibly hit a rate limit? 
I checked my Google Quotas and I haven't hit any limit. I'm tired of staring at this please assist!
2

I don't know anything about this type of configuration but google suggests :

See also GoogleCloud API call failed: googleapi: Error 403: Forbidden, forbidden · Issue #2630 · cert-manager/cert-manager · GitHub

2 Likes
3

Thanks, Webprofusion. I already read this stack post. Both my staging and prod issuers are in the same project. This is all isolated to just one project. I honestly don't know what is wrong :frowning:

I am setting out a good logging sink tonight, and I hope the API logs shine some light on this.

1 Like
4

I'd guess by the 403 that the api secret you have is either incorrect or it doesn't have the required permissions (like listing zones), maybe double check it's the same as your working account and has permissions for the correct zone: Google CloudDNS - cert-manager Documentation

1 Like
5

You are correct, however, I haven't found a solution yet. This is the call that is failing and returning a 403: cloud.dns.api.v1.ManagedZonesService.List

6

It could be they will all fail and it's just the zone list is just the first api call it hits. I think you might have more luck if there is a dedicated cert-manager community somewhere? It does come up here, but not so often.

I presume you are just following the ClusterIssuer parts of Google CloudDNS - cert-manager Documentation and have already configured the project IAM links. Could it be a namespace thing?

1 Like
7

Thank you for your help! I am currently doing some googling and going through this CloudDNS guide again. I have been sucked into this issue so maybe I'm overlooking something..

I thought it might be a namespace issue, but what would be the issue? This is a cluster issuer that shouldn't matter.

It really threw me off when it provisioned a certificate for nginx-ingress.zero-sum-seattle.net. If you aren't going to work at least be consistent. :slight_smile:

1 Like
8

I fixed it. I must have been really tired and forgot to annotate the service account. I'm guessing between all the things I have been doing to debug the Google API used the service account key instead of workload identity and provisioned the certificate.

Use Workload Identity  |  Kubernetes Engine Documentation  |  Google Cloud helped a ton to spot my mistake.

This worked fine for the IAM account: gcloud dns managed-zones list --impersonate-service-account

However, you can't forget this step:

kubectl annotate serviceaccount KSA_NAME \
    --namespace NAMESPACE \
    iam.gke.io/gcp-service-account=GSA_NAME@GSA_PROJECT.iam.gserviceaccount.com

Thanks for linking me that google cloud guide. That helped a lot! I can sleep good tonight! :smiley:

2 Likes
9

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.