Cert-manager + istio + GCP = unable to get local issuer certificate (20)

Help
1

Hello.

I try to get Let'sEncrypt wildcard certificate for my domain and use it in GCP HTTP LB. I used your article for that.
The article fully describes my use case. But during the following/debugging I've got some problems and made my modifications.
Finally my config looks like:

---
apiVersion: cert-manager.io/v1
kind: ClusterIssuer
metadata:
  name: letsencrypt-cloudflare-issuer
  namespace: istio-ingress
spec:
  acme:
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    email: it@example.com
    privateKeySecretRef:
      name: clusterissuer-account-key
    solvers:
    - dns01:
        cloudflare:
          email: it@example.com
          apiTokenSecretRef:
            name: cloudflare-api-token-secret
            key: api-token


---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: wildcard-dev-uacaus-com-crt
  namespace: istio-ingress
spec:
  secretName: letsencrypt-secret-wildcard
  issuerRef: 
    kind: ClusterIssuer
    name: letsencrypt-cloudflare-issuer
  dnsNames:
    - "*.dev.example.com"
    - dev.example.com

---
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: istio-ingress-dev
  namespace: istio-ingress
  annotations:
    kubernetes.io/ingress.class: gce
    kubernetes.io/ingress.allow-http: "true"
    kubernetes.io/ingress.global-static-ip-name: istio-ingress-gclb-dev
    cert-manager.io/cluster-issuer: letsencrypt-cloudflare-issuer
spec:
  tls:
    - secretName: letsencrypt-secret-wildcard
      hosts:
        - "*.dev.example.com"
        - "dev.example.com"
  defaultBackend:
    service:
      name: istio-ingress
      port:
        number: 80

---
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: istio-gateway-dev
  namespace: istio-ingress
spec:
  selector:
    istio: ingress
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
      - "*.dev.example.com"

---
apiVersion: cloud.google.com/v1
kind: BackendConfig
metadata:
  name: istio-ingress-dev-backendconfig
  namespace: istio-ingress
spec:
  timeoutSec: 120
  healthCheck:
    checkIntervalSec: 10
    timeoutSec: 2
    port: 15021
    type: HTTP
    requestPath: /healthz/ready

As the result I have wildcard certificate, Google LB got and start use it, and everything looks good. Only one thing that I can't fix:
SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
Test curl request:

curl https://nginx-dev.dev.example.com -vkI
*   Trying 34.110.169.139:443...
* Connected to nginx-dev.dev.example.com (34.110.169.139) port 443 (#0)
* ALPN: offers h2
* ALPN: offers http/1.1
* (304) (OUT), TLS handshake, Client hello (1):
* (304) (IN), TLS handshake, Server hello (2):
* (304) (IN), TLS handshake, Unknown (8):
* (304) (IN), TLS handshake, Certificate (11):
* (304) (IN), TLS handshake, CERT verify (15):
* (304) (IN), TLS handshake, Finished (20):
* (304) (OUT), TLS handshake, Finished (20):
* SSL connection using TLSv1.3 / AEAD-AES256-GCM-SHA384
* ALPN: server accepted h2
* Server certificate:
*  subject: CN=*.dev.example.com
*  start date: Mar 25 14:23:09 2023 GMT
*  expire date: Jun 23 14:23:08 2023 GMT
*  issuer: C=US; O=(STAGING) Let's Encrypt; CN=(STAGING) Artificial Apricot R3
*  SSL certificate verify result: unable to get local issuer certificate (20), continuing anyway.
* Using HTTP2, server supports multiplexing
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* h2h3 [:method: HEAD]
* h2h3 [:path: /]
* h2h3 [:scheme: https]
* h2h3 [:authority: nginx-dev.dev.example.com]
* h2h3 [user-agent: curl/7.86.0]
* h2h3 [accept: */*]
* Using Stream ID: 1 (easy handle 0x7f7d5b811e00)
> HEAD / HTTP/2
> Host: nginx-dev.dev.example.com
> user-agent: curl/7.86.0
> accept: */*
> 
< HTTP/2 200 
HTTP/2 200 
< server: istio-envoy
server: istio-envoy
< date: Sat, 25 Mar 2023 20:51:28 GMT
date: Sat, 25 Mar 2023 20:51:28 GMT
< content-type: text/html
content-type: text/html
< content-length: 615
content-length: 615
< last-modified: Tue, 13 Dec 2022 15:53:53 GMT
last-modified: Tue, 13 Dec 2022 15:53:53 GMT
< etag: "6398a011-267"
etag: "6398a011-267"
< accept-ranges: bytes
accept-ranges: bytes
< x-envoy-upstream-service-time: 1
x-envoy-upstream-service-time: 1
< via: 1.1 google
via: 1.1 google
< alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000
alt-svc: h3=":443"; ma=2592000,h3-29=":443"; ma=2592000

< 
* Connection #0 to host nginx-dev.dev.example.com left intact

Please, give me any advice!

2

You're using the staging environment, which is good for testing indeed. However, that also means the root certificate of the resulting staging cert chain isn't recognised by anything, which is also the way it should be. (You don't want the staging environment root in your root stores!)

If you're done testing and want a production certificate, change the ACME server from testing to production.

5 Likes
4

Fake/Staging environment:

Fake/Test domain:

It you want it "to work", switch to the production environment and an FQDN [from a valid domain].
Although, it is very likely that a real domain was being used and you just obfuscated it here.
[How else would Cloudflare have been involved and IP 34.110.169.139 have been tested? - LOL]

4 Likes
5

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.