Failed to Interact with Lets Encrypt with DNS

Hi Everyone, It seems that the previous post became a little bit chaotic, so i was thinking maybe its better to put everything in one post it will be more clear. Any help is welcome ...

So we are using Azure Firewall Premium and all outbound connectivity's are configure including also Lets Encrypt endpoint as URL filter:
In first case on Azure Firewall when we leave the DNS section as Default Azure Provided we get this error on Traefik pod:
"x509: certificate signed by unknown authority"

/ $ wget https://acme-v02.api.letsencrypt.org/directory Connecting to acme-v02.api.letsencrypt.org (172.65.46.172:443) ssl_client: acme-v02.api.letsencrypt.org: certificate verification failed: self signed certificate in certificate chain.

This is what I get in first case with wget command

In second case when we update our settings on Azure Firewall with our Custom Dns Ip of Simply.com provider we get this error as above:

dial tcp: lookup acme-v02.api.letsencrypt.org on 10.41.0.10:53: server misbehaving
This private ip is our AKS dns service ip

I am unable to run this command inside Traefik pod inside my cluster, looks like I don't have also permissions to install packages.
/ $ whoami
whoami: unknown uid 65532
/ $ echo | openssl s_client -connect acme-v02.api.letsencrypt.org:443 | head
sh: openssl: not found

This is our Traefik script

traefik:
deployment:
initContainers:

  • name: volume-permisions
    image: busybox:1.31.1
    command: ["sh", "-c", "chmod -Rv 600 /data/*"]
    volumeMounts:
  • name: data
    mountPath: /data

service:
annotations:
service.beta.kubernetes.io/azure-dns-label-name: xxx
service.beta.kubernetes.io/azure-load-balancer-internal: "true"
service.beta.kubernetes.io/azure-load-balancer-internal-subnet: "loadbalancer"

certResolvers:
le:
email: system@xxx.com
tlsChallenge: true
storage: "/data/acme.json"
caServer: https://acme-v02.api.letsencrypt.org/directory

persistence:
enabled: true

ingressClass:
enabled: true
isDefaultClass: true

providers:
kubernetesCRD:
allowExternalNameServices: true

kubernetesIngress:
  publishedService:
    enabled: true

ports:
web:
redirectTo: websecure
websecure:
tls:
enabled: true
certResolver: "le"
domains:

Thank you in advance!

I don't know that the prior thread was messy and I had to go review it anyway before responding so this is actually harder.

Your system is not making proper outbound requests. And, you have limited tools to debug (like no openssl). It is likely something in your firewall like https inspection causing trouble.

What does this do

wget https://cloudflare.com
3 Likes

Seems like fixing DNS should be your first concern.

It would be nice if we could see the cert found.

3 Likes

@MikeMcQ we get this :

wget https://cloudflare.com
Connecting to cloudflare.com (104.16.132.229:443)
ssl_client: cloudflare.com: certificate verification failed: self signed certificate in certificate chain
wget: error getting response: Connection reset by peer

cloudflare.com serves a completely different chain than the Let's Encrypt ACME server. To me, it sounds like something else is using a self signed certificate (either directly or as private root) to hijack the outgoing connections like a Man-in-the-Middle attack.

It's unfortunate you don't have the openssl binary to debug this issue further. Also, the wget is probably not the "real" wget, but just busybox, so that also doesn't have much to work with probably.

1 Like

Try: wget --help and look for "verbose" and "debug" flags.

The command is likely:

wget -v -d https://acme-v02.api.letsencrypt.org/directory

If you have wget via busybox as @Osiris suggested, I don't think the debug options are available.

The debug info should show the certificate for the connection.

3 Likes

wget is verbose by default and debugging needs to be enabled at compile time.. So while it might work, chances are even with a regular wget, debugging info won't show up.

3 Likes

You're right about the verbose levels. I tried the debug flag on a few FULL operating systems I had access to, and it was supported on all. I expect micro OS's to be different and not support it.

4 Likes

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.