Environment:
Gitlab AutoDevops
Google Kubernetes Service 1.12
User nginx-igress-controller and cert-bot for kubernetes version 0.5.2
Issue:
I push the code from Gitlab it deploys on Staging and then production. Both jobs are going successful without any errors but when you load production website its redirects to default-backend not to actual pod.
Since when: For last 2 weeks nothing we change in our setup
Error:
When I go into the logs in the nginx -ingress-controller textPayload: "10.138.0.52 - [10.138.0.52] - - [13/Oct/2019:13:06:33 +0000] "GET /.well-known/acme-challenge/5u89-Af72fyhOiVZiuLBaFf_KtQlydbOcluM4qZAohg HTTP/1.1" 404 21 "-" "Go-http-client/1.1" 205 0.000 [upstream-default-backend] 10.4.1.30:8080 21 0.000 404
Bear in mind that staging is still working fine. I am using real certificates for even staging not the fake one. So I dont understand what is the problem but if you go to the website its properly loading the certificate.
It looks like a preflight request from cert-manager (or "self check" in its language). Since it is a 404, the order is never submitted to Let's Encrypt.
Are you sure that you're using Certbot? To me, it looks like you are probably using cert-manager.
I think it's going to be tough for you to get an answer for this problem on this forum - there's a lot of non-Let's Encrypt moving parts in GKE + nginx-ingress-controller + cert-manager. You might have better luck asking in the respective communities of those projects.
If you’re running cert-manager 0.5.2, you should probably upgrade. I don’t know if it would help with this issue, but there have been other important improvements.