Setup: Using cert-manager with nginx-ingress which creates a http-solver to serve as http01 challenge provider. As per my debugging so far, cert manager is working as expected but LetsEncrypt is giving an error while verifying domain.
Error reported is: acme: authorization for identifier c.inspiredocs.com is invalid
Relevant logs: I checked the logs to find that LetsEncrypt is able to access the http-solver and get 200 OK. Logs from nginx:
Do you mean the 200OK snippet?
cert-manager performs a cleanup on the http-solver it created after LetsEncrypt reports an error (which is considered as final state) and deletes the ingress rules as well as the solver instance so currently it is reporting 404.
The 308: Are there some internal redirects?
I have checked all my configs but haven't found any 308 redirects, there might be a possibility of 302 but no 308s. Also, it is weird that 308 occurs before 200 in the logs.
When no routes match the current request nginx will forward it to default backend which is a server running to process such requests. For acme-request the route should exist so it should not take this path when the http-solver server is running.
Or is there a better log?
Below I have attached detailed log from cert-manager keeping the relavant parts to this conversation. Basically it does the following:
Creates a certificate (self signed)
Creates an order using acme API
Creates a challenge
Creates a http-solver (which should provide endpoint to resolve http01 challenge)
Wait for the http-solver endpoint to return 200
Sync changes with acme api
Wait for authorization from acme api
Hope this can give some information regarding what the acme-client is doing.
I0515 17:37:34.743659 1 sync.go:144] Invoking issue function as existing certificate does not exist
I0515 17:37:34.743685 1 issue.go:257] Attempting to fetch existing certificate private key
I0515 17:37:34.743696 1 issue.go:276] Generating new private key for default/c-secret
I0515 17:37:34.972516 1 issue.go:61] Storing new certificate private key for default/c-secret
I0515 17:37:35.126748 1 issue.go:288] Creating new Order resource for Certificate default/c-secret
I0515 17:37:35.144796 1 issue.go:311] Created new Order resource named "c-secret-2745897455" for Certificate default/c-secret
I0515 17:37:35.150686 1 controller.go:184] orders controller: syncing item 'default/c-secret-2745897455'
I0515 17:37:35.152778 1 logger.go:38] Calling CreateOrder
I0515 17:37:35.156139 1 controller.go:170] Event(v1.ObjectReference{Kind:"Certificate", Namespace:"default", Name:"c-secret", UID:"1fd266b8-7738-11e9-ba2c-42010aa00044", APIVersion:"certmanager.k8s.io/v1alpha1", ResourceVersion:"12984093", FieldPath:""}): type: 'Normal' reason: 'OrderCreated' Created Order resource "c-secret-2745897455"
I0515 17:37:35.198893 1 issue.go:160] Order default/c-secret-2745897455 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0515 17:37:36.543579 1 logger.go:73] Calling GetAuthorization
I0515 17:37:36.851268 1 logger.go:93] Calling HTTP01ChallengeResponse
I0515 17:37:36.866302 1 sync.go:274] Need to create 1 challenges
I0515 17:37:36.876074 1 sync.go:323] Waiting for all challenges for order "c-secret-2745897455" to enter 'valid' state
I0515 17:37:36.876144 1 controller.go:170] Event(v1.ObjectReference{Kind:"Order", Namespace:"default", Name:"c-secret-2745897455", UID:"20109545-7738-11e9-ba2c-42010aa00044", APIVersion:"certmanager.k8s.io/v1alpha1", ResourceVersion:"12984106", FieldPath:""}): type: 'Normal' reason: 'Created' Created Challenge resource "c-secret-2745897455-0" for domain "c.inspiredocs.com"
I0515 17:37:36.876381 1 sync.go:323] Waiting for all challenges for order "c-secret-2745897455" to enter 'valid' state
I0515 17:37:37.664275 1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/c-secret-2745897455-0". One will be created.
I0515 17:37:37.689323 1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/c-secret-2745897455-0". One will be created.
I0515 17:37:37.708388 1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=993003170,certmanager.k8s.io/acme-http-token=1092030118
I0515 17:37:37.708851 1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/c-secret-2745897455-0". One will be created.
I0515 17:37:37.856680 1 sync.go:176] propagation check failed: wrong status code '404', expected '200'
I0515 17:37:37.863502 1 controller.go:184] orders controller: syncing item 'default/c-secret-2745897455'
I0515 17:37:37.889923 1 sync.go:176] propagation check failed: wrong status code '404', expected '200'
I0515 17:37:58.008773 1 sync.go:269] Accepting challenge for domain "c.inspiredocs.com"
I0515 17:37:58.008849 1 logger.go:63] Calling AcceptChallenge
I0515 17:37:58.362886 1 sync.go:286] Waiting for authorization for domain "c.inspiredocs.com"
I0515 17:37:58.362953 1 logger.go:78] Calling WaitAuthorization
I0515 17:38:00.629909 1 controller.go:170] Event(v1.ObjectReference{Kind:"Challenge", Namespace:"default", Name:"c-secret-2745897455-0", UID:"21191cc1-7738-11e9-ba2c-42010aa00044", APIVersion:"certmanager.k8s.io/v1alpha1", ResourceVersion:"12984126", FieldPath:""}): type: 'Warning' reason: 'Failed' Accepting challenge authorization failed: acme: authorization for identifier c.inspiredocs.com is invalid
I0515 17:38:00.639819 1 logger.go:43] Calling GetOrder
I0515 17:38:00.950203 1 sync.go:416] Error issuing certificate for default/c-secret: applying acme order back-off for certificate default/c-secret because it has failed within the last 1h0m0s
Are there other options like “manual”?
One solution can be to manually launch a server to fulfil the requirement of http-solve so that we can confirm the hypothesis that http-solver is indeed causing the problem. If we are not able to reach any solution from above then I will give this a go
looks like the tool tests the own webserver and has the wrong http status 404.
So this
I0515 17:37:37.708851 1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/c-secret-2745897455-0". One will be created.
looks like the tool tests the own webserver and has the wrong http status 404.
At this point it is just waiting for the http-solver server to startup. Initially, since the server hasn't started it returns 404 from default backend and it keeps on polling until it is 200. Only after it receives a 200OK from the solver does it proceed to accept the challenge.
However, I get your point. I will try with the manual process and check if I can succeed. Will get back to you after this.
I have found out the issue. The issue was with user-agent check setup. It allowed only chrome and for others it reported 403.
I am not sure why and how the 403 was getting interpreted by letsencrypt as 404 but after fixing the user-agent to allow for lets’encrypt validation server’s user agent, which is Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org), the problem was resolved.