Facing 403 forbidden during domain authorization by LetsEncrypt via http01

#1

Domain: c.inspiredocs.com

Setup: Using cert-manager with nginx-ingress which creates a http-solver to serve as http01 challenge provider. As per my debugging so far, cert manager is working as expected but LetsEncrypt is giving an error while verifying domain.

Error reported is: acme: authorization for identifier c.inspiredocs.com is invalid

Relevant logs: I checked the logs to find that LetsEncrypt is able to access the http-solver and get 200 OK. Logs from nginx:

10.160.0.12 - [10.160.0.12] - - [15/May/2019:13:42:18 +0000] "GET /.well-known/acme-challenge/GhXJLK_yc4mKQTnQQHBJsKiReIjWoX_tTgtwRLf-zgY HTTP/1.1" 200 895 "http://inspiredocs.com/.well-known/acme-challenge/GhXJLK_yc4mKQTnQQHBJsKiReIjWoX_tTgtwRLf-zgY" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 371 0.000 [staging-static-content-80] 10.4.1.95:80 2736 0.001 200 495f589a62ae1adcc9abbb226f1064d1

Along with these, there was also some status 308 logs before 200 OK though I am not sure why:

10.160.0.12 - [10.160.0.12] - - [15/May/2019:13:42:18 +0000] "GET /.well-known/acme-challenge/GhXJLK_yc4mKQTnQQHBJsKiReIjWoX_tTgtwRLf-zgY HTTP/1.1" 308 171 "-" "Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org)" 267 0.000 [staging-static-content-80] - - - - 82b8739888fad4a34ddfdd050b0c821c

Additionally, logs from acme-api is:

  "error": {
    "type": "urn:ietf:params:acme:error:unauthorized",
    "detail": "Invalid response from http://c.inspiredocs.com/.well-known/acme-challenge/kIGIf0SMd5NZqtnm6TcgXClkDr-zx78qhTUi9SAlpX4 [35.200.128.101]: 404",
    "status": 403
  },

My web server is: nginx 1.15.8

Setup is running on kubernetes GKE (v1.12.7-gke.10)

Let me know if any more information is required. Any help would be greatly appreciated.

#2

Hi @lahsivjar

the 403 isn’t the correct error. There is a http status 404 - Not Found reported

"detail": "Invalid response from 
http://c.inspiredocs.com/.well-known/acme-challenge/kIGIf0SMd5NZqtnm6TcgXClkDr-zx78qhTUi9SAlpX4 
[35.200.128.101]: 404"

Checking your domain there is a “default backend” ( https://check-your-website.server-daten.de/?q=c.inspiredocs.com ):

Domainname Http-Status redirect Sec. G
http://c.inspiredocs.com/
35.200.128.101 404 0.737 M
Not Found
http://www.c.inspiredocs.com/
35.200.128.101 200 0.747 H
https://c.inspiredocs.com/
35.200.128.101 404 3.310 N
Not Found
Certificate error: RemoteCertificateChainErrors
https://www.c.inspiredocs.com/
35.200.128.101 404 3.320 N
Not Found
Certificate error: RemoteCertificateNameMismatch, RemoteCertificateChainErrors
http://c.inspiredocs.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
35.200.128.101 404 0.730 A
Not Found
Visible Content: default backend - 404
http://www.c.inspiredocs.com/.well-known/acme-challenge/check-your-website-dot-server-daten-dot-de
35.200.128.101 200 0.740
Visible Content: Sorry, currently we only support web versions of chrome

Which instance creates that snippet?

The 308: Are there some internal redirects?

#3

Hi Juergen,

Thank you for the prompt response.

Which instance creates that snippet?

Do you mean the 200OK snippet?
cert-manager performs a cleanup on the http-solver it created after LetsEncrypt reports an error (which is considered as final state) and deletes the ingress rules as well as the solver instance so currently it is reporting 404.

The 308: Are there some internal redirects?

I have checked all my configs but haven’t found any 308 redirects, there might be a possibility of 302 but no 308s. Also, it is weird that 308 occurs before 200 in the logs.

#4

No, the

default backend - 404

snippet.

The problem is simple: This

looks like a “closed world”. So I have no idea what this ACME-client is doing.

Are there other options like “manual”? Or is there a better log?

#5

default backend - 404

When no routes match the current request nginx will forward it to default backend which is a server running to process such requests. For acme-request the route should exist so it should not take this path when the http-solver server is running.

Or is there a better log?

Below I have attached detailed log from cert-manager keeping the relavant parts to this conversation. Basically it does the following:

  1. Creates a certificate (self signed)
  2. Creates an order using acme API
  3. Creates a challenge
  4. Creates a http-solver (which should provide endpoint to resolve http01 challenge)
  5. Wait for the http-solver endpoint to return 200
  6. Sync changes with acme api
  7. Wait for authorization from acme api

Hope this can give some information regarding what the acme-client is doing.

I0515 17:37:34.743659       1 sync.go:144] Invoking issue function as existing certificate does not exist
I0515 17:37:34.743685       1 issue.go:257] Attempting to fetch existing certificate private key
I0515 17:37:34.743696       1 issue.go:276] Generating new private key for default/c-secret
I0515 17:37:34.972516       1 issue.go:61] Storing new certificate private key for default/c-secret
I0515 17:37:35.126748       1 issue.go:288] Creating new Order resource for Certificate default/c-secret
I0515 17:37:35.144796       1 issue.go:311] Created new Order resource named "c-secret-2745897455" for Certificate default/c-secret
I0515 17:37:35.150686       1 controller.go:184] orders controller: syncing item 'default/c-secret-2745897455'
I0515 17:37:35.152778       1 logger.go:38] Calling CreateOrder
I0515 17:37:35.156139       1 controller.go:170] Event(v1.ObjectReference{Kind:"Certificate", Namespace:"default", Name:"c-secret", UID:"1fd266b8-7738-11e9-ba2c-42010aa00044", APIVersion:"certmanager.k8s.io/v1alpha1", ResourceVersion:"12984093", FieldPath:""}): type: 'Normal' reason: 'OrderCreated' Created Order resource "c-secret-2745897455"
I0515 17:37:35.198893       1 issue.go:160] Order default/c-secret-2745897455 is not in 'valid' state. Waiting for Order to transition before attempting to issue Certificate.
I0515 17:37:36.543579       1 logger.go:73] Calling GetAuthorization
I0515 17:37:36.851268       1 logger.go:93] Calling HTTP01ChallengeResponse
I0515 17:37:36.866302       1 sync.go:274] Need to create 1 challenges
I0515 17:37:36.876074       1 sync.go:323] Waiting for all challenges for order "c-secret-2745897455" to enter 'valid' state
I0515 17:37:36.876144       1 controller.go:170] Event(v1.ObjectReference{Kind:"Order", Namespace:"default", Name:"c-secret-2745897455", UID:"20109545-7738-11e9-ba2c-42010aa00044", APIVersion:"certmanager.k8s.io/v1alpha1", ResourceVersion:"12984106", FieldPath:""}): type: 'Normal' reason: 'Created' Created Challenge resource "c-secret-2745897455-0" for domain "c.inspiredocs.com"
I0515 17:37:36.876381       1 sync.go:323] Waiting for all challenges for order "c-secret-2745897455" to enter 'valid' state
I0515 17:37:37.664275       1 pod.go:64] No existing HTTP01 challenge solver pod found for Certificate "default/c-secret-2745897455-0". One will be created.
I0515 17:37:37.689323       1 service.go:51] No existing HTTP01 challenge solver service found for Certificate "default/c-secret-2745897455-0". One will be created.
I0515 17:37:37.708388       1 ingress.go:49] Looking up Ingresses for selector certmanager.k8s.io/acme-http-domain=993003170,certmanager.k8s.io/acme-http-token=1092030118
I0515 17:37:37.708851       1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge "default/c-secret-2745897455-0". One will be created.
I0515 17:37:37.856680       1 sync.go:176] propagation check failed: wrong status code '404', expected '200'
I0515 17:37:37.863502       1 controller.go:184] orders controller: syncing item 'default/c-secret-2745897455'
I0515 17:37:37.889923       1 sync.go:176] propagation check failed: wrong status code '404', expected '200'
I0515 17:37:58.008773       1 sync.go:269] Accepting challenge for domain "c.inspiredocs.com"
I0515 17:37:58.008849       1 logger.go:63] Calling AcceptChallenge
I0515 17:37:58.362886       1 sync.go:286] Waiting for authorization for domain "c.inspiredocs.com"
I0515 17:37:58.362953       1 logger.go:78] Calling WaitAuthorization
I0515 17:38:00.629909       1 controller.go:170] Event(v1.ObjectReference{Kind:"Challenge", Namespace:"default", Name:"c-secret-2745897455-0", UID:"21191cc1-7738-11e9-ba2c-42010aa00044", APIVersion:"certmanager.k8s.io/v1alpha1", ResourceVersion:"12984126", FieldPath:""}): type: 'Warning' reason: 'Failed' Accepting challenge authorization failed: acme: authorization for identifier c.inspiredocs.com is invalid
I0515 17:38:00.639819       1 logger.go:43] Calling GetOrder
I0515 17:38:00.950203       1 sync.go:416] Error issuing certificate for default/c-secret: applying acme order back-off for certificate default/c-secret because it has failed within the last 1h0m0s

Are there other options like “manual”?

One solution can be to manually launch a server to fulfil the requirement of http-solve so that we can confirm the hypothesis that http-solver is indeed causing the problem. If we are not able to reach any solution from above then I will give this a go

#6

There you see the problem. It’s a closed world.

That

I0515 17:37:37.856680 1 sync.go:176] propagation check failed: wrong status code ‘404’, expected ‘200’
I0515 17:37:37.863502 1 controller.go:184] orders controller: syncing item ‘default/c-secret-2745897455’
I0515 17:37:37.889923 1 sync.go:176] propagation check failed: wrong status code ‘404’, expected ‘200’

looks like the tool tests the own webserver and has the wrong http status 404.

So this

I0515 17:37:37.708851 1 ingress.go:98] No existing HTTP01 challenge solver ingress found for Challenge “default/c-secret-2745897455-0”. One will be created.

may not work as expected.

#7

looks like the tool tests the own webserver and has the wrong http status 404.

At this point it is just waiting for the http-solver server to startup. Initially, since the server hasn’t started it returns 404 from default backend and it keeps on polling until it is 200. Only after it receives a 200OK from the solver does it proceed to accept the challenge.

However, I get your point. I will try with the manual process and check if I can succeed. Will get back to you after this.

#8

@JuergenAuer,

I have found out the issue. The issue was with user-agent check setup. It allowed only chrome and for others it reported 403.

I am not sure why and how the 403 was getting interpreted by letsencrypt as 404 but after fixing the user-agent to allow for lets’encrypt validation server’s user agent, which is Mozilla/5.0 (compatible; Let's Encrypt validation server; +https://www.letsencrypt.org), the problem was resolved.

Thanks for your help

1 Like