Attempting HTTP challenge behind AWS cloudfront and AWS network load balancer

My domain is:

I ran this command:
Using traefik's integration with let's encrypt. I ran no command directly.

My web server is (include version):
traefik 2.8.3

The operating system my web server runs on is (include version):
amazon linux 2

I can login to a root shell on my machine (yes or no, or I don't know): yes

I'm using a control panel to manage my site (no, or provide the name and version of the control panel):

The version of my client is (e.g. output of certbot --version or certbot-auto --version if you're using Certbot): NA, using traefik

AWS Cloudfront -> AWS Network Load Balancer -> traefik (traefik:v2.8.3 docker containers)

I'm getting this error:

Cannot retrieve the ACME challenge for (token \"_VUVT-UK84_sTe5Jh6TjMeFZ_8ajaMN7OTTgg8_r96I\"): cannot find challenge for token \"_VUVT-UK84_sTe5Jh6TjMeFZ_8ajaMN7OTTgg8_r96I\" (

I think it's because LE is making a request to (cloudfront) which makes a request to the origin (the domain), and traefik sees the request as if it's on that domain and not -- so it's throwing the above error.

At least I think that's what's going on...?

So, question - is it even possible to do a HTTP challange given the above setup? Is the domain translation jacking everything up?

I've looked high and low for some configuration that might "forward" the originating domain through the load balancer, but i've yet to see something like that. If there is, I'd love to try it out.

Any ideas? Thank you!

PS. Double posted this "issue" to the traefik forums here. A bit more traefik specific information there, but I think this is more an AWS/let's encrypt issue, than a traefik issue:


Hello @jaaasshh, welcome to the Let's Encrypt community. :slightly_smiling_face:

And SSL Labs on one of your IP Addresses:

I find no IP Address for your domain name.

ping: No address associated with hostname

But I do find IP Address for

PING ( 56(84) bytes of data.
64 bytes from ( icmp_seq=1 ttl=245 time=15.1 ms
64 bytes from ( icmp_seq=2 ttl=245 time=13.4 ms

Walking out the door so sorry this not fully thought out but

  1. Can you use a DNS challenge from traefik (instead of http)?
  2. Do you need to use HTTPS rather than HTTP between the AWS LB and Traefik?
  3. Could Traefix be "tricked" by sending your domain in the HOST request header? I'm not sure what options the AWS Network LB offers but I know Cloudfront can do things with the HOST header.

You are getting Response Header: HTTP/1.1 502 Bad Gateway

Using Redirect Checker | Check your Statuscode 301 vs 302
I see:

And from SSL Server Test: (Powered by Qualys SSL Labs)


As mentioned: There is no IP for

CloudFront has a valid certificate for (only)

As verified by:
openssl s_client -connect -servername
openssl s_client -connect -servername

Do you need a cert within the docker container (within their system)?


This is expected. I only have a CNAME setup pointing to cloudfront.

For a bit of context as to why, I'm explicitly testing "whitelabeling" our application wherein a customer would point a subdomain at our infrastructure in order to re-sell our product under their brand. "" is that subdomain that I am testing.

Thanks for the reply!

Not without some added difficulty. As stated in my previous post, I won't have full DNS control of the TLD here. Just the single subdomain. I think the http challenge is the simplest logistically because it only requires our customers to set a single CNAME record - then we can do the rest. At least, that's the plan :slight_smile:

I don't think so? treafik is set up to respond on both 443 and 80, but the LB is a network LB, so there's no HTTP/HTTPS configuration there - it just passes through the traffic as it's configured (and I have it configured to forward 80 to 80 and 443 to 443 - nothing fancy)
Also, I think the http challenge requires 80 to be open all the time?

This... I dunno. I think Cloudfront sets X-Forwarded-Host, and you'd think traefik would respect that if present. Will see if I can get CF to also set Host - can't hurt.

1 Like

This is expected. I'm trying to issue a cert only for

I'm not 100% sure I follow you. I think yes? I'm attempting to issue the cert in traefik because that's where I want to do ssl-termination.

I had caching enabled in Cloudfront - and have now disabled that. I've also set the "Origin Request Policy" to "All Viewer" which I believe is supposed to just forward everything.

I'm now no longer seeing any reference to the LB domain in my logs -- but, I still see lots of Error getting challenge for token retrying in 2.583049201s

So, possible step in the right direction, but not quite there yet?

AWS Console:

No, that's not what those options mean. You should review the CloudFront docs.

You have a very complex (even convoluted) architecture. A CDN in front of a Network Load Balancer in front of Traefik. Getting that to work is well beyond the scope of this forum. Just getting a cert using the HTTP challenge I think will be very difficult for you. A DNS challenge for Route53 takes some care too but probably easier than the HTTP challenge for your chosen architecture.

I will explain a simple CloudFront case to clarify how the certs work.

A client, say a browser, requests your domain name It may try with HTTP or HTTPS and CF can optionally auto-redirect to HTTPS. In fact, you have CloudFront redirecting HTTP to HTTPS right now.

So, CF terminates the original HTTPS request and needs its own cert for this purpose. You can see the cert AWS has for this using a tool like this.

CF then connects to your Origin Server(s). It can use HTTP or HTTPS or even match the method the browser used. If using HTTPS, this is a second HTTPS connection which has its own cert in your Origin Server. Your Origin Server does not need to have the same domain name as the inbound request. And, in fact, often is not for example when using S3 as an origin.

Right now your CF redirects HTTP to HTTPS. But, the connection to the origin fails and gives a 502 http error. See AWS 502 debug info

(many headers omitted for clarity)
curl -I
HTTP/1.1 301 Moved Permanently
Server: CloudFront

(following redirect manually for test)
curl -I
HTTP/2 502
server: CloudFront
x-cache: Error from cloudfront

I understand everything you said, however, I think you missed that CF doesn't redirect* to https. I have an explicit behavior configured for this path in CF.

It forwards that request to the LB and then to traefik as http all the way through. This should allow the http challenge to work. No?

Yes, fair, I did not try the acme challenge URL. But, I can't reach anything using a test acme challenge. Nor can Let's Debug test site.

curl -I -m 10
curl: (28) Operation timed out after 10001 milliseconds with 0 bytes received

So, it looks like your CF Behavior works right but something behind that is not.


So I noticed that. And I think that's expected...?? That seems to happen whenever I do what you did and curl some arbitrary acme-challenge Mike123

time="2022-08-22T12:42:40Z" level=error msg="Cannot retrieve the ACME challenge for token Mike123: cannot find challenge for token Mike123" providerName=acme - - [22/Aug/2022:12:41:47 +0000] "HEAD /.well-known/acme-challenge/Mike123 HTTP/1.1" 404 0 "-" "-" 2028 "acme-http@internal" "-" 53382ms

... I mean that's just certbot/letsencrypt dorking around 53 seconds trying to find a "Mike123" challenge token - which definitely doesn't exist.

No. The HTTP challenge relies on a running webserver to reply to the URL. It's no different than a browser asking for an html page or whatever.

So, given Mike123 does not exist, a very fast http 404 is expected.


Then what's all this? Is this cerbot (still assuming traefik uses certbot under the hoodd) output? Or is it more likely some traefik logs?

time="2022-08-22T12:56:30Z" level=error msg="Error getting challenge for token retrying in 10.04149151s" providerName=acme
time="2022-08-22T12:56:34Z" level=error msg="Error getting challenge for token retrying in 11.826312625s" providerName=acme
time="2022-08-22T12:56:39Z" level=error msg="Error getting challenge for token retrying in 302.57813ms" providerName=acme
time="2022-08-22T12:56:39Z" level=error msg="Error getting challenge for token retrying in 779.217737ms" providerName=acme
time="2022-08-22T12:56:40Z" level=error msg="Error getting challenge for token retrying in 18.232927934s" providerName=acme
time="2022-08-22T12:56:40Z" level=error msg="Error getting challenge for token retrying in 1.202932949s" providerName=acme
time="2022-08-22T12:56:41Z" level=error msg="Error getting challenge for token retrying in 2.500183347s" providerName=acme
time="2022-08-22T12:56:44Z" level=error msg="Error getting challenge for token retrying in 2.952985805s" providerName=acme
time="2022-08-22T12:56:46Z" level=error msg="Error getting challenge for token retrying in 19.876821786s" providerName=acme
time="2022-08-22T12:56:47Z" level=error msg="Error getting challenge for token retrying in 2.010148969s" providerName=acme

That looks to me like certbot is attempting to fetch something from lets encrypt, failing and retrying.

These logs happen after making a request to /.well-known/acme-challenge/foo

I'm starting to think that traefik isn't correctly sharing (amongst itself, there are 3 instances) whatever token it generates when it initiates a certificate request.

I thought Traefik had its own ACME client. Not sure certbot is involved at all. That log looks like a Traefik config problem. Have you tried asking on their forum?

An actual HTTP challenge starts with an ACME client creating a file (*) accessible to your webserver. The client then asks Let's Encrypt server to verify the domain and the LE Server sends a request for that file. The webserver should respond with the correct contents and the cert is issued. (Certbot is just one of many ACME clients)

When we do test curl's it is just the same format for the challenge URL. We expect them to fail with 404. There is nothing magical about the path in the URL.

(*) I say "file" and this is often the case. But, it does not have to be a file and can be a static value the webserver knows about.


Ah, yes. Likely a problem (maybe not only one). I don't know how Traefik manages that. Their forum is better source.

There are many threads in this forum that discuss multi-instance setups. They won't directly pertain to Traefik but may help conceptualize. One such thread:


That definitely sounds like an issue.
You should ask in the traefik help channels OR switch to DNS authentication.


Quick thanks to everyone who helped out here!

I think originally I had several issues mostly pertaining to routing port 80 through my infrastructure to traefik.

The "big" confusing issue was the result of how traefik stores certificates when you configure more than one resolver (DNS and HTTP). The fix was to direct traefik to store those certificates in different files since it seems to conflict otherwise.

Traefik config below - note that storage is different in the two resolvers.

  email = ""
  storage = "/acme/acme.json"
    provider = "route53"
    delayBeforeCheck = 0
    resolvers = ["", ""]

  email = ""
  storage = "/acme/acme-http.json"
    entryPoint = "http"