Unexpected and repeated requests for challenge that we don't have

[aeijdenberg]

Hi, we’ve been happily using Let’s Encrypt across our infrastructure for more than a year and have had no issues building automation for the renewal of ~100 certificates using HTTP and DNS validation across a number of environments.

Recently we started seeing intermittent monitoring errors on our primary server that handles ACME challenge responses for HTTP validation.

It turned out that we had too many files open and upon further investigation (lsof) we noticed thousands of established connections with:
ec2-13-238-131-82.ap-southeast-2.compute.amazonaws.com

Although our AWS accounts are also based in the AWS ap-southeast-2 region, we’re fairly sure that IP does not belong to any of our instances or network interfaces - although it’s possible we’re mistaken.

In addition to adding idle connection timeouts to our server to mitigate the too many open files issue, we added additional logging and noted hundreds of requests for:

/.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ

(e.g. with timestamps in UTC for the past hour:)

2019/02/26 00:01:19 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:02:19 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:03:38 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:05:21 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:07:35 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:08:20 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:10:17 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:03 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:04 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:05 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:07 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:09 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:11 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:14 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:19 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:24 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:31 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:39 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:14:50 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:15:07 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:15:26 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:15:53 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:16:26 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:17:06 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:18:06 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:19:19 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:20:47 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:22:41 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:25:19 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:28:49 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:28:50 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:28:51 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:28:52 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:28:54 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:28:57 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:00 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:05 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:10 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:17 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:25 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:37 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:29:51 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:30:12 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:30:35 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:31:10 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:31:49 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:32:50 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:34:04 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:35:39 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:37:51 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:40:21 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:44 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:45 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:46 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:48 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:50 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:52 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:43:55 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:44:00 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:44:05 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:44:12 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:44:22 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:44:34 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:44:48 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:45:07 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:45:33 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:46:05 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:46:51 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:47:50 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:49:03 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:50:35 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:52:43 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ
2019/02/26 00:55:23 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ

Our frontend HAProxy actually redirects to this server, so while we don’t capture the host header here, we added additional logging to our HAProxy in an attempt to capture the host header for which this repeated challenge is issued, however although when we manually issue a browser request the host header is capture and logged, it does not seem to appear for these requests, suggesting perhaps (?) it isn’t being set?

Appreciate any help in tracking down. So far as we are aware, all of our certs are good and accounted for - we’re not sure what is initiating this request or making it repeat so often.

My domain is: we’re unsure!

I ran this command: n/a

It produced this output: n/a

My web server is (include version): HAProxy

The operating system my web server runs on is (include version): Linux

My hosting provider, if applicable, is: CloudFoundry on AWS with our own HAProxy based TLS termination in front.

I can login to a root shell on my machine (yes or no, or I don’t know): yes

[Osiris]

Do your logs provide a client identification header?

[JuergenAuer]

Hi @aeijdenberg

@aeijdenberg

2019/02/26 00:01:19 404 /.well-known/acme-challenge/kEwZvvvDENtLnHYAT5ZUIbLn11GNB4i5rh7ug1lmvBQ

the file name is the token of the challenge. Every challenge has a new and unique token, tokens are not published.

So it may be your own client or an own test that produces that error.

Or it's a random user / wrong configured bot, that checks that.

So check the ip of that request.

[aeijdenberg]

Am pretty sure we don’t have any tests that emulate a LE domain validation request.

And it seems odd that real LE would keep retrying the same challenge (over a few weeks now).

I’ll try capturing the User-Agent and see if that can tell us anything about the request.

[_az]

Curiously, that host stopped responding to ICMP Echo since earlier this morning. So maybe the requests will disappear now.

[schoen]

@lestaff, this machine isn’t part of Let’s Encrypt infrastructure, is it?

[aeijdenberg]

Ah, so ec2-13-238-131-82.ap-southeast-2.compute.amazonaws.com is a Let’s Encrypt host (or ENI?)

Can confirm we’re still receiving a request every 5 minutes or so to our HAProxy (with the challenge above), which then issues a 302 redirect to our server that handles these, which then then seems to get about 1 request a minute which it 404s.

[_az]

I don’t think they are. Let’s Encrypt’s staging infra does use EC2, but I’ve only ever seen them use us-west, us-east and eu-central.

Who knows, maybe they added Sydney, but Let’s Encrypt doesn’t retry challenges on its own, and you would receive requests from multiple servers at once if it initiated via ACME.

(If it turns out to be not Let’s Encrypt, you could try https://aws.amazon.com/forms/report-abuse)

[aeijdenberg]

by way of update, I added HAProxy config for logging for User-Agent, and like for Host, for some reason it isn’t propagating through our infrastructure for just these requests (or isn’t being set, which would be simpler explanation).

[aeijdenberg]

Managed to figure out why the host/user agent wasn’t being logged. These requests were coming in via HTTPS (not HTTP like normal), as this particular tenant has CloudFlare in front of them (I’m guessing that IP belongs to them), which was redirecting to HTTPS before hitting our termination (and thus not triggering the right HAProxy config to add the extra log fields).

In any case, we now know who the tenant is on our platform and will follow-up with them whether they have something that has been triggering these requests.

Appreciate your help.

[ezekiel]

Glad to hear you’re on the trail of a possible cause!
@schoen and @_az are correct, at the time of writing we do not utilize instances in that region.