Googlebot continuously hit /.well-known/acme-challenge failed

Hi,
My nginx error.log got many 404 hits (2 times per minute) in /.well-known/acme-challenge/{same token pattern} for these days.

2018/01/26 09:06:00 [error] 916#916: *46 open() “/var/www/mydomain.com/.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws” failed (2: No such file or directory), client: 66.249.92.9, server: mydomain.com, request: “GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1”, host: "www.mydomain.com"
2018/01/26 09:06:06 [error] 916#916: *47 open() “/var/www/mydomain.com/.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs” failed (2: No such file or directory), client: 66.249.92.209, server: mydomain.com, request: “GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1”, host: "mydomain.com"
2018/01/26 09:07:01 [error] 916#916: *48 open() “/var/www/mydomain.com/.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws” failed (2: No such file or directory), client: 66.249.92.94, server: mydomain.com, request: “GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1”, host: "www.mydomain.com"
2018/01/26 09:07:07 [error] 916#916: *49 open() “/var/www/mydomain.com/.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs” failed (2: No such file or directory), client: 66.249.92.209, server: mydomain.com, request: “GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1”, host: "mydomain.com"
2018/01/26 09:08:02 [error] 916#916: *50 open() “/var/www/mydomain.com/.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws” failed (2: No such file or directory), client: 66.249.92.218, server: mydomain.com, request: “GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1”, host: "www.mydomain.com"
2018/01/26 09:08:08 [error] 916#916: *51 open() “/var/www/mydomain.com/.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs” failed (2: No such file or directory), client: 66.249.92.213, server: mydomain.com, request: “GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1”, host: "mydomain.com"
2018/01/26 09:17:15 [error] 670#670: *1 open() “/var/www/mydomain.com/.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs” failed (2: No such file or directory), client: 66.249.92.211, server: mydomain.com, request: “GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1”, host: "mydomain.com"
2018/01/26 09:18:16 [error] 670#670: *4 open() “/var/www/mydomain.com/.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws” failed (2: No such file or directory), client: 66.249.92.218, server: mydomain.com, request: “GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1”, host: “www.mydomain.com

It filled all over my log file and my disk space…

The IPs looks like came from google, I’m so confused and have no idea what happened?

Hi @niiiu,

It’s hard to know without more information, but it could be:

  • Someone has linked to a challenge file for your domain somewhere, and so a search engine or other bot is trying to index or archive it
  • You have a broken Let’s Encrypt client running automatically somewhere that is trying unsuccessfully to get certificates for your own domain
  • Someone else is maliciously but unsuccessfully trying to get certificates for your domain

or maybe something else that I haven’t thought of.

@jsha, any thoughts about where this might be coming from?

@schoen Thanks for your reply, it just began about 3 days ago (2018/01/23), before that all things went well.

I would work backwards and check what pages Googlebot visited before they started looking at those request URLs.

I saw something like this before and the reason was that the server had web shell malware on it that allowed browsing the filesystem, including system log files. Not saying that’s what happening to you, but you can try see if Google found something it shouldn’t have.

zgrep -Ei "(letsencrypt|acme-challenge).*googlebot" /var/log/nginx/*
# or summary (depending on your log format)
zgrep -Ei "(letsencrypt|acme-challenge).*googlebot" /var/log/nginx/* | cut -d'"' -f 2 | sort -h | uniq

You can also tell Googlebot to stay away from your /.well-known prefix using robots.txt, which should work because those are legitimate Google IPs (unless somebody has found an SSRF in Google)

@_az Thanks, I went through access.log and found out what maybe happened. They came form Firebase:

/var/log/nginx/access.log:66.249.92.9 - - [26/Jan/2018:10:36:43 +0800] "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1" 404 162 "-" "Google-Firebase"
/var/log/nginx/access.log:66.249.92.213 - - [26/Jan/2018:10:36:46 +0800] "GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1" 404 162 "-" "Google-Firebase"
/var/log/nginx/access.log:66.249.92.94 - - [26/Jan/2018:10:37:44 +0800] "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1" 404 162 "-" "Google-Firebase"
/var/log/nginx/access.log:66.249.92.211 - - [26/Jan/2018:10:37:47 +0800] "GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1" 404 162 "-" "Google-Firebase"

My site was hosted in Firebase few month ago, but I've deleted that firebase project and hosted on my own vps now. Looks like Firebase try to renew the cert with my deleted project?

Don't know how to stop that (wait for more days till they got tired?) but at least I know what happened.
Thanks again. :slight_smile:

Well, that’s that. Maybe you can email Firebase support to remove you, since they are probably the only ones who can control this functionality.

Apart from that, you can inhibit the logging of 404s with e.g.

location /.well-known/acme-challenge/ {
    log_not_found off;
}

Just remember it’s there if you need to check it in future :smiley: .

This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.