niiiu
January 26, 2018, 1:39am
1
Hi,
2018/01/26 09:06:00 [error] 916#916: *46 open() "/var/www/mydomain.com/.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws" failed (2: No such file or directory), client: 66.249.92.9, server: mydomain.com , request: "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1", host: "www.mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1", host: "mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1", host: "www.mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1", host: "mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1", host: "www.mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1", host: "mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/IS_wD9jF1prgOxPGTSg4mT2S86OfrOemoOyksDwsAMs HTTP/1.1", host: "mydomain.com "mydomain.com , request: "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1", host: "www.mydomain.com "
It filled all over my log file and my disk space...
The IPs looks like came from google, I'm so confused and have no idea what happened?
schoen
January 26, 2018, 1:44am
2
Hi @niiiu ,
It’s hard to know without more information, but it could be:
Someone has linked to a challenge file for your domain somewhere, and so a search engine or other bot is trying to index or archive it
You have a broken Let’s Encrypt client running automatically somewhere that is trying unsuccessfully to get certificates for your own domain
Someone else is maliciously but unsuccessfully trying to get certificates for your domain
or maybe something else that I haven’t thought of.
@jsha , any thoughts about where this might be coming from?
niiiu
January 26, 2018, 2:22am
3
@schoen Thanks for your reply, it just began about 3 days ago (2018/01/23), before that all things went well.
_az
January 26, 2018, 2:33am
4
I would work backwards and check what pages Googlebot visited before they started looking at those request URLs.
I saw something like this before and the reason was that the server had web shell malware on it that allowed browsing the filesystem, including system log files. Not saying that’s what happening to you, but you can try see if Google found something it shouldn’t have.
zgrep -Ei "(letsencrypt|acme-challenge).*googlebot" /var/log/nginx/*
# or summary (depending on your log format)
zgrep -Ei "(letsencrypt|acme-challenge).*googlebot" /var/log/nginx/* | cut -d'"' -f 2 | sort -h | uniq
You can also tell Googlebot to stay away from your /.well-known prefix using robots.txt, which should work because those are legitimate Google IPs (unless somebody has found an SSRF in Google)
niiiu
January 26, 2018, 2:55am
5
@_az Thanks, I went through access.log and found out what maybe happened. They came form Firebase:
/var/log/nginx/access.log:66.249.92.9 - - [26/Jan/2018:10:36:43 +0800] "GET /.well-known/acme-challenge/0uOPNpUYZnr5gdMzC2lvhjSFpoyW3-LKylmoSAUUpws HTTP/1.1" 404 162 "-" "Google-Firebase"
My site was hosted in Firebase few month ago, but I've deleted that firebase project and hosted on my own vps now. Looks like Firebase try to renew the cert with my deleted project?
Don't know how to stop that (wait for more days till they got tired?) but at least I know what happened.
_az
January 26, 2018, 3:00am
6
Well, that’s that. Maybe you can email Firebase support to remove you, since they are probably the only ones who can control this functionality.
Apart from that, you can inhibit the logging of 404s with e.g.
location /.well-known/acme-challenge/ {
log_not_found off;
}
Just remember it’s there if you need to check it in future
system
February 25, 2018, 3:01am
7
This topic was automatically closed 30 days after the last reply. New replies are no longer allowed.
.