Today I encountered a weird behavior while requesting a cert with "certbot certonly ...." : Just a few minutes after obtaining my cert on a FQDN I just invented, configured and told noone about, my web logs show weird scan pattern that don't look friendly from IPs I can't relate to let's encrypt.
I have repeated the same operation and the scan takes place every single time.
Step to reproduce :
setup a vhost on an url you invented like iioooii.domain.ext
request a cert for iioooii.domain.ext
let's encrypt bot is fetching the acme challenge
activate your ssl enabled vhost
wait ~3 minutes
my apache logs show for this specific vhost I just created :
89.175.184.250 - - [26/Apr/2022:17:52:38 +0200] "GET / HTTP/1.1" 403 5729 "-" "Mozilla/5.0 (Windows NT 5.1) AppleWebKit/537.14 (KHTML, like Gecko) Chrome/46.0.2490.86 Safari/537.14"
89.175.184.250 - - [26/Apr/2022:17:52:42 +0200] "GET / HTTP/1.1" 403 5729 "-" "Mozilla/5.0 (Windows NT 6.1; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2255.0 Safari/537.36"
45.155.204.20 - - [26/Apr/2022:17:52:58 +0200] "GET /.env HTTP/1.1" 404 5697 "-" "Opera/9.80 (Linux i686; Opera Mobi/1040; U; en) Presto/2.5.24 Version/10.00"
51.81.167.146 - - [26/Apr/2022:17:52:58 +0200] "GET / HTTP/1.1" 403 458 "-" "Mozilla/5.0 (iPhone; CPU iPhone OS 15_1_1 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/15.1 Mobile/15E1
48 Safari/604.1"
45.155.204.20 - - [26/Apr/2022:17:53:18 +0200] "GET /.git/config HTTP/1.1" 404 5697 "-" "Mozilla/5.0 (Macintosh; Intel Mac OS X 1083) AppleWebKit/537.36 (KHTML like Gecko) Chrome/28.0.1469.0 Safari/5
37.36"
52.90.30.23 - - [26/Apr/2022:17:55:27 +0200] "GET / HTTP/1.1" 403 514 "-" "Mozilla/5.0 (Windows NT 6.2; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/32.0.1667.0 Safari/537.36"
185.220.100.248 - - [26/Apr/2022:18:05:18 +0200] "GET / HTTP/1.1" 403 5744 "-" "curl/7.58.0"
My guess is that along the line, my vhost url I just created was leaked.
Adding: CT is a widely supported internet standard. I don't believe the CA/B Forum currently requires CT logging, however many browser and operating system vendors (such as chrome and apple) require CAs to log and will reject unlogged certificates as untrusted. This essentially makes CT mandatory for all publicly trusted CAs and anyone using them.
Thanks for the explanation !
It's good to know that those logs are also automatically monitored by bad people and it's better to keep that in mind before providing sentitive information on a page someone might believe hidden...
These types of log entries are more and more common now, but have been seen in logs for years...
I suggest looking in to configuring fail2ban or some other IDS. Out of scope on this forum, but it could help reduce unwanted probes of your system(s).
Might Help
As a rule, assume that any web server you run that's open to the Internet will be found by automated scanners. Even if it weren't for certificate transparency logs, there less than 4 billion public IPv4 addresses, meaning it's practical for botnets to scan literally all of them periodically. If you give a web s server a public IPv4 address, it's gonna get found by bots pretty quickly.
Basically, never rely on "no one knows about this web server" as a security mechanism - anything sensitive should be behind some sort of authentication system. (i.e. a login page or HTTP basic authentication)