Weird ‘hex’ (?) IP addresses in Apache access logs

@jsha very interesting find!
[even a stock install brings more than expected]

How about also modifying the %h to %{c}h (or %ch)?
[not sure about the correct syntax there - this is new to me]

Perhaps this is also in play:
HostnameLookups On

Also: core - Apache HTTP Server Version 2.4

if you have hostname-based Require directives, a hostname lookup will be performed regardless of the setting of HostnameLookups .

3 Likes
/etc/apache2//httpd.conf:# with "/", the value of ServerRoot is prepended -- so "logs/access_log"
/etc/apache2//httpd.conf:# server as "/usr/local/apache2/logs/access_log", whereas "/logs/access_log"
/etc/apache2//httpd.conf:# will be interpreted as '/logs/access_log'.
/etc/apache2//httpd.conf:# configuration, error, and log files are kept.
/etc/apache2//httpd.conf:# ErrorLog: The location of the error log file.
/etc/apache2//httpd.conf:# If you do not specify an ErrorLog directive within a <VirtualHost>
/etc/apache2//httpd.conf:# logged here.  If you *do* define an error logfile for a <VirtualHost>
/etc/apache2//httpd.conf:# container, that host's errors will be logged there and not here.
/etc/apache2//httpd.conf:ErrorLog "/private/var/log/apache2/error_log"
/etc/apache2//httpd.conf:# LogLevel: Control the number of messages logged to the error_log.
/etc/apache2//httpd.conf:    # The location and format of the access logfile (Common Logfile Format).
/etc/apache2//httpd.conf:    # If you do not define any access logfiles within a <VirtualHost>
/etc/apache2//httpd.conf:    # define per-<VirtualHost> access logfiles, transactions will be
/etc/apache2//httpd.conf:    CustomLog "/private/var/log/apache2/access_log" common
/etc/apache2//httpd.conf:    # If you prefer a logfile with access, agent, and referer information
/etc/apache2//httpd.conf:    #CustomLog "/private/var/log/apache2/access_log" combined

No entries in the error log since Mon 10/12:

[Mon Oct 12 16:01:03.791991 2020] [ssl:warn] [pid 4908] AH01873: Init: Session Cache is not configured [hint: SSLSessionCache]
[Mon Oct 12 16:01:03.926722 2020] [mpm_prefork:notice] [pid 4908] AH00163: Apache/2.4.41 (Unix) LibreSSL/2.8.3 PHP/7.3.11 configured -- resuming normal operations
[Mon Oct 12 16:01:03.926763 2020] [core:notice] [pid 4908] AH00094: Command line: '/usr/sbin/httpd -D FOREGROUND'
2 Likes

That was empty...

How about:
grep -Ri hostname /etc/apache2/

Please show:
uptime

3 Likes
  1. Neither HostnameLookups nor RemoteIPHeader appears in my httpd.conf file.

  2. mod_remoteip is commented out: #LoadModule remoteip_module libexec/apache2/mod_remoteip.so

  3. curl -H "X-Client-IP: hi-friend" http://<Your Host> does not cause ANY entry to appear in the error log (except the normal access), nor does "friend" appear in error or access log.

  4. Not sure how to accomplish X-Forwarded-For.

3 Likes

Same as the other curl command, curl -H "X-Forwarded-For: hi-friend" http://<Your Host>. But Based on the other information you've shared, I doubt it will yield results. I'm stumped!

4 Likes
iGib 20-10-15 12:39 ~ 🍸 grep -Ri hostname /etc/apache2/
/etc/apache2//original/extra/httpd-vhosts.conf:# If you want to maintain multiple domains/hostnames on your
/etc/apache2//original/extra/httpd-default.conf:# When set "Off", Apache will use the Hostname and Port supplied
/etc/apache2//original/extra/httpd-default.conf:# HostnameLookups: Log the names of clients or just their IP addresses
/etc/apache2//original/extra/httpd-default.conf:HostnameLookups Off
/etc/apache2//extra/httpd-vhosts-backup.conf:# If you want to maintain multiple domains/hostnames on your
/etc/apache2//extra/httpd-default.conf~previous:# When set "Off", Apache will use the Hostname and Port supplied
/etc/apache2//extra/httpd-default.conf~previous:# HostnameLookups: Log the names of clients or just their IP addresses
/etc/apache2//extra/httpd-default.conf~previous:HostnameLookups Off
/etc/apache2//extra/httpd-vhosts.conf:# If you want to maintain multiple domains/hostnames on your
/etc/apache2//extra/httpd-default.conf~orig:# When set "Off", Apache will use the Hostname and Port supplied
/etc/apache2//extra/httpd-default.conf~orig:# HostnameLookups: Log the names of clients or just their IP addresses
/etc/apache2//extra/httpd-default.conf~orig:HostnameLookups Off
/etc/apache2//extra/httpd-vhosts.conf~previous:# If you want to maintain multiple domains/hostnames on your
/etc/apache2//extra/httpd-default.conf:# When set "Off", Apache will use the Hostname and Port supplied
/etc/apache2//extra/httpd-default.conf:# HostnameLookups: Log the names of clients or just their IP addresses
/etc/apache2//extra/httpd-default.conf:HostnameLookups Off
/etc/apache2//extra/httpd-vhosts.conf~orig:# If you want to maintain multiple domains/hostnames on your

iGib 20-10-15 12:48 ~ 🍸 uptime
12:49  up 18 days, 20:10, 3 users, load averages: 1.61 1.46 1.46
3 Likes

All are set to OFF, so that is NOT IT.
HostnameLookups Off

I'm also stumped :frowning:

3 Likes

You're right, same lack of results. This is a real mystery if you guys are stumped!!

2 Likes

Perhaps an idea to try and learn Wireshark anyway? Isn't that hard to learn I think. I'm very curious what's being transfered over the wire!

If you want to see inside TLS packets, you'd want to capture the pre-master secrets for the TLS connection too. See https://security.stackexchange.com/questions/80158/extract-pre-master-keys-from-an-openssl-application on how to do that. That is however quite advanced computer stuff.. :thinking:

4 Likes

Actually, I do have Wireshark…which is how I know it's beyond my ken! I tried, but there's too much underlying knowledge that I just don't have. :exploding_head:

3 Likes

Are we talking about the same program? :thinking:

For me, Wireshark is some sort of "point & click" program: I start it, double click on the network interface currently in use (my WiFi) in the main/center window and it'll start capturing packets. If I think I have the required packets, I press the big red "STOP" button at the top left list of buttons. That part is rather simple, right? :smiley:

The next part might seem difficult, but it actually is just some dumb clicking, but just on the right packets.
SYN/ACK TCP handshake packets have different colours, so are easily recognised and signify a single and separate TCP connection. For outgoing connections they often are accompanied by DNS packets, also easily recognisable due to different colours I believe. The contents (requested hostnames) of the DNS packets often reveal the destination of the TCP packets directly behind it. Then I press the right mouse button on one of the SYN/ACK handshake packets and go to "Follow" -> "TCP stream" in the menu(s). If it's not the TCP stream I want, I press the "Filter out" button. If it is the TCP stream I'm interested in, I'd just press "Close" so you only have the packets from that stream selected and visible.
So if you recognise the colours of those specific packets and understand what they do, you'd see the above is just some try and repeat kind of method. Not very sophisticated :wink:

The hard part in this case I think is how to recognise the correct TCP stream. As these are inbound connections, there's no correlation with DNS requests possible obviously. And the IP address is mangled in the Apache logs.
It seems the pre-master keys logged by sslkeylog.so I linked above is constantly being read by Wireshark, so it should be possible to just let Apache log pre-master keys until a mangled IP address shows up in the log and run Wireshark in parallel. If such a mangled IP address shows up, you can stop Wireshark and look for the same request from your log in your Wireshark. For example, those requests to /////nette.micro?callback=shell_exec&cmd=ifconfig. But you also had some normal IP addresses with the same request, so you might need to correlate with the time stamps for example.

5 Likes

Um…apparently not! Thanks for the encouragement. Okay, I have it running, just lying in wait for one of the SOBs to strike…

4 Likes

I captured data around another weird "IP" but I'm not sure that Wireshark has provided an answer (I'm in time zone -0500):

\xa0:\x03=\xcd\x7f - - [17/Oct/2020:08:24:38 -0500] "GET /robots.txt HTTP/1.1" 301 239

13:24:37.221650	192.168.1.2	50.17.230.78	TLSv1.2	104	Application Data
13:24:37.256312	50.17.230.78	192.168.1.2	TLSv1.2	98	Application Data
13:24:37.256378	192.168.1.2	50.17.230.78	TCP	66	51948 → 443 [ACK] Seq=3421 Ack=2881 Win=2047 Len=0 TSval=2157647855 TSecr=165627466
13:24:37.495017	192.168.1.2	62.241.4.43	TCP	54	[TCP Keep-Alive] 56495 → 993 [ACK] Seq=1269 Ack=4362 Win=262144 Len=0
13:24:37.641479	62.241.4.43	192.168.1.2	TCP	60	[TCP Keep-Alive ACK] 993 → 56495 [ACK] Seq=4362 Ack=1270 Win=16768 Len=0
13:24:37.782850	a2:40:a0:70:71:1e	Spanning-tree-(for-bridges)_00	STP	60	Conf. Root = 32768/0/a0:40:a0:70:71:1e  Cost = 0  Port = 0x8003
13:24:38.228215	192.168.1.2	80.80.81.81	DNS	87	Standard query 0xf5e5 PTR 96.117.211.198.in-addr.arpa
13:24:38.266435	80.80.81.81	192.168.1.2	DNS	118	Standard query response 0xf5e5 PTR 96.117.211.198.in-addr.arpa PTR stage.focusmx.net
13:24:38.416178	192.168.1.96	192.168.1.2	VNC	100	
13:24:38.416253	192.168.1.2	192.168.1.96	TCP	66	63894 → 5900 [ACK] Seq=673 Ack=165162961 Win=31692 Len=0 TSval=2157649010 TSecr=713459546
13:24:39.781078	a2:40:a0:70:71:1e	Spanning-tree-(for-bridges)_00	STP	60	Conf. Root = 32768/0/a0:40:a0:70:71:1e  Cost = 0  Port = 0x8003
13:24:40.059759	TexasIns_7d:02:49	Broadcast	ARP	60	Who has 192.168.1.1? Tell 192.168.1.102

Anyone see anything curious or unexpected there?

2 Likes

Just the list of packages wouldn't be enough, the contents are probably the most interesting, those of the HTTP or HTTPS (depending on which VirtualHost the request ended up in). Although I do see a reverse DNS lookup for 198.211.117.96, resolving to stage.focusmx.net. Does that IP and/or hostname look familiair?

To be sure: does Wireshark log the timestamps in UTC/GMT? Because the hours don't match, unless 08:00 at UTC-5 = 13:00 UTC.

Ah, it seems Wireshark can use any timestamp possibility there is: UTC, local time, time since start, it's all possible.

4 Likes

No! But that's Digital Ocean, and I get a ton of (bogus) hits from them.

How do I capture contents?

2 Likes

Wireshark also captures the contents. In the "View" menu, under "Packet List" you can also select "Packet Details" and "Packet Bytes". It will show you other windows with the contents.

4 Likes

Hmmm…I captured two of the last 3 "weird IPs" and they both appear to be connected with Dropbox. It seems Dropbox sends an incoming connection to the router on port 443; the router port-forwards it to my server machine. In the one second of 09:02:34CT on 10/16, there are 333 occurrences of "dropbox" in 337 entries. However, I haven't been able to search/find any occurrence of "\x" (the actual hex values are often invalid, as "\x01?\x01-\x0a*", so I can't search for them). That's too many entries to scan manually.

Does this add anything to the mix?

2 Likes

Just a thought. Do you use DropBox? If you do, is DropBox trying to connect to sync your files or checking for newer versions? And if you do have DropBox, are there 333 files it's trying to reach to check whether they need to be sync/updated?

3 Likes

Yes, I do use Dropbox, although it contains only 183 files. But why would it attempt an incoming connection via https? That sounds like an unreliable thing to do, since such a connection would go to whatever web server might be there, not necessarily the local instance of Dropbox. You're thinking outside the box, though, and that's probably what is needed to find the solution :ok_hand:

1 Like

I was trying to think of the relationship of those entries and DropBox. Neither the 337 (entries), 333 (DropBox mentions) or the 183 (files) match up. Still scratching... :thinking:

3 Likes