Weird ‘hex’ (?) IP addresses in Apache access logs

[griffin]

This was my thought:

Malicious URL’s Using DWORD Formatted IP Addresses | Zscaler

[Osiris]

@griffin That's not it. The browser will first decode the DWORD into a regular IP address. Nothing strange on the server end there.

[griffin]

Thought the implementer of the "attack" may have just screwed up the attack...

[rg305]

The log entries are not referencing the destination IP (which can be made into a number or that number plus any multiple of 4B).
It is supposed to show the source IP address - which can't be obfuscated this way (across the entire Internet).

[griffin]

Fair enough. What level of control do the "attackers" have over the IP addresses that get recorded?

[toolz]

If the http server is Apache, the client IP for CGI is in the environment variable, REMOTE_ADDR. For static html, this IP is logged. If behind a pound reverse proxy, the client IP for CGI is in the environment variable, HTTP_X_FORWARDED_FOR. For static html, a custom log must be specified. If you have coded a CGI 404 error handler, it will have access to the environment variables. No idea why you would ever need to decode
"weird hex code". Blocking the IP could be as simple as specifying a null route for that IP, or you could add the IP to the firewall. We use the null route method, and remove the route
after 3 days.

[rg305]

The OP states this is a default Apache config with no proxy, nothing special.
The default log should show %h there.
That variable can't be overridden by the request itself.
If you fake the actual source IP to such a weird hex address, how would the connection exist?
Are the routers along the way ignoring the bogus source IP?
Would blocking such made up values even be effective? [they could change that value indefinitely]

[Buffer-Overflow]

[griffin]

@rg305

That's exactly why I wondered if this was some kind of "legitimate"/recognizable format or if it was happening at the "termination" point. A disguise that's dysfunctional would seem to me to only work once the work's done. So to speak.

[gibhenry]

@jvanasco

• It looks like you're running off an OSX Server. How did you install Apache? Source, a specific package installer/binary, etc? if so, which one.
• Have you analyzed the logs to determine any visitor paths that are common, and try to match that with known visitors? Knowing if this is a bot or human can help.
• What about enabling a custom log format and extending the logging activity to show the browser ident and/or other info.

I really appreciate all the suggestions, thank you all!

• I didn't really install it, Mac OS X Catalina installed it for me, automatically.
• These might well be bots because many (not all) are going for /robots.txt; but then one went for /wp-login.php, and I don't use WordPress…
• Extending the log format is beyond my ken.

So…where do we go from here?! Cheers//Gib

[gibhenry]

@toolz

These are code injection attempts. Look up the IPs, and you will probably find that they are from China or Russia. These are very common to public facing http hosts. We get hundreds of these every day. In your 404 handler, block the IP address.

I have employed the sinokorea.txt blocklist with Little Snitch, and a review of LS shows no successful connection attempts from China, Korea, or Russia. Of course, there are bad guys everywhere…

[jvanasco]

Sounds like your server is being probed.

You can run mod_security for apache, which will try and defend you from, among other things, bots using known patterns.

Assuming everything is stock, you can talk to Apple about the weird IPs. It could be a bug, a vulnerability, a weird artifact of traffic from your gateway... so many things.

[gibhenry]

Just the fact that so many knowledgeable people are stumped makes this all the more interesting, and makes me all the more determined to find an answer! Thanks, y'all!

[jsha]

@gibhenry

My installation of Apache is pretty much stock; I don't have a custom log format!

It's entirely likely that your Apache shipped with a default config that included a LogFormat directive. See the list of examples at Apache Logging Basics - The Ultimate Guide To Logging. Can you share your full Apache config, or at least any CustomLog and LogFormat directives from it?

To me it seems as if the most likely answer is that Catalina's default LogFormat assumes you'll be behind a trusted reverse proxy, and does something like interpolating the X-Real-Ip or X-Forwarded-For headers into the logs when they're present. But of course if you're not behind a reverse proxy, anyone can send you any value they like for those headers. And Apache will escape any non-ASCII characters with the hex escaping we see here.

[rg305]

@rg305

IP (which can be made into a number or that number plus any multiple of 4B).

Stupid IP tricks:
ping 555819297
ping 4850786593

[Osiris]

@rg305

ping 4850786593

osiris@erazer ~ $ ping 4850786593
ping: 4850786593: Name or service not known
osiris@erazer ~ $ 

The first one does work indeed.

[rg305]

Maybe it's just a Windows thing...

ping 555819297
Pinging 33.33.33.33 with 32 bytes of data:

ping 4850786593
Pinging 33.33.33.33 with 32 bytes of data:

[gibhenry]

@jsha

It's entirely likely that your Apache shipped with a default config that included a LogFormat directive. See the list of examples at Apache Logging Basics - The Ultimate Guide To Logging. Can you share your full Apache config, or at least any CustomLog and LogFormat directives from it?

To me it seems as if the most likely answer is that Catalina's default LogFormat assumes you'll be behind a trusted reverse proxy, and does something like interpolating the X-Real-Ip or X-Forwarded-For headers into the logs when they're present. But of course if you're not behind a reverse proxy, anyone can send you any value they like for those headers. And Apache will escape any non-ASCII characters with the hex escaping we see here.

<IfModule log_config_module>
    #
    # The following directives define some format nicknames for use with
    # a CustomLog directive (see below).
    #
    LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\"" combined
    LogFormat "%h %l %u %t \"%r\" %>s %b" common

    <IfModule logio_module>
      # You need to enable mod_logio.c to use %I and %O
      LogFormat "%h %l %u %t \"%r\" %>s %b \"%{Referer}i\" \"%{User-Agent}i\" %I %O" combinedio
    </IfModule>

    #
    # The location and format of the access logfile (Common Logfile Format).
    # If you do not define any access logfiles within a <VirtualHost>
    # container, they will be logged here.  Contrariwise, if you *do*
    # define per-<VirtualHost> access logfiles, transactions will be
    # logged therein and *not* in this file.
    #
    CustomLog "/private/var/log/apache2/access_log" common

    #
    # If you prefer a logfile with access, agent, and referer information
    # (Combined Logfile Format) you can use the following directive.
    #
    #CustomLog "/private/var/log/apache2/access_log" combined
</IfModule>

[gibhenry]

Interesting…I hadn't seen anyone fishing for "nette.micro" before. Two of them from hexed IPs…and shortly thereafter the same target from legitimate IP addresses. Sounds as though there might be some connection, but I can't figure out what it is:

\xa0\x14\x84>\xcd\x7f - - [15/Oct/2020:08:55:39 -0500] "GET /////nette.micro?callback=shell_exec&cmd=ifconfig HTTP/1.1" 301 277
\xa08\x01?\xcd\x7f - - [15/Oct/2020:09:01:14 -0500] "GET / HTTP/1.1" 301 229
\xa0\xb6\x01-\xcd\x7f - - [15/Oct/2020:09:03:58 -0500] "GET /////nette.micro?callback=shell_exec&cmd=ifconfig HTTP/1.1" 301 277
\xa0:\x03=\xcd\x7f - - [15/Oct/2020:09:03:58 -0500] "GET /robots.txt HTTP/1.1" 301 239
62.171.155.85 - - [15/Oct/2020:09:06:05 -0500] "GET /////nette.micro?callback=shell_exec&cmd=ifconfig HTTP/1.1" 301 277
213.136.92.190 - - [15/Oct/2020:09:08:09 -0500] "GET /////nette.micro?callback=shell_exec&cmd=ifconfig HTTP/1.1" 301 277
213.136.92.190 - - [15/Oct/2020:09:08:18 -0500] "GET /nette.micro?callback=shell_exec&cmd=ifconfig HTTP/1.1" 404 196
148.75.240.166 - - [15/Oct/2020:10:00:14 -0500] "GET /.well-known/openpgpkey/hu/384c5u4eekq6tcfp4qps8twxtb6fjupy?l=gib HTTP/1.1" 404 196

x

[rg305]

Can you show the vhost configs to see which log types are in use?
Try:
grep -Ri log /etc/apache2/ | grep -Ei 'error|access'

[jsha]

@rg305, it seems most likely the log format in use is "common", because of this line @gibhenry shared:

    CustomLog "/private/var/log/apache2/access_log" common

It's of course possible there's another line somewhere else overriding that, but given this is a "stock" Apache install, it's likely the truth is right in front of us.

@gibhenry, it looks like the first entry on the line is "%h", which is a fairly standard entry. But I hadn't actually read the log format help page in detail till now:

 %h	Remote hostname. Will log the IP address if HostnameLookups is set to Off, which is the default. If it logs the hostname for only a few hosts, you probably have access control directives mentioning them by name. See the Require host documentation.
%{c}h	Like %h, but always reports on the hostname of the underlying TCP connection and not any modifications to the remote hostname by modules like mod_remoteip.

That second one implies that the first one can be influence by mod_remoteip. Do you have that enabled?

You can test something: run curl -H "X-Client-IP: hi-friend" http://<Your Host>. Does hi-friend show up in your logs? Try the same for X-Forwarded-For and any other options you can think of.

Also, try searching your configs for RemoteIPHeader.