Weird ‘hex’ (?) IP addresses in Apache access logs

My Apache web server access log has numerous odd-looking entries like these, wherein the IP address is either blank or in some highly variable format which I do not understand. Internet searches produce Russian output; hex decoders can’t make sense of it. I don’t have any other resources! Can anyone here tell me what these coded IP addresses mean? (Yes, I do get mostly normal-looking IPv4 and the occasional IPv6 IPs too…I included one "normal-looking" entry at the end):

\xa0\x9a\x01\xfa\x80\x7f - - [06/Sep/2020:21:59:58 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xaa\x80\xfb\x80\x7f - - [06/Sep/2020:21:59:58 -0500] "GET / HTTP/1.1" 301 229
\xa0\xaa\x80\xfb\x80\x7f - - [07/Sep/2020:08:00:29 -0500] "GET /admin/ HTTP/1.1" 301 235
\xa0\xaa\x80\xfb\x80\x7f - - [07/Sep/2020:08:00:29 -0500] "GET /robots.txt HTTP/1.0" 301 239
\xa0\xaa\x80\xfb\x80\x7f - - [08/Sep/2020:07:59:00 -0500] "GET /mount-failed.png HTTP/1.1" 301 245
\xa0\xaa\x80\xfb\x80\x7f - - [08/Sep/2020:07:59:00 -0500] "GET /small-low-res-screen.png HTTP/1.1" 301 253
\xa0\xe2\x06\xfb\x80\x7f - - [08/Sep/2020:08:02:12 -0500] "GET /mount-failed.png HTTP/1.1" 301 245
\xa0\xaa\x80\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET /wp-login.php HTTP/1.1" 301 241
\xa0\xaa\x80\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET / HTTP/1.1" 301 229
\xa0\xaa\x80\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET /humans.txt HTTP/1.1" 301 239
\xa0\xe2\x06\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xaa - - [12/Sep/2020:07:05:56 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0t\x86\xfa\x80\x7f - - [12/Sep/2020:07:06:29 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xe2\x06\xfb\x80\x7f - - [12/Sep/2020:07:06:29 -0500] "GET /sitemap_index.xml HTTP/1.1" 301 246
\xa0\xaa - - [12/Sep/2020:07:13:11 -0500] "GET / HTTP/1.1" 301 229
\xa0t\x86\xfa\x80\x7f - - [12/Sep/2020:07:13:11 -0500] "GET / HTTP/1.1" 301 229
\xa0\x9a\x01\xfa\x80\x7f - - [12/Sep/2020:07:13:11 -0500] "GET / HTTP/1.1" 301 229
\xa0t\x86\xfa\x80\x7f - - [12/Sep/2020:07:13:27 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xaa - - [12/Sep/2020:07:14:01 -0500] "GET / HTTP/1.1" 301 229
\xa0\x9a\x01\xfa\x80\x7f - - [12/Sep/2020:07:15:21 -0500] "GET / HTTP/1.1" 301 229
 - - [21/Sep/2020:18:36:35 -0500] "GET / HTTP/1.0" 400 362
192.71.42.108 - - [05/Oct/2020:01:21:18 -0500] "GET /humans.txt HTTP/1.1" 404 196

Thanks in advance for any insight you can offer. Cheers//Gib Henry

3 Likes

Hi @gibhenry! This is probably not the right forum for this question, as we specialize in Let's Encrypt issues rather than general Apache support. But, to not leave you lacking any information at all - I would check your Apache configs for LogFormat and CustomLog directives. You might have something odd going on there. Best luck!

3 Likes

Thank you! Can you suggest a more appropriate forum for this info? Cheers,

3 Likes

I haven't checked out these two forums, but they could give you a start.

https://forums.digitalpoint.com/forums/apache.49/

https://www.apachelounge.com/

4 Likes

I would try using a "better" custom log format.
But I am interested in knowing what your LogFormat looks like now.
In any case, best of luck with that :slight_smile:

Cheers from Miami :beers:

3 Likes

I searched around out of curiosity and tried various ways of decoding it myself to no avail. All I primarily found were @gibhenry's own posts asking the same question elsewhere.

3 Likes

It is possible to send the output to a single log file from multiple vhosts.
So I would try to locate the vhost that is sending these logs and then show the LogFormat in use.
I believe we will find something in there that can explain this output.

3 Likes

Well OBVIOUSLY somebody is trying to invent a new request smuggling technique.

But wherever you do post a new thread, make sure to mention if your server is behind a reverse-proxy of any kind.

4 Likes

My installation of Apache is pretty much stock; I don't have a custom log format!

2 Likes

At least in my case, there are no vhosts. Cheers//Gib Henry

2 Likes

I just searched on google and found this over at serverfault.

4 Likes

That's a different thing, though…the hex is in the request, not the section for the IP from which it's received. I know mine is hex encoded, but decoding it doesn't make sense in the context of an IP address. Cheers//Gib Henry

2 Likes

Might it be related to this:

2 Likes

Different format, though…appears to be decimal. The mystery remains!

2 Likes

It starts as hex though. Wonder if they just didn't convert to decimal?

2 Likes

Well, try it! I can't convert some of that stuff, for example
\xa0Z\x81`\xd8\x7f

and
\xa0*\x85_\xd8\x7f

(Yes, that's a left single quote and an asterisk embedded in a hex code.) They both appear multiple times. What's that about?!

3 Likes

I will try a bit later. Trouble is I can't authenticate the result.

2 Likes

Some are so much sorter than others - it has no rhyme or reason (that I can see).
I say memory corruption.
The location that stores the %h pointer now points to lalaland.

2 Likes

I'm also guessing hardware errors.

2 Likes

If you reboot and it happens straightaway, I would be concerned.
How many days up?
How is it doing on resources?

2 Likes