Weird ‘hex’ (?) IP addresses in Apache access logs

Help
1

My Apache web server access log has numerous odd-looking entries like these, wherein the IP address is either blank or in some highly variable format which I do not understand. Internet searches produce Russian output; hex decoders can’t make sense of it. I don’t have any other resources! Can anyone here tell me what these coded IP addresses mean? (Yes, I do get mostly normal-looking IPv4 and the occasional IPv6 IPs too…I included one "normal-looking" entry at the end):

\xa0\x9a\x01\xfa\x80\x7f - - [06/Sep/2020:21:59:58 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xaa\x80\xfb\x80\x7f - - [06/Sep/2020:21:59:58 -0500] "GET / HTTP/1.1" 301 229
\xa0\xaa\x80\xfb\x80\x7f - - [07/Sep/2020:08:00:29 -0500] "GET /admin/ HTTP/1.1" 301 235
\xa0\xaa\x80\xfb\x80\x7f - - [07/Sep/2020:08:00:29 -0500] "GET /robots.txt HTTP/1.0" 301 239
\xa0\xaa\x80\xfb\x80\x7f - - [08/Sep/2020:07:59:00 -0500] "GET /mount-failed.png HTTP/1.1" 301 245
\xa0\xaa\x80\xfb\x80\x7f - - [08/Sep/2020:07:59:00 -0500] "GET /small-low-res-screen.png HTTP/1.1" 301 253
\xa0\xe2\x06\xfb\x80\x7f - - [08/Sep/2020:08:02:12 -0500] "GET /mount-failed.png HTTP/1.1" 301 245
\xa0\xaa\x80\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET /wp-login.php HTTP/1.1" 301 241
\xa0\xaa\x80\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET / HTTP/1.1" 301 229
\xa0\xaa\x80\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET /humans.txt HTTP/1.1" 301 239
\xa0\xe2\x06\xfb\x80\x7f - - [09/Sep/2020:08:29:26 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xaa - - [12/Sep/2020:07:05:56 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0t\x86\xfa\x80\x7f - - [12/Sep/2020:07:06:29 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xe2\x06\xfb\x80\x7f - - [12/Sep/2020:07:06:29 -0500] "GET /sitemap_index.xml HTTP/1.1" 301 246
\xa0\xaa - - [12/Sep/2020:07:13:11 -0500] "GET / HTTP/1.1" 301 229
\xa0t\x86\xfa\x80\x7f - - [12/Sep/2020:07:13:11 -0500] "GET / HTTP/1.1" 301 229
\xa0\x9a\x01\xfa\x80\x7f - - [12/Sep/2020:07:13:11 -0500] "GET / HTTP/1.1" 301 229
\xa0t\x86\xfa\x80\x7f - - [12/Sep/2020:07:13:27 -0500] "GET /robots.txt HTTP/1.1" 301 239
\xa0\xaa - - [12/Sep/2020:07:14:01 -0500] "GET / HTTP/1.1" 301 229
\xa0\x9a\x01\xfa\x80\x7f - - [12/Sep/2020:07:15:21 -0500] "GET / HTTP/1.1" 301 229
 - - [21/Sep/2020:18:36:35 -0500] "GET / HTTP/1.0" 400 362
192.71.42.108 - - [05/Oct/2020:01:21:18 -0500] "GET /humans.txt HTTP/1.1" 404 196

Thanks in advance for any insight you can offer. Cheers//Gib Henry

3 Likes
2

Hi @gibhenry! This is probably not the right forum for this question, as we specialize in Let's Encrypt issues rather than general Apache support. But, to not leave you lacking any information at all - I would check your Apache configs for LogFormat and CustomLog directives. You might have something odd going on there. Best luck!

4 Likes
3

Thank you! Can you suggest a more appropriate forum for this info? Cheers,

3 Likes
4

I haven't checked out these two forums, but they could give you a start.

https://forums.digitalpoint.com/forums/apache.49/

https://www.apachelounge.com/

5 Likes
5

I would try using a "better" custom log format.
But I am interested in knowing what your LogFormat looks like now.
In any case, best of luck with that :slight_smile:

Cheers from Miami :beers:

4 Likes
6

I searched around out of curiosity and tried various ways of decoding it myself to no avail. All I primarily found were @gibhenry's own posts asking the same question elsewhere.

4 Likes
7

It is possible to send the output to a single log file from multiple vhosts.
So I would try to locate the vhost that is sending these logs and then show the LogFormat in use.
I believe we will find something in there that can explain this output.

4 Likes
8

Well OBVIOUSLY somebody is trying to invent a new request smuggling technique.

But wherever you do post a new thread, make sure to mention if your server is behind a reverse-proxy of any kind.

5 Likes
9

My installation of Apache is pretty much stock; I don't have a custom log format!

2 Likes
10

At least in my case, there are no vhosts. Cheers//Gib Henry

2 Likes
11

I just searched on google and found this over at serverfault.

4 Likes
12

That's a different thing, though…the hex is in the request, not the section for the IP from which it's received. I know mine is hex encoded, but decoding it doesn't make sense in the context of an IP address. Cheers//Gib Henry

2 Likes
13

Might it be related to this:

3 Likes
14

Different format, though…appears to be decimal. The mystery remains!

2 Likes
15

It starts as hex though. Wonder if they just didn't convert to decimal?

3 Likes
16

Well, try it! I can't convert some of that stuff, for example
\xa0Z\x81`\xd8\x7f

and
\xa0*\x85_\xd8\x7f

(Yes, that's a left single quote and an asterisk embedded in a hex code.) They both appear multiple times. What's that about?!

3 Likes
17

I will try a bit later. Trouble is I can't authenticate the result.

3 Likes
18

Some are so much sorter than others - it has no rhyme or reason (that I can see).
I say memory corruption.
The location that stores the %h pointer now points to lalaland.

3 Likes
19

I'm also guessing hardware errors.

3 Likes
20

If you reboot and it happens straightaway, I would be concerned.
How many days up?
How is it doing on resources?

3 Likes