This continues across multiple restarts. httpd uses minimal resources, often zero (not a high-traffic site!).
2 Likes
Can you show the top of top ?
2 Likes
I'll second this suggestion: could it be that you have an Apache module enabled (like mod_remoteip or the old mod_cloudflare) that logs the IP address from a header like X-Forwarded-For, instead of the "real" client IP address?
Another step to troubleshoot this further could be to capture raw traffic with tcpdump, match up times with the weird log entries, and see what's really in those packets. Wireshark may be able to decrypt TLS traffic if you load your private key.
5 Likes
My apache installation is very much stock, no remoteip or cloudflare or X-Forwarded-For. And I fear that Wireshark is beyond my paygrade!
But I can upload a screenshot of top (Activity Monitor):
Processes: 685 total, 2 running, 683 sleeping, 3681 threads 17:18:57
Load Avg: 1.40, 1.26, 1.25 CPU usage: 5.35% user, 3.14% sys, 91.49% idle SharedLibs: 520M resident, 105M data, 38M linkedit.
MemRegions: 264779 total, 13G resident, 266M private, 4810M shared. PhysMem: 38G used (4499M wired), 2221M unused.
VM: 7367G vsize, 1998M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 98233785/59G in, 128375681/113G out.
Disks: 60125464/1118G read, 22761017/473G written.
PID COMMAND %CPU TIME #TH #WQ #POR MEM PURG CMPR PGRP PPID STATE BOOSTS %CPU_ME %CPU_OTHRS UID FAUL COW MSGS MSGR SYSB
4951 httpd 0.0 00:00.01 1 0 18 1148K 0B 0B 4908 4908 sleeping *0[1] 0.00000 0.00000 70 642 87 29 12 300
2 Likes
Sorry, but I don't see all that I was looking for...
How about the output of:
free
3 Likes
I enlarged my terminal window to the size of the 27" screen and got 80 lines of PIDs, but httpd wasn't among them. I don't know of a way to limit top output to httpd. As to free,
iGib 20-10-12 17:37 ~ 🍸 free
zsh: command not found: free
I will upload screenshots of Activity Monitor, showing similar info, limited to httpd (there are 10 instances of it, each showing 0 cpu and a few kb read).
2 Likes
The only tab that shows mem was reduced to only show the mem used by http.
It says nothing else - not even how much mem is installed/usable.
If you have access to SSH, try top there. The top part (of top) should look something like:
or try:
htop
2 Likes
I thought I sent that above, but here it is:
Processes: 693 total, 2 running, 691 sleeping, 3746 threads 18:14:47
Load Avg: 1.62, 1.53, 1.45 CPU usage: 11.3% user, 14.23% sys, 74.73% idle SharedLibs: 520M resident, 105M data, 38M linkedit.
MemRegions: 266780 total, 13G resident, 271M private, 5277M shared. PhysMem: 39G used (4630M wired), 1497M unused.
VM: 7407G vsize, 1991M framework vsize, 0(0) swapins, 0(0) swapouts. Networks: packets: 98374306/59G in, 128498656/113G out.
Disks: 60254439/1119G read, 22810868/474G written.
(Don't have htop)
2 Likes
Was that run on the VM or the host?
2 Likes
It's the host; there's no VM.
2 Likes
Well then it doesn't appear to be a lack of resource thing.
2 Likes
I assume that requires a non-perfect forward secrecy cipher suite?
2 Likes
Edit: I was wrong, it is possible using just the key. Neat.
5 Likes
I've never seen you go though so many gyrations, _az. Makes me feel a bit better about my own corrections. 
2 Likes
I would suggest not to edit away the method with SSLKEYLOGFILE as I didn't know that feature! 
4 Likes
These are code injection attempts. Look up the IPs, and you will probably find that they are from China or Russia. These are very common to public facing http hosts. We get hundreds of these every day. In your 404 handler, block the IP address.
1 Like
Easier said than done when the actual IP address(es) have been replaced with indiscernible hex entries. Sure, he can go one by one and match the timestamps with the firewall logs - but that is just way too much work for any human to do.
Where should he look to get the IP (first) ?
2 Likes
If only we managed to descramble the weird hex code to uncover the IP addresses 
3 Likes
I've been following this for days hoping to an answer pop up...
Questions:
-
It looks like you're running off an OSX Server. How did you install Apache? Source, a specific package installer/binary, etc? if so, which one.
-
Have you analyzed the logs to determine any visitor paths that are common, and try to match that with known visitors? Knowing if this is a bot or human can help.
-
What about enabling a custom log format and extending the logging activity to show the browser ident and/or other info.
2 Likes