[SlugBug] server health

[Matthew Collins]

On Thu, Mar 18, 2004 at 02:33:17PM +0000, Bill Best wrote:
> cheers, chris, for such a complete answer.
> 
> not much - here you go sorted by CPU (shift+p)
> 
>   3:07pm  up 266 days,  6:18,  1 user,  load average: 0.00, 0.00, 0.00
> 56 processes: 55 sleeping, 1 running, 0 zombie, 0 stopped
> CPU0 states: 18.0% user, 81.0% system,  0.0% nice,  0.0% idle
> CPU1 states: 23.0% user, 76.0% system,  0.0% nice,  0.0% idle
> Mem:  1036048K av, 550412K used, 485636K free, 21516K shrd, 185188K buff
> Swap: 2048248K av,    184K used, 2048064K free            296616K cached

Now this is weird. I'm not saying for certain, but it looks bloody odd
to me.

Your load avg. is 0.00, but both your cpu's are busy. The question I'm
asking myself is can you trust top? Can you trust ps?

I'm wondering if you've been r00ted.

My home machine is rarely busy, but I've never seen a load avg below
0.01.

I'm not saying it's a definate, but something about these listings just
doesn't ring true.

If you have been, there are two types of rootkit that could have been
used.

a file replacement rootkit, or a kernel patching rootkit.

Now, although some of these kits are very good, often they are used
sloppily.

First thing I'd do is do a netstat to find all open & listen ports, then
do an nmap of the machine from another machine.

Check /proc/loadavg, and compare it with uptime & top. Look in /dev and
/tmp for hidden files. These are often named ".. "
Use ls, although those are often hacked to remove file listings. You
could try bash file completion to get a list of files in the directory.
ls /dev/.<tab><tab> should provide you with a list of all the hidden
files in the directory. Compare it with the listing from ls -a.

Download a fresh copy of "lsof" and run it as root. This will show you
all open files (this includes sockets) and what programs are using them.

If none of these shows anything odd, then you'll need to reboot the
machine from a rescue disk and have a look around.

The fact your machine is spending so much time in the kernel meens it's
doing something odd. This could be file processing, or it could be busy
packet flooding somebody.

Hopefully it will just turn out to be something weird going on with your
streaming server, but best to make sure.

-- 
Hulver's site. Community blogging site running scoop.
http://www.hulver.com/scoop/