[SlugBug] server health

[Bill Best]

Matthew Collins wrote:

> Now this is weird. I'm not saying for certain, but it looks bloody odd
> to me.
> 
> Your load avg. is 0.00, but both your cpu's are busy. The question I'm
> asking myself is can you trust top? Can you trust ps?

can one, indeed...

> I'm wondering if you've been r00ted.

well, i don't know about that...

> My home machine is rarely busy, but I've never seen a load avg below
> 0.01.
> 
> I'm not saying it's a definate, but something about these listings just
> doesn't ring true.

the kernel does seems to be very busy with firewalling.

> If you have been, there are two types of rootkit that could have been
> used.
> 
> a file replacement rootkit, or a kernel patching rootkit.

i'll have a look with chkrootkit taking all appropriate precautions.

i run tripwire and portsentry and the server was very securely set-up 
with Trustix but one can never be complacent.

> Now, although some of these kits are very good, often they are used
> sloppily.
> 
> First thing I'd do is do a netstat to find all open & listen ports, then
> do an nmap of the machine from another machine.

will do.

> Check /proc/loadavg, and compare it with uptime & top. Look in /dev and
> /tmp for hidden files. These are often named ".. "
> Use ls, although those are often hacked to remove file listings. You
> could try bash file completion to get a list of files in the directory.
> ls /dev/.<tab><tab> should provide you with a list of all the hidden
> files in the directory. Compare it with the listing from ls -a.

yep, thanks for this - useful stuff.

> Download a fresh copy of "lsof" and run it as root. This will show you
> all open files (this includes sockets) and what programs are using them.

ok.

> If none of these shows anything odd, then you'll need to reboot the
> machine from a rescue disk and have a look around.

Telehouse here I come...

> The fact your machine is spending so much time in the kernel meens it's
> doing something odd. This could be file processing, or it could be busy
> packet flooding somebody.

the network traffic for this machine shows no strange patterns so i 
think i _might_ be alright on that score.

> Hopefully it will just turn out to be something weird going on with your
> streaming server, but best to make sure.

i really feel a reboot coming on  :o)

many thanks to all and best regards

bb