[SlugBug] Web Server High Utilisation

[Alan Dawson]

HELP!!!

I just noticed my web server becoming unresponsive

I ssh in to it and after an age could log in

 w reports high load averages  

2 users,  load average: 91.10, 98.21, 104.29

top shows lots of apache processes

top - 14:44:38 up 12 days, 55 min,  2 users,  load average: 94.95, 100.67,
105.68
Tasks: 236 total,   6 running, 229 sleeping,   0 stopped,   1 zombie
Cpu(s):  5.1% us,  5.7% sy,  0.0% ni,  0.0% id, 88.9% wa,  0.0% hi,  0.4% si
Mem:   2054252k total,  2047676k used,     6576k free,     1580k buffers
Swap:  7815580k total,  4433928k used,  3381652k free,    13812k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
 1709 www-data  16   0  162m  21m 112m D  2.3  1.1   0:08.13 apache
 1453 www-data  15   0  163m  21m 112m D  1.6  1.1   0:19.51 apache
 3022 worker    17   0  9528 1376 9164 R  1.3  0.1   0:00.21 top
   52 root      15   0     0    0    0 D  1.0  0.0   1:33.47 kswapd1
 1473 www-data  15   0  160m  26m 112m D  1.0  1.3   0:07.18 apache
 1680 www-data  18   0  161m  22m 112m R  1.0  1.1   0:07.69 apache
 1725 www-data  15   0  161m  23m 112m D  1.0  1.2   0:10.76 apache
 2934 mkdoc     15   0 38256  13m  25m D  1.0  0.7   0:00.88 031..rss_routin
 1880 root      15   0 20772 1416  18m S  0.7  0.1   0:11.34 snmpd
 1354 www-data  15   0  172m  16m 112m D  0.7  0.8   0:24.69 apache
 1623 www-data  17   0  157m  23m 112m D  0.7  1.2   0:07.46 apache
 1642 www-data  15   0  161m  23m 112m D  0.7  1.2   0:09.43 apache
 1692 www-data  18   0  156m  22m 112m D  0.7  1.1   0:08.94 apache
 1697 www-data  17   0  161m  26m 112m D  0.7  1.3   0:08.04 apache
 1710 www-data  17   0  158m 9.8m 112m D  0.7  0.5   0:07.79 apache
 1740 www-data  16   0  158m  19m 112m D  0.7  1.0   0:05.70 apache
   37 root       5 -10     0    0    0 S  0.3  0.0   0:08.66 kblockd/0
   53 root      15   0     0    0    0 S  0.3  0.0   1:30.24 kswapd0
28918 www-data  15   0  177m  19m 112m D  0.3  0.9   0:59.29 apache
 1472 www-data  15   0  163m  22m 112m D  0.3  1.1   0:19.15 apache
 1504 www-data  17   0  161m  20m 112m D  0.3  1.0   0:08.56 apache
 1511 www-data  15   0  157m  22m 112m D  0.3  1.1   0:10.98 apache
 1563 www-data  17   0  161m  17m 112m D  0.3  0.9   0:08.11 apache
 1570 www-data  16   0  161m  15m 112m S  0.3  0.8   0:06.49 apache
 1586 www-data  17   0  161m  20m 112m D  0.3  1.0   0:08.11 apache
 1588 www-data  18   0  162m  22m 112m D  0.3  1.1   0:08.81 apache
 1589 www-data  17   0  161m  19m 112m D  0.3  1.0   0:07.97 apache
 1599 www-data  18   0  161m  12m 112m D  0.3  0.6   0:07.23 apache
 1600 www-data  17   0  161m  16m 112m R  0.3  0.8   0:08.55 apache
 1604 www-data  15   0  161m  21m 112m D  0.3  1.1   0:08.27 apache
 1625 www-data  18   0  151m  19m 111m D  0.3  1.0   0:05.63 apache
 1626 www-data  18   0  161m  18m 112m D  0.3  0.9   0:07.48 apache


ps ax | grep apach | wc -l 

shows 155 processes

/var/log/messages shows 

Mar  8 13:41:31 miro apache: PHP Warning:  mysql_connect(): Too many connections


My traffic stats show a huge peak about that time.


I do stop apache and restart it and thing return to normal.

Whats a good place to start tracking this down ?


Cheers

Alan Dawson
-- 
"If you make decisions about software -- or anything -- based solely on
short-term cost and benefit, someone with a longer view can easily
manoeuver you into a trap from which it is hard to escape."