arghedy argh argh

[japester.livejournal.com]

something you never ever want to see on a production server:


# sar -f sa27 -g

SunOS <hostname> 5.9 Generic_117171-08 sun4u 04/27/2005

00:00:01  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
01:00:00     0.94     1.98     1.98     0.00     0.05
02:00:00     0.76     1.06     1.05     0.00     0.96

<snippety>

16:00:00    11.38    25.32    25.31     0.00     0.00
16:20:00    10.85    26.22    26.21     0.00     0.00
16:40:00    38.04   270.21   273.24    60.48     0.00
17:00:00    57.50   590.54   592.73   151.67     0.00
17:20:00    44.11   569.10   571.56   161.28     0.00
17:40:01    79.33   795.55   812.00   667.62     0.00
18:00:01    59.29   703.71   723.77  1465.26     0.00

Average      9.51    65.60    66.46    51.43     0.71

the explanation for the non unix geeks. Page in/outs are chunks (pages) of memory being swapped to or from disk. Page scans are requests for a free page of memory. 200 is the upper limit you'd ever expect to see on a fully loaded server. This poor beast seemed to have some root owned process doing:

while (true) {
malloc();
}

and we wondered why it became completely unresponsive at about 18:30 ... We hadn't noticed before because we were too busy playing CounterStrike. Would have been much worse if we'd done the normal 17:00 thing and gone home then (and not noticed until 0830 the next morning)