vmstat case studies

Contributed by Michilimackinac April 1, 2004  
Tagged: Solaris

Look at some problem systems…

When running vmstat there are a few things to keep in mind; running vmstat itself
does impact the system. I would not recommend running vmstat with less than a
five second interval. Also it should be noted that on solaris the first vmstat
entry is bogus; some look close but I would recommend ignoring them.

Also when running these we need to take snapshots at different times to compare

as systems sometimes have busy periods.

These three formulas are a good starting point:

CPU shortage when procs->r column is greater than (number of CPU’s*4)
procs->r is the number of runnable processes that are ready to go.

I/O problem (disks, controllers etc.) when procs->b is greater than 0
procs->b is the number of processes blocked waiting for I/O

Memory shortfall when “sr” column is greater than 150
sr is scan rate which is the rate a which the system is scanning for
free pages of memory.

We can look at some data:

Note: assuming 4 CPUs in the below examples


% vmstat 5 10
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s1 s1 s1 sd in sy cs us sy id
31 6 0 4542592 1747840 378 3631 94 262 245 0 0 0 25 25 0 2365 17793 4086 62 18 20
52 2 0 4446880 1556736 325 2474 35 300 294 0 0 0 0 0 0 1681 22277 2279 83 11 6
70 1 0 4451632 1557256 567 5702 35 404 390 0 0 0 2 2 0 1879 24490 2432 79 17 4
56 2 0 4461664 1562864 553 6001 60 289 265 0 0 0 1 1 0 1611 16779 2288 69 17 14
58 2 0 4454664 1562192 450 3765 179 374 356 0 0 0 0 0 0 1848 20948 2345 84 14 2
62 2 0 4457192 1561640 360 3179 9 504 491 0 0 0 0 0 0 1789 22680 2201 83 13 5
12 2 0 4448856 1553080 305 2701 22 216 208 0 0 0 1 2 0 1499 12331 1934 84 10 7
97 2 0 4448936 1556288 615 5030 43 582 571 0 0 0 1 1 0 2128 29096 2560 84 16 0
54 1 0 4453744 1559144 770 8629 3 286 272 0 0 0 0 0 0 1988 20503 2553 78 21 1
6 2 0 4459680 1561424 272 2773 27 280 265 0 0 0 1 1 0 1373 10061 2027 63 9 28


CPU bottleneck on a 4 CPU system over 16 is a problem and this is consistently above that

Memory is okay as scan rate sr is 0 and there is not much paging activity

I/O is being blocked as there are non-zero values under the proc->b column we need to look at disks controllers with the “iostat” command to get more information


procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s3 -- in sy cs us sy id
0 0 0 143184 53920 0 2 0 0 0 0 0 0 0 2 12 13 0 0 99
0 0 0 307400 1820 8 67 537 1128 1204 0 271 172 0 545 1356 341 2 45 53
0 0 0 307396 1564 3 59 476 974 993 0 45 166 0 525 1122 286 2 35 63
0 0 0 307396 1228 0 91 716 1207 1333 0 225 203 0 600 1597 379 4 46 51
0 0 0 307408 1344 0 87 699 1207 1352 0 118 200 0 595 1590 368 2 47 51
0 0 0 307412 1340 0 87 636 1232 1376 0 194 181 0 538 1485 351 3 40 57
0 0 0 307396 1440 0 115 666 1234 1347 0 190 185 0 543 1800 370 4 51 45
0 0 0 307392 1456 0 87 700 1368 1534 0 252 195 0 573 1584 384 3 44 53
0 0 0 307392 1520 0 82 660 1301 1339 0 100 193 0 579 1494 354 2 42 56
0 0 0 307392 1320 0 83 667 1083 1194 0 149 195 0 583 1514 359 3 45 52
0 0 0 307404 1540 0 83 669 1216 1332 0 158 196 0 593 1514 354 5 46 49
0 0 0 307404 1372 1 82 661 1242 1361 0 158 194 0 586 1518 362 2 45 54
0 0 0 307404 1428 0 92 742 1320 1495 0 203 206 0 597 1678 392 4 52 44


CPU’s look okay here, no cpu’s blocked waiting for I/O or runnable processes

Memory is in short supply here we are doing some memory intensive operations and is very close to needing more memory sr has value consistently above 150

I/O appears to be okay looking at the vmstat data


procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 s3 s4 s4 in sy cs us sy id
0 3 0 8895904 6139168 2798 210 1275 4 4 0 0 0 2 0 2 863 1877 688 22 9 69
0 3 0 8225008 5616824 3105 1 26808 0 0 0 0 0 2 0 2 2865 4674 2304 20 6 74
0 4 0 8224144 5620464 3681 2471 27750 4 4 0 0 0 1 0 1 2918 10765 3044 24 10 66
0 4 0 8224944 5634848 3496 0 28283 6 6 0 0 0 0 0 0 2866 5254 2427 23 6 71
0 8 0 8224944 5643128 3254 0 26294 0 0 0 0 0 3 0 4 3295 4696 3358 20 8 72
0 6 0 8224328 5644360 3167 105 26665 0 0 0 0 0 0 0 0 2930 4161 2447 17 6 77
0 3 0 8223912 5648304 3314 0 26913 3 3 0 0 0 0 0 0 2839 3815 2344 16 6 78

0 5 0 8223912 5645112 3494 0 28464 3 3 0 0 0 2 0 2 3030 5123 2644 22 7 71
0 5 0 8223912 5643728 3362 0 28664 0 0 0 0 0 0 0 0 2869 5343 2284 22 7 71
0 7 0 8223912 5634344 3522 0 29249 3 3 0 0 0 18 0 17 3087 5386 2410 22 7 70


Cpu’s are blocked waiting for I/O so they are showing up as idle cpu->idle

Memory on this system looks okay some paging but no scanning (sr)

I/O this system has blocked processes and no runnable ones definate problems here; again running “iostat” command to get more information