vmstat Case Studies

Posted April 1, 2004 by Michilimackinac in Solaris


When running vmstat, we need to be mindful of three important points. First, running vmstat impacts the system. In fact, running vmstat with less than a
five second interval is not recommended. Second, on Solaris, the first vmstat
entry is bogus. Some of the bogus entries appear authentic, but they should be ignored. Finally, when running these, we need to take snapshots at different times to compare since systems sometimes have busy periods.


These three formulas provide a logical starting point:

CPU shortage when procs->r column is greater than (number of CPU’s*4)
procs->r is the number of runnable processes that are ready to go.

I/O problem (disks, controllers etc.) when procs->b is greater than 0
procs->b is the number of processes blocked waiting for I/O

Memory shortfall when “sr” column is greater than 150
sr is scan rate which is the rate a which the system is scanning for
free pages of memory.

We can examine the following data:


Note: assuming 4 CPUs in the below examples


% vmstat 5 10
procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s1 s1 s1 sd in sy cs us sy id
31 6 0 4542592 1747840 378 3631 94 262 245 0 0 0 25 25 0 2365 17793 4086 62 18 20
52 2 0 4446880 1556736 325 2474 35 300 294 0 0 0 0 0 0 1681 22277 2279 83 11 6
70 1 0 4451632 1557256 567 5702 35 404 390 0 0 0 2 2 0 1879 24490 2432 79 17 4
56 2 0 4461664 1562864 553 6001 60 289 265 0 0 0 1 1 0 1611 16779 2288 69 17 14
58 2 0 4454664 1562192 450 3765 179 374 356 0 0 0 0 0 0 1848 20948 2345 84 14 2
62 2 0 4457192 1561640 360 3179 9 504 491 0 0 0 0 0 0 1789 22680 2201 83 13 5
12 2 0 4448856 1553080 305 2701 22 216 208 0 0 0 1 2 0 1499 12331 1934 84 10 7
97 2 0 4448936 1556288 615 5030 43 582 571 0 0 0 1 1 0 2128 29096 2560 84 16 0
54 1 0 4453744 1559144 770 8629 3 286 272 0 0 0 0 0 0 1988 20503 2553 78 21 1
6 2 0 4459680 1561424 272 2773 27 280 265 0 0 0 1 1 0 1373 10061 2027 63 9 28


CPU bottleneck on a 4 CPU system over 16 is a problem, and this is consistently above that.

Memory is acceptable since scan rate sr is 0, and there is not much paging activity.

I/O is blocked because there are non-zero values under the proc->b column. We need to look at disks controllers with the “iostat” command to get more information.


procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s3 -- in sy cs us sy id
0 0 0 143184 53920 0 2 0 0 0 0 0 0 0 2 12 13 0 0 99
0 0 0 307400 1820 8 67 537 1128 1204 0 271 172 0 545 1356 341 2 45 53
0 0 0 307396 1564 3 59 476 974 993 0 45 166 0 525 1122 286 2 35 63
0 0 0 307396 1228 0 91 716 1207 1333 0 225 203 0 600 1597 379 4 46 51
0 0 0 307408 1344 0 87 699 1207 1352 0 118 200 0 595 1590 368 2 47 51
0 0 0 307412 1340 0 87 636 1232 1376 0 194 181 0 538 1485 351 3 40 57
0 0 0 307396 1440 0 115 666 1234 1347 0 190 185 0 543 1800 370 4 51 45
0 0 0 307392 1456 0 87 700 1368 1534 0 252 195 0 573 1584 384 3 44 53
0 0 0 307392 1520 0 82 660 1301 1339 0 100 193 0 579 1494 354 2 42 56
0 0 0 307392 1320 0 83 667 1083 1194 0 149 195 0 583 1514 359 3 45 52
0 0 0 307404 1540 0 83 669 1216 1332 0 158 196 0 593 1514 354 5 46 49
0 0 0 307404 1372 1 82 661 1242 1361 0 158 194 0 586 1518 362 2 45 54
0 0 0 307404 1428 0 92 742 1320 1495 0 203 206 0 597 1678 392 4 52 44


CPU’s are acceptable here. There are no cpu’s blocked, waiting for I/O or runnable processes.

Memory is in short supply here. We are doing some memory intensive operations and will soon need more memory since sr has a value consistently above 150.

I/O appears to be satisfactory, looking at the vmstat data.


procs memory page disk faults cpu
r b w swap free re mf pi po fr de sr s6 s3 s4 s4 in sy cs us sy id
0 3 0 8895904 6139168 2798 210 1275 4 4 0 0 0 2 0 2 863 1877 688 22 9 69
0 3 0 8225008 5616824 3105 1 26808 0 0 0 0 0 2 0 2 2865 4674 2304 20 6 74
0 4 0 8224144 5620464 3681 2471 27750 4 4 0 0 0 1 0 1 2918 10765 3044 24 10 66
0 4 0 8224944 5634848 3496 0 28283 6 6 0 0 0 0 0 0 2866 5254 2427 23 6 71
0 8 0 8224944 5643128 3254 0 26294 0 0 0 0 0 3 0 4 3295 4696 3358 20 8 72
0 6 0 8224328 5644360 3167 105 26665 0 0 0 0 0 0 0 0 2930 4161 2447 17 6 77
0 3 0 8223912 5648304 3314 0 26913 3 3 0 0 0 0 0 0 2839 3815 2344 16 6 78

0 5 0 8223912 5645112 3494 0 28464 3 3 0 0 0 2 0 2 3030 5123 2644 22 7 71
0 5 0 8223912 5643728 3362 0 28664 0 0 0 0 0 0 0 0 2869 5343 2284 22 7 71
0 7 0 8223912 5634344 3522 0 29249 3 3 0 0 0 18 0 17 3087 5386 2410 22 7 70


Cpu’s are blocked, waiting for I/O, so they are showing up as idle cpu->idle

Memory on this system looks satisfactory. There is some paging but no scanning. (sr)

I/O this system has blocked processes, and there are no runnable ones. Definite problems are apparent. Once again, we are running the “iostat” command to get more information.

The Conversation

Follow the reactions below and share your own thoughts.