[Horwitz02] Section 11.7 Gauging Memory Performance and Capacity

Gauging Memory Performance and Capacity

Memory is needed torun everything from the smallest Unix utility to the largest instanceof an Oracle database. Running out of memory incurs an enormous penaltyin performance when the system begins to swap, so monitoring of itsperformance and usage is critical.

Virtual Memory Implementations

Unix virtual memory isa complicated subsystem that system administrators frequentlymisunderstand, which is unfortunate because it is such an importantpart of the operating system. Virtual memory makes possible thetransparent sharing of code and data between processes, the allocationof more memory than is physically present on a system, and even plays apart in file system performance on some operating systems by providingcaching functionality. Virtual memory implementations differ widelyamong the many flavors of Unix, but the following four concepts arealways present:

• Real memory

• Paging

• Swap space

• Swapping

Real Memory

Real memory is asystem's actual physical memory, known as RAM. Real memory is thefastest and most efficient way to store temporary data, so applicationsshould be given as much of it as possible. RAM comes in many formstoday: SIMMs, DIMMs, Rambus, DDR, and even custom memory boards such asthe mezzanine boards used in Sun Netra t1 105 servers. Error correction(usually ECC) is also available on most forms of RAM, though it comes at a premium.

Use Error Correction in Server Hardware

Always install RAM with error correction in server hardware. The cost is higher, but it significantly reduces the risk that a failing memory module will crash your system.

Swap Space

Swap space isan area of a disk dedicated to the temporary storage of memory, usuallyinvoked by paging and swapping process data. Some operating systemssuch as Linux do not require swap space; others, for example, Solarisrunning an Oracle database, often recommend swap spaces two to fourtimes the size of the real memory on the system. Only experience canexpose the true needs of a system.

Solaris Uses Swap for More Than One Function

Solaris uses swap space as a crash dump device when the operating system panics, so the kernel image can be debugged. The kernel writes the dump at the end of the swap partition to prevent overwriting at the next reboot, so most administrators maintain twice as much swap space as real memory on a Solaris server.

Paging

Paging is the processof transferring data from disk to memory and vice versa. The process iscalled paging because Unix memory is divided into evenly sized pages;common page sizes are 4KB and 8KB. When an executable program runs, itsimage is “paged in” from disk one page at a time. Likewise, when pagesof memory remain idle for a certain period of time, they may be pagedout to swap space to make the idle RAM available to other programs thatneed it.

Many file system operations are tied to the virtual memory subsystem (mmapin particular), so significant file system activity can result insignificant paging activity. If your system is paging constantly but isnot low on real memory, you may want to check file system I/O operations usingvmstat oriostat.

Swapping

Paging is often confusedwith swapping, but they are far from the same thing. While pagingtransfers idle parts of a process's memory footprint to disk, swappingtransfers entire processes to disk when there is not enough real memoryto satisfy new allocations. Swapping activity is usually an indicationof a low memory condition.

Displaying Swap Statistics

Each operating system has a different method for displaying swap space statistics. Using Solaris as an example,swap-l lists the swap devices that are configured on the system and the size of each, andswap-s displays the usage statistics for each swap device.

bash$ usr/sbin/swap -l
swapfile             dev  swaplo blocks   free
/dev/dsk/c0t0d0s1   32,1      16 2101536 1956640
bash$ /usr/sbin/swap -s
total: 316952k bytes allocated + 17824k reserved = 334776k used, 1045936k
available

Other operating systems have different ways of displaying this information, as summarized in Table 11.3.

Table 11.3. Commands for Display Swap Statistics on Different Unix Operating Systems

Usingvmstat to Monitor Memory Usage

The Solaris version ofvmstat can report detailed virtual memory statistics. The important fields are as follows:

Theswap andfree fields are self-explanatory,but the others bear some explanation. An abnormally high number ofpage-in operations may be alarming, but in most cases it is normal.Remember that during paging in, data is being read into memory; pagingin can also be indicative of file system activity. In most cases, youcan ignore this column.

Thefree list also plays a part in paging. Unix operating systems typicallyreserve a small amount of physical memory, called the free list, so there is always memory available for new processes. When the free list hits a low-water mark (calledlotsfreeon Solaris), the operating system scans for idle pages. The system canpage out those pages to reclaim some of that memory. Paging out isnormal behavior for most Unix systems, but excessive page-outs can havea drastic effect on the performance of a system, as it takes both CPUcycles and disk I/O to complete a paging operation.

Thescan rate specifies how many pages per second the Solaris pagedaemon isscanning for page-out eligibility. On Solaris 7 and earlier, if thisnumber remains higher than 200 for an extended period of time, thesystem needs more memory. However, on Solaris 8 the scan rate shouldalways be zero in all but the direst of memory conditions; Solaris 8introduces the file system free list, a list of free memory dedicatedto file system caching. Heavy file system activity will never force aprocess out of memory on Solaris 8; if the scan rate is above zero, itmeans that other processes are forcing page outs, and this is adefinite indicator of a memory shortage.

Paging Detail in Linux

The Linux version of vmstat does not provide the paging detail that the Solaris version provides and is not very useful for assessing paging activity.

Usingsar to Monitor Paging Activity

The tool that provides the most detail about paging activity issar.sar-p reports activity related to page-ins andsar-g reports activity related to page-outs.

bash$ sar -s 00:00 -e 00:30 -pg

SunOS chills 5.8 Generic_108528-09 sun4u    03/23/02

00:00:00  atch/s  pgin/s ppgin/s  pflt/s  vflt/s slock/s
00:05:00    5.31    0.54    1.31    7.48   19.28    0.00
00:10:00    5.42    0.91    2.71    5.10   15.21    0.00
00:15:00   37.62    9.36   36.48    5.78   14.22    0.00
00:20:00    6.98    0.92    2.77    7.27   20.25    0.00
00:25:01    4.51    0.62    1.55    5.26   14.46    0.00
00:30:00    4.06    0.44    0.82    6.76   16.70    0.00

Average    10.64    2.13    7.60    6.27   16.69    0.00

00:00:00  pgout/s ppgout/s pgfree/s pgscan/s %ufs_ipf
00:05:00     0.07     0.07     0.07     0.00     0.00
00:10:00     0.06     0.06     0.06     0.00     0.00
00:15:00     0.06     0.07     0.07     0.00     0.00
00:20:00     0.06     0.06     0.06     0.00     0.00
00:25:01     0.06     0.06     0.06     0.00     0.00
00:30:00     0.07     0.07     0.07     0.00     0.00

Average      0.06     0.07     0.06     0.00     0.00

Tools to Monitor Process Memory Usage

After you have determinedthe memory usage of the overall system, you can examine the per-processutilization. This will help you isolate the individual applications orcommands that are consuming the memory on a system. There are manytools to help us gather this information, some which we have seenbefore:top,ps, andpmap.

Usingtop to Monitor Process Memory Usage

top is very good at reporting memory usage for individual processes as well as total memory utilization. In the following example,topis run on an Oracle database server and shows the top 15 processessorted by resident memory size (RES). It also shows the real memory(RAM) and swap utilizations.

load averages:  0.69,  0.29,  0.18                                     00:41:51
146 processes: 144 sleeping, 1 running, 1 on cpu
CPU states: 60.8% idle, 35.7% user,  3.4% kernel,  0.0% iowait,  0.0% swap
Memory: 2048M real, 712M free, 923M swap in use, 8707M swap free

  PID USERNAME THR PRI NICE  SIZE   RES STATE   TIME    CPU COMMAND
20881 oracle     1  58    0  695M  625M sleep  45:08  0.04% oracle
24462 oracle     1   0    0  475M  439M run     1:13 37.58% oracle
20302 oracle     1  58    0  484M  437M sleep   9:56  0.00% oracle
20296 oracle     1  58    0  483M  437M sleep   5:50  0.00% oracle
20294 oracle     1  58    0  484M  436M sleep   7:56  0.00% oracle
20298 oracle     1  59    0  484M  436M sleep   7:29  0.00% oracle
20334 oracle     1  58    0  483M  436M sleep   6:24  0.01% oracle
29130 oracle     1  59    0  483M  436M sleep   4:05  0.01% oracle
20300 oracle     1  58    0  483M  436M sleep   7:29  0.00% oracle
16106 oracle     1  58    0  482M  436M sleep   1:33  0.02% oracle
16117 oracle     1  54    0  482M  435M sleep   1:59  0.13% oracle
16119 oracle     1  59    0  482M  435M sleep   2:09  0.00% oracle
21886 oracle     1  58    0  457M  435M sleep   4:55  0.00% oracle
21882 oracle     1  58    0  454M  432M sleep   2:30  0.00% oracle
21888 oracle     1  58    0  454M  432M sleep   1:28  0.00% oracle


  

Per-process memory consumption is reported by theSIZE andRES columns.SIZE is the total size of the process, including the text segment (code), data segment (statically allocated data), and stack.RES is the resident size of the process, or how much physical RAM is being used by the process (some versions oftop label the resident sizeRSS instead ofRES).

You may have noticed that the total size of the processes listed in thetopoutput exceeds the total physical memory on the system, approaching thetotal size of swap. These numbers are displayed despite the fact thatthe system reports 712MB of free physical memory and over 8GB of freeswap. An efficient Unix virtual memory subsystem is to blame; it allowssome parts of a process to be shared with other processes, such asexecutable code and shared libraries. Most of the reported memory usageby eachoracle process is actually shared among all of them.

Usingps to Monitor Process Memory Usage

Although not overly verbose,ps can provide some very simple memory usage statistics for a process. On a System V platform (Solaris),ps-l can show the size of a process in pages in theSZ field, as follows:

bash$ ps -lp 382
 F S   UID   PID  PPID  C PRI NI     ADDR     SZ    WCHAN TTY      TIME CMD
 8 S     0   382     1  0  40 20        ?    423        ? ?        0:05 httpd


  

Thepagesize command can be used to determine the page size on Solaris, which in this case is 8KB. So the total size of thehttpd process's address space is 423 × 8KB = 3,384KB.

The BSD version ofps provides some more useful information akin to the output oftop:ps aux will display the resident size as well as the total size of a process, just liketop. On Solaris, the BSD version ofps is located at/usr/ucb/ps; on Red Hat Linux,ps will behave with BSD semantics if you use BSD-style arguments likeaux. Note how the total sizeSZ in the following code is identical to the previous calculation:

jhorwitz@nausea[build]$ /usr/ucb/ps aux 382
USER       PID %CPU %MEM   SZ  RSS TT       S    START  TIME COMMAND
root       382  0.0  0.4 3384 1752 ?        S   Aug 08  0:04 /usr/local/apache/


  

Usingpmap to Monitor Process Memory Usage

Solaris 2.6 introduced thepmapcommand, which provides a map specifying what parts of each process areusing which parts of memory. This includes executable code, sharedlibraries, the heap, and the stack.pmap prints total usage at the bottom of the following output:

# pmap 382
382:    /usr/local/apache/bin/httpd
00010000    376K read/exec         /usr/local/apache/bin/httpd
0007C000     32K read/write/exec   /usr/local/apache/bin/httpd
00084000    120K read/write/exec     [ heap ]
FEFF0000    112K read/exec         /usr/lib/libelf.so.1
FF01C000      8K read/write/exec   /usr/lib/libelf.so.1
FF030000    208K read/exec         /usr/lib/libresolv.so.2
FF074000     16K read/write/exec   /usr/lib/libresolv.so.2
FF078000      8K read/write/exec   /usr/lib/libresolv.so.2
FF080000    744K read/exec         /usr/local/apache/libexec/mod_wl.so
FF148000     56K read/write/exec   /usr/local/apache/libexec/mod_wl.so
FF156000      8K read/write/exec   /usr/local/apache/libexec/mod_wl.so
FF170000     48K read/write/shared   [ anon ]
FF180000    672K read/exec         /usr/lib/libc.so.1
FF238000     32K read/write/exec   /usr/lib/libc.so.1
FF250000     40K read/exec         /usr/ucblib/libucb.so.1
FF26A000      8K read/write/exec   /usr/ucblib/libucb.so.1
FF270000     16K read/exec         /usr/platform/sun4u/lib/libc_psr.so.1
FF280000    552K read/exec         /usr/lib/libnsl.so.1
FF31A000     32K read/write/exec   /usr/lib/libnsl.so.1
FF322000     32K read/write/exec   /usr/lib/libnsl.so.1
FF330000      8K read/write/exec     [ anon ]
FF340000     16K read/exec         /usr/lib/libmp.so.2
FF354000      8K read/write/exec   /usr/lib/libmp.so.2
FF370000      8K read/write/exec     [ anon ]
FF380000     40K read/exec         /usr/lib/libsocket.so.1
FF39A000      8K read/write/exec   /usr/lib/libsocket.so.1
FF3A0000      8K read/exec         /usr/lib/libdl.so.1
FF3B0000    136K read/exec         /usr/lib/ld.so.1
FF3E2000      8K read/write/exec   /usr/lib/ld.so.1
FFBEA000     24K read/write/exec     [ stack ]
 total     3384K


  

pmap in Solaris 8

On Solaris 8, pmap is located in the standard path in /usr/bin. Earlier versions put pmap in /usr/proc/bin.

• 
• Create Bookmark (Key: b)Create Bookmark
• Create Note or Tag (Key: t)Create Note or Tag
• Download (Key: d)Download
• Email This Page (Key: e)Email This Page
• PrintPrint

• Html View (Key: h)Html View
• Zoom Out (Key: -)Zoom Out
• Zoom In (Key: +)Zoom In
• Toggle to Full Screen (Key: f)
• Previous (Key: p)Previous
• Next (Key: n)Next

Related Content

Swapping and Paging
From: Solaris™ Operating Environment Boot Camp

System Memory and Swap Space
From: HP-Certified HP-UX System Administration

The Swap File System
From: Solaris 10 System Administration Exam Prep: Exam CX-310-202, Part II

Memory and Swap Space Management
From: HP-Certified HP-UX System Administration

How Big Should It Be?
From: Solaris™ Operating Environment Boot Camp

How Much Swap Space Do I Need?
From: HP-UX CSE Official Study Guide and Desk Reference

Optimizing Memory
From: Windows® 98 Unleashed

The Swap File System
From: Solaris™ 9 Training Guide (CX-310-014 & CX-310-015): System Administrator Certification

Sizing Memory and the Swap Area
From: HP-UX 11i Tuning and Performance

Linux memory management
From: Linux on IBM eServer zSeries and S/390: Performance Measurement and Tuning