[Milberg09] Chapter 16. AIX 6.1

Chapter 16. AIX 6.1

Manyof the changes in AIX 6.1 are really less about kernel innovations andmore about ancillary features, such as improving default parameters tomore accurately reflect real-world data processing. Other enhancementsinclude restricted tunables, unique tunable documentation (a usefulfeature that provides help messages via the new -h option for the tunable commands, including ioo, nfso, no, raso, schedo, and vmo), and various other improvements to certain subsystems.

16.1. Introduction

AIX 6.1 provides many important innovations and improvements, including enhancements in the following categories:

• Virtualization — Features such as workload partitioning and Live Application Mobility

• Security — Features such as encrypted file systems, trusted AIX, and role-based access control (RBAC)

• Availability — Features such as AIX concurrent updates and dynamic tracing

• Manageability — Features such as the new Systems Director Console for AIX and the Workload Partition Manager

The6.1 release also provides support for POWER6 performance innovations,such as advanced simultaneous multithreading, shared dedicatedprocessors, and variable page size. It's important to fully understandwhich innovations and enhancements are a reflection of POWER6, AIX 6.1,or a combination of both. For example, from a purely operating systemperspective, AIX improves on the older tunable defaults for the aio, ioo, nfso, no, schedo, and vmo commands.

AlthoughAIX 6.1 includes some real performance enhancements, such asimprovements in I/O pacing and AIX's implementation of asynchronous I/O(AIO) servers, I must say that there is nothing breathtakinglydifferent. In fact, IBM made more performance changes from AIX 5.1 toAIX 5.2 and from 5.2 to 5.3 (including new monitoring and tuning tools,new tunables that changed how you set Virtual Memory Manager settings,and concurrent I/O improvements) than you will see in moving from AIX5.3 to AIX 6.1. In AIX 6.1, all the tuning commands remain the same(except for those that have been taken away, such as aioo),and there are no new monitoring tools. Other changes reflect updatesmade to the utilities to reflect support for other workloadpartitioning innovations; the updated utilities include curt, filemon, iostat, netpmon, pprof, procmon, proctree, svmon, topas, tprof, and vmstat.

Workloadpartitions (WPARs) enable the use of separate virtual partitions withinone AIX image. This feature is more of a complement to logicalpartitions (LPARs) than a replacement for them. WPARs actually runinside LPARs and are similar in concept to Solaris containers.

I'vebuilt WPARs in less than 15 minutes. In fact, we'll do some of ouranalysis inside WPARs so that you can actually view some of the updatedtools that now support WPARs. Note that WPARs are possible only in AIX6.1, and a POWER6 is not necessary. Some commands also run differentlyor don't run at all within WPARs; we'll discuss a few of these whereapplicable.

16.2. Memory

Throughthe years, many have complained about the default VMM parameters ofAIX. The complaints have been that the default parameters just haven'treflected the reality of what most users run on top of AIX — forexample, mission-critical database applications such as Oracle. Becauseof this, systems administrators have had to change the default settingson many subsystems — most notably those related to virtual memory(i.e., minperm and maxperm).IBM engineering has listened and in AIX 6.1 has changed the parametersto reflect that reality. Note that you shouldn't rely exclusively onthese settings. Further, always check with your ISV to verify itsrecommended settings for AIX 6.1; then make changes accordingly.

Themost important changes to default settings were made to address pagingissues, where database servers frequently page out computational pageseven though the system has enough free memory. In the AIX 5.3 memorytuning discussion, I recommended changing the relevant parameters todefaults fairly close to what was indicated on the table. The changesare indicated in the AIX 5.3 tuning recommendation column.

InAIX 6.1, IBM now classifies many tunables as "restricted" in an attemptto discourage junior administrators from changing certain parametersdeemed critical enough to be classified as restricted. The net is thatyou can change only 29 vmotunables without receiving a firm warning; 30 others are now deemedrestricted tunables, which IBM officially states should not be modifiedunless instructed by "IBM support professionals."

A new vmo flag, -F,lets you view all the parameters, including the restricted ones. Thefollowing snippet of content includes an example from the restrictedsection.

# vmo -F -a

   force_relalias_lite = 0
      vmm_default_pspa = −1
##Restricted tunables
              maxperm% = 90

Even restricted tunables can be changed. If you make such a change, you just receive a stern warning:

# vmo -o maxperm%=99

Setting maxperm% to 99
Warning: a restricted tunable has been modified

Whena restricted parameter is changed after a reboot, you'll receive afurther rebuke and be asked to confirm whether you really want to makethe change. You'll have to physically type in "yes" to reply.

The most important out-of-the-box performance changes related to memory include new values for minperm, maxperm, maxclient, and strict_maxclient. This update is a continuation of changes that first appeared in AIX 5.3, when you no longer had to turn off strict_maxclient, increase minfree and maxfree, or reduce minperm, maxperm, and maxclient. The new recommendation (now incorporated as the default value in AIX 6.1) is to turn off the repage ratio check (lru_file_repage) to ensure that working storage is not paged and to consider only file paging.

InAIX 6.1, the VMM replacement default is changed to use up to 90 percentof its real memory for file caching, favoring computational pages overfile pages. Unless the amount of active virtual memory exceeds 97percent of the size of real memory, minperm is reduced to 3 percent to ensure that computational pages will not be stolen. Let's try changing it:

# vmo -o minperm%=97
Value of the tunable minperm% cannot be changed in a WPAR

Asyou can see from the error message, some changes will not work inWPARs. WPARs are a subset of an LPAR, but they are still part of thesingle operating system image.

Anotherimportant change includes VMM dynamic variable page size support(VPSS). Pages are defined as fixed-length data blocks held in virtualmemory. In AIX 6.1 (on POWER6 processors only), VMM can now dynamicallyuse the larger page size based on application memory usage, whichshould substantially improve performance. This feature is completelytransparent to applications. AIX uses the larger page size only ifdoing so does not result in increasing process memory usage. The use oflarger pages improves performance because fewer hardware addresstranslations need to be made. This feature is supported only forworking storage memory, not persistent storage. The new parameter is vmm_default_pspa (it works in conjunction with the existing vmm_mpsize_support tunable).

Let's view the tunable setting for VPSS:

vmo -a | grep pspa
    vmm_default_pspa = −1

16.3. CPU

In AIX 6.1, only 27 of the schedocommand's 42 CPU-related tunables are restricted, leaving 15 parametersthat you can modify without explicit warnings. Although some defaultshave changed, no substantial changes have been made with respect to CPUmonitoring and tuning in AIX 6.1.

16.4. Disk I/O

Of the 48 tunables you can control with the iooI/O tuning command, 27 are now restricted, leaving 21 that you canmodify without explicit warnings. The most important changes affect I/Opacing and AIO dynamic tunables.

16.5. JFS2

16.6. iSCSI

16.7. I/O Pacing

16.8. Asynchronous I/O

AIOis an AIX software subsystem that permits processes to issue I/Ooperations without waiting for I/O to finish. Because I/O operationsand application processing operate concurrently, they essentially runin the background and improve performance. This advantage isparticularly important in a database environment.

Thereare two types of AIX subsystems: Legacy AIO and POSIX AIO. Thedifferences between them involve different parameter passing at theapplication layer. In other words, the developers pick theimplementation that the application uses. Regardless of which subsystemis chosen, both run concurrently on AIX. In AIX 5L, if applications useAIO, the subsystem must be explicitly activated in the autoconfigparameter. The system also requires a reboot because the kernelextensions must be loaded. In fact, any release before AIX 5.3 TL_5requires reboots if any changes are made to the following tunables: maxreqs, maxservers, and minservers.

AIX 5.3 provided the aioocommand, which lets you make these changes dynamically without a reboot(decreasing required reboots). This command does not change the ObjectData Manager (ODM) attributes, meaning that changes will not persistacross a reboot. In AIX 6.1, the tunables fastpath and fsfastpath are now restricted and are set to 1 by default. The new setting has the following effect on the tunables:

• fastpath — AIO requests that raw logical volumes be passed directly to the disk layer.

• fsfastpath — AIO requests that files opened with concurrent I/O on JFS2 be passed directly to the Logical Volume Manager or to disk.

##Restricted tunables
   aio_fastpath = 1
 aio_fsfastpath = 1

Further,AIO subsystems are now loaded by default and not activated. They arestarted automatically when the application initiates the AIO I/Orequests. AIX 6.1 no longer provides the aioo command (what a short life span), and these tunables are now used only with ioo.

The old method (AIX 5.3):

# # aioo -a

            minservers = 1
            maxservers = 1
             maxreqs= 4096
            fsfastpath = 0

The new method with AIX 6.1:

# ioo -a | grep active
                    aio_active = 0
              posix_aio_active =

It's worth noting that there are no more AIO devices in the ODM. Two new parameters have also been added to ioo: aio_active and posix_aix_active.These settings can be changed only by AIX, and they are set to 1 onlywhen AIO kernel extensors are used and pinned. If you do a grep, you won't find any more AIO servers. You'll now see aioLpools and aioPpools,the kernel processes that manage the AIO subsystems for Legacy andPOSIX. As a result of this change, there is less pinned memory andfewer processes running on the system; both have positive effects onoverall systems performance.

Here's a look at the new AIO kernel processes:

# pstat -a | grep aio
 39 a   2704e      1  2704e     0     0      1  aioLpool
 40 a   28050      1  28050     0     0      1  aioPpool

The minserver and maxserverparameters, as they relate to AIO servers, are now tuned per each CPUtunable. Changing these values will not result in changes to the numberof available servers on the system; that number depends on the numberof concurrent I/O requests.

The following shows the new default values for minservers and maxservers:

# ioo -a | grep minservers
                aio_minservers = 3
          posix_aio_minservers = 3

# ioo -a | grep maxservers
                aio_maxservers = 30
          posix_aio_maxservers = 30

16.9. Network

Of the 133 no command tunables, IBM has classified only these five as restricted:

#no  -F -a

##Restricted tunables
         extendednetstats = 0
          inet_stack_size = 16
        net_malloc_police = 16384
             pseintrstack = 24576
                 use_isno = 1

Anew network caching daemon has also been introduced to improveperformance when resolving names using Domain Name Server (DNS). Youcan start this daemon from the System Resource Controller (SRC). Itsmain configuration file is /etc/netcd.conf, and you can copy the one in /usr/samples/tcpip to /etc and use that as a template.

The command used to manage the daemon is netcdctrl.With this command, you can dump the cache contents to a file, displaycache usage statistics, flush the cache table, and change the logginglevel of the daemon.

16.10. NFS

Ofthe 24 Network File System (NFS) tunables, IBM has classified 21 asrestricted. The only noteworthy change here is that RFC1323 (on theTCP/ IP stack) is now enabled by default, letting TCP connections usethe TCP scaling window for any NFS connections. The default number of biod daemons has also increased to 32 for each NFS Version 3 mount point.