Identifying Memory Management Bugs Within App...

Skip to ContentSun
Sun.com
About Sun
Downloads
Products
Solutions
Support
Training
 
Java
Java for your computer
Stay up to date with the latest versions of Java for your desktop computer.
Free and Open Source Java
Get your own copy of the underlying software code for the Java language.
Download the latest JDK
The basic developer kit for Java developers.
Download the Java EE SDK
The SDK supports Java SE 6 and the latest Java EE 5 technologies.
Download NetBeans IDE
Get the award-winning, open-source tool suite for developing Java applications.
Java Developer Resources
Visit java.sun.com for everything you need to know about the Java technology.
Java Developer Tools
See and download all software tools available from Sun.
Java Standard Edition
For developing and deploying Java applications for the desktop, servers, embedded, and real-time environments.
Java Enterprise Edition
For enterprise, server-side Java applications.
Java Micro Edition
For Java applications running on mobile devices.
Java Training
Sharpen your Java skills with courses from the source.
Java Support
Get dedicated help from Sun including technical assistance, product support, and support for deployed Java applications.
Solaris
OpenSolaris
Download, develop and collaborate with OpenSolaris
Solaris
Download the most advanced operating system in the world
Sun Studio
Optimizing compilers and tools for C/C++/Fortran application development
Solaris Developer Center
Explore the resources and community available to the Solaris developer.
Sun Developer Services
Get technical assistance, product support, training, and other services from the source.
BigAdmin
A community site with Solaris system administration information, hardware compatibility, a script library, and other resources for administrators of Sun products.
CommunitiesMy SDN Account
Update My Profile
» search tips 
APIs
Downloads
Products
Support
Training
Participate
SDN Home >Solaris >Reference >Technical Articles and Tips >

Article
Identifying Memory Management Bugs Within Applications Using the libumem Library

Print-friendly Version
By Robert Benson, June 2003

Contents:
Overview
Introduction
Debugging Infrastructure
Debugging Methodology
Examples
References
Overview

This article will introduce the user space slab allocator, libumem, shipped in the Solaris 9 Operating System (Solaris 9 OS), Update 3. Of particular interest is the debugging infrastructure provided by the libumem library. This paper will focus on the application developer's use of the debugging features provided by the libumem library to find and fix memory management bugs efficiently within existing code.
First, we will introduce the libumem library and briefly describe some of the advantages of using the slab allocator for application memory management. Next, we'll describe the details of the debugging infrastructure provided by the libumem library, and the tools to take advantage of the infrastructure. Finally, we will walk through a few examples using the libumem library and the Solaris OS Modular Debugger (MDB) to illustrate the ease of finding memory management bugs (that is, memory corruption and leaks).
This article was originally published on Sun's Access1 site and is reprinted with permission.
Introduction

The creation of the user space slab allocator was inspired by the kernel space slab allocator introduced in SunOS 5.4 [1].   The kernel slab allocator was created by engineers investigating system memory management in an effort to find new ways to make the virtual memory (VM) subsystem faster, more efficient, and scalable. The slab allocator provides faster and more efficient memory allocation by using an object caching strategy. Object caching is a strategy by which memory that is frequently allocated and freed will be cached, so that the overhead of creating the same data structure is decreased.This strategy has proven to be very efficient due to the large amount of variable reuse within most code. The scalability of the slab allocator has been improved upon over the last few years by using a per-CPU set of caches. This addition allowed for a far less contentious locking scheme when requesting memory from the system and thus, has created a more scalable memory allocator.  Once the slab allocator was found to work well within kernel space, the next step was to port those wins to user space. Hence, the user space slab allocation library, libumem, was created. Beginning with the Solaris 9 OS, Update 3, the libumem library will be a standard part of the Solaris OS.
The user space slab allocator is based upon a set of umem caches whose size is determined before the first allocation. The umem caches are built using slabs of memory from the system. The slab nomenclature denotes one or more contiguous virtual memory (VM) pages which are split into equal size chunks called buffers. The buffer contains the user's data and in addition can, depending on the environment settings, contain the debug information that will help the application developer find and repair memory management bugs.
For a detailed discussion about the structure and theory behind the slab allocator, please refer toThe Slab Allocator: An Object-Caching Kernel Memory Allocator, by Jeff Bonwick, andMagazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources, by Jeff Bonwick and Jonathan Adams.
Debugging Infrastructure

Here we will describe the sections of the buffer created when an application requests memory resources. In addition, the meanings of the boundary values seen within this buffer will be explained. This is intended to provide the developer with an understanding of the way in which the libumem library sets up the infrastructure by which the application's memory transactions can be scrutinized for validity.
Anatomy of the Buffer
The buffer is divided into four sections, as seen in Figure 1.
Metadata section
User data section
Redzone section
Debug
metadata section
Figure 1: The structure of a buffer created by the libumem library
The first section is devoted to storing 8 bytes of metadata with which we will not concern ourselves in this article. The second section contains the memory that the application will use to store its data. The third section is called the redzone, and purposely separates the user data and debug metadata sections. In addition, the redzone section contains a value that can be used to determine the size of the application's memory request. The fourth, and final, section is used to store the debug metadata which the developer can use to determine the state and history of the buffer.
User Data Section
The user data section is the portion of the buffer which is reserved for the application's data. To understand the functionality of this section of the buffer we must understand the basic building blocks of the slab allocator, the umem caches. The slab allocator is based upon umem caches that consist of buffers with predetermined sizes. Thus, when an application requests memory from the system, the system will allocate memory from the umem cache that has a user data section of equal or greater size than the request. The size of the user data section will typically be larger than the amount of memory requested by the application, as seen in Figure 2.
Memory available to the application 0xbb Memory not available to the application
Figure 2:  The structure of the user data section
Each umem cache consists of a set of buffers of one predetermined size in order to facilitate object reuse and to minimize memory fragmentation within the system. Therefore, most of the memory allocation requests by an application do not require the full amount of space provided by the buffer's user data section.
The memory requested by the application begins at the start of the user data section and ends at the boundary value of 0xbb. The 0xbb boundary value is placed just after the last byte of memory requested by the application. The memory between the 0xbb value and the start of the redzone section, the next section within the buffer,  is not to be used by the application. In the following output from MDB, the 0xbb value is written just after the tenth byte in the user data section, as is appropriate for a 10-byte application allocation request.
> 0x49fc0/10X 0x49fc0: 12 3a10bfee baddcafe baddcafe baddbbfe baddcafe feedface 11a7 50000 a115c8ed
Note: The previous hexadecimal dump is the output of a MDB command. This particular command displays ten 4 byte hexadecimal values starting at address 0x49fc0. This output represents an entire libumem buffer starting from the address 0x49fc0. Please refer to the documentation athttp://docs.sun.com for more details about MDB.
If the application requests an amount of memory which happens to be exactly the same size as the user data section predetermined by the size of the umem cache, the 0xbb value will occupy the first byte of the redzone section.
Please note that the value 0xbaddcafe is written to all of the uninitialized memory segments within the buffer's user data section. This is a feature of the debugging infrastructure provided by the libumem library in order to determine when an application is accessing data that has not been previously initialized.
Redzone Section
This section of the buffer is 8 bytes in size and is used to differentiate between the user data section and the debug metadata section within the buffer. The boundary value 0xfeedface indicates the beginning of the redzone section, as can be seen below.
> 0x49fc0/10X 0x49fc0: 12 3a10bfee baddcafe baddcafe baddbbfe baddcafe feedface 11a7 50000 a115c8ed
As was noted previously, if the application requests an amount of memory which happens to be exactly the same size as the entire user data section predetermined by the size of the umem cache, the 0xbb value will occupy the first byte of the  redzone section. Thus, the redzone will not start with 0xfeedface but with 0xbbedface.
The redzone boundary value can be verified to determine whether a buffer overflow has taken place. In addition, the last 4 bytes of the redzone section, 0x11a7 in the previous dump, can be used to to verify the amount of memory requested by the application. As can be seen in the /usr/include/umem_impl.h header file, this value has been encoded by the following macro:
#define UMEM_SIZE_ENCODING(x)         ( 251 * (x) + 1 )
where the value x is the size of the application's memory request, plus 8 bytes. Thus, we'll use the previous dump to verify this behavior.
> 0x11a7=D               4519
By dividing the decimal value of 4519 by 251, and then subtracting 8, we find that the application requested 10 bytes of memory from the system.
Debug Metadata
This section of the buffer contains 8 bytes that consist of a 4 byte pointer to a umem_bufctl_audit structure and a 4 byte checksum.  The umem_bufctl_audit structure, as seen within the /usr/include/umem_impl.h header file, contains the following:
typedef struct umem_bufctl_audit {     struct umem_bufctl  *bc_next;   /* next bufctl struct */     void            *bc_addr;    /* address of buffer */     struct umem_slab    *bc_slab;    /* controlling slab */     umem_cache_t        *bc_cache;   /* controlling cache */     hrtime_t        bc_timestamp;  /* transaction time */     thread_t        bc_thread;    /* thread doing transaction */     struct umem_bufctl  *bc_lastlog   /* last log entry */     void            *bc_contents;  /* contents at last free */     int            bc_depth;    /* stack depth */     uintptr_t       bc_stack[1]; /* pc stack */ } umem_bufctl_audit_t;
Of particular interest is the pointer to the stack trace for the last thread that allocated or freed the buffer. The second 4 byte value within the debug metadata section, called the bxstat value, is a checksum that can be used to verify that the buffer is in a known state. The value of the pointer to the umem_bufctl_audit structure XOR'ed to the value of the bxstat checksum should result in 0xa110c8ed for an allocated buffer (as seen below) or 0xf4eef4ee for a freed buffer. If this is not the case, the buffer has become corrupt.
> 0x49fc0/10X 0x49fc0: 12 3a10bfee baddcafe baddcafe baddbbfe baddcafe feedface 11a7 50000 a115c8ed > 50000^a115c8ed=K a110c8ed
Debugging Methodology

The malloc() and free() memory management methods are used by many application developers. An application can be written without a dependence on any particular memory management programming interface by using the standard memory management methods malloc() and free(). This section will outline the steps needed to take advantage of the libumem library to debug an application's memory transactions.
Library Interposition and libumem Flags
If the libumem library is interposed (by setting the LD_PRELOAD environment variable) when executing an application, the malloc() and free() methods defined within the libumem library will be used whenever the application calls malloc() or free(). In order to take advantage of the debugging infrastructure of the libumem library, one needs to set the UMEM_DEBUG and the UMEM_LOGGING flags in the environment where the application is being executed.  The most common values for these flags are as follows: UMEM_DEBUG=default and UMEM_LOGGING=transaction.  With these settings, a thread ID, high-resolution time stamp, and stack trace are recorded for each memory  transaction initiated by the application. In addition, the libumem library will:
Fill all the allocated and  freed memory segments within the buffer with special patterns to help detect the use of uninitialized data (0xbaddcafe) and  previously  freed buffers (0xdeadbeef). Create a redzone section after the user data section which is checked for integrity when the buffer is allocated and freed. Create a debug metadata section after the redzone section which consists of a pointer to a umem_bufctl_audit structure and a bxstat checksum.
The following are examples of the commands used to set the appropriate debug flags and interpose the libumem library when executing an application.
(csh)
%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction; setenv LD_PRELOAD libumem.so.1; ./a.out)
or
(bash)
bash-2.04$UMEM_DEBUG=default UMEM_LOGGING=transaction LD_PRELOAD=libumem.so.1 ./a.out
More details about the debug flags (UMEM_DEBUG and UMEM_LOGGING) can be found in the umem_debug(3MALLOC) man page.
MDB Commands
The developer can view the debug information pertaining to an application's memory management transactions by using MDB.  The following commands within MDB can be used to provide a great deal of information about the memory transactions that took place during the execution of the application.
::umem_status
Prints the status of the umem indicating if the logging features have been turned on or off
> ::umem_status Status: ready and active Concurrency: 1 Logs: transaction=64k Message buffer:
::findleaks
Prints a summary of the memory leaks found within the application
> ::findleaks CACHE LEAKED BUFCTL CALLER
0003d888 1 00050000 main+0xc ---------------------------------- Total 1 buffer, 24 bytes
::umalog
Prints the memory transactions initiated by the application and the correlated stack traces
> ::umalog T-0.000000000 addr=55fb8 umem_alloc_32 libumem.so.1`umem_cache_alloc+0x13c libumem.so.1`umem_alloc+0x44 libumem.so.1`malloc+0x2c main+0x18 _start+0x108 T-0.000457800 addr=49fc0 umem_alloc_24 libumem.so.1`umem_cache_alloc+0x13c libumem.so.1`umem_alloc+0x44 libumem.so.1`malloc+0x2c main+0xc _start+0x108
::umem_cache
Prints the details about each of the umem caches
> ::umem_cache ADDR NAME FLAG CFLAG BUFSIZE BUFTOTL 0003c008 umem_magazine_1 000e 80080000 8 0 0003c1c8 umem_magazine_3 000e 80080000 16 0 0003c388 umem_magazine_7 000e 80080000 32 0 0003c548 umem_magazine_15 000e 80080000 64 0 0003c708 umem_magazine_31 000e 80080000 128 0 0003c8c8 umem_magazine_47 000e 80080000 192 0 0003ca88 umem_magazine_63 000e 80080000 256 0 0003cc48 umem_magazine_95 000e 80080000 384 0 0003ce08 umem_magazine_143 000e 80080000 576 0 0003cfc8 umem_slab_cache 000e 80080000 28 170 0003d188 umem_bufctl_cache 000e 80080000 12 0 0003d348 umem_bufctl_audit_cache 000e 80080000 100 408 0003d508 umem_alloc_8 020f 80000000 8 0 0003d6c8 umem_alloc_16 020f 80000000 16 0 0003d888 umem_alloc_24 020f 80000000 24 204 0003da48 umem_alloc_32 020f 80000000 32 170 ...snip...
[address]::umem_log
Prints the umem transaction log for the application
> ::umem_log CPU ADDR BUFADDR TIMESTAMP THREAD 0 0002e064 00055fb8 10475e3dd1c98 00000001 0 0002e000 00049fc0 10475e3d62050 00000001 0003483c 00000000 0 00000000 000348a0 00000000 0 00000000 00034904 00000000 0 00000000 ... snip ...
[address]::umem_verify
Prints the integrity of the umem caches which is useful in determining if a buffer has been corrupted
> ::umem_verify Cache Name Addr Cache Integrity umem_magazine_1 3c008 clean umem_magazine_3 3c1c8 clean umem_magazine_7 3c388 clean umem_magazine_15 3c548 clean umem_magazine_31 3c708 clean umem_magazine_47 3c8c8 clean umem_magazine_63 3ca88 clean umem_magazine_95 3cc48 clean umem_magazine_143 3ce08 clean umem_slab_cache 3cfc8 clean umem_bufctl_cache 3d188 clean umem_bufctl_audit_cache 3d348 clean umem_alloc_8 3d508 clean umem_alloc_16 3d6c8 clean umem_alloc_24 3d888 clean umem_alloc_32 3da48 clean ... snip ...
address$Prints the contents of the umem_bufctl_audit structure as defined in the /usr/include/umem_impl.h header file
> 50000$Examples

The following basic examples will show how to use MDB in conjunction with the libumem library to examine the history of an application's memory transactions.
Traditional Memory Leak
In order to examine if an application has a memory leak, one can execute the following steps to narrow down the section of the code which is causing the leak.
1. The libumem library is only available on systems which are running the Solaris 9 OS, Update 3 and above.
%uname -a SunOS fountainhead 5.9 Generic_112233-05
2. Execute the application with the libumem library interposed and the appropriate debug flags set.
%(setenv UMEM_DEBUG default; setenv UMEM_LOGGING transaction; setenv LD_PRELOAD libumem.so.1; ./a.out)
3. Use the gcore (1) command to get an application core to analyze the application's memory transactions.
%ps -ef | grep a.out user1 970 714 0 10:42:42 pts/4 0:00 ./a.out
%gcore 970 gcore: core.970 dumped
4. Use MDB to analyze the core for memory leaks using the commands described in the previous section.
%mdb core.970 Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]
> ::umem_log CPU ADDR BUFADDR TIMESTAMP THREAD 0 0002e0c8 00055fb8 159d27e121a0 00000001 0 0002e064 00055fb8 159d27e0fce8 00000001 0 0002e000 00049fc0 159d27da1748 00000001 00034904 00000000 0 00000000 00034968 00000000 0 00000000 ... snip ...
Here we can see that there have been three transactions by thread #1 on cpu #0.
> ::umalog T-0.000000000 addr=55fb8 umem_alloc_32 libumem.so.1`umem_cache_free+0x4c libumem.so.1`process_free+0x68 libumem.so.1`free+0x38 main+0x18 _start+0x108 T-0.000009400 addr=55fb8 umem_alloc_32 libumem.so.1`umem_cache_alloc+0x13c libumem.so.1`umem_alloc+0x44 libumem.so.1`malloc+0x2c main+0x10 _start+0x108 T-0.000461400 addr=49fc0 umem_alloc_24 libumem.so.1`umem_cache_alloc+0x13c libumem.so.1`umem_alloc+0x44 libumem.so.1`malloc+0x2c main+4 _start+0x108
The three transactions consist of one allocation to the 24 byte umem cache, and one memory allocation and release from the 32 byte umem cache. Note that the high resolution timestamp output in the upper left hand corner is relative to the last memory transaction initiated by the application.
> ::findleaks CACHE LEAKED BUFCTL CALLER 0003d888 1 00050000 libumem.so.1`malloc+0x0 ----------------------------------------------------------------------  Total 1 buffer, 24 bytes
This shows that there is one 24 byte buffer which has been leaked.
> 00050000$We can find the stack trace for the allocation which resulted in the memory leak by dumping the bufctl structure. The address of this structure can be gathered from the previous ::findleaks output.
> 49fc0/10X 0x49fc0: 12 3a10bfee baddcafe baddcafe baddbbfe baddcafe feedface 11a7 50000 a115c8ed
Looking at the values within the buffer we see that the size of the allocation was 10 bytes. This can be calculated by dividing the redzone value of 0x11a7 by 251, and then subtracting 8 bytes.
%cat test.c #include #include #include void test_sleep(int); void main(){ int *test; test = malloc(10); // This is the memory allocation that is never freed!! test = malloc(20); free(test); test_sleep(24); } void test_sleep( int interval ){ printf("Starting to sleep for %d seconds...\n", interval); sleep(interval); printf("Stopped sleeping...\n\n"); }
Once we look at the code of the executable, we can use the function in the stack trace and the size of the allocation to determine the piece of memory which has leaked.
Traditional Memory Corruption
The following example will list the steps used to examine an application core for a memory corruption bug.
1. Follow the first two steps listed above in thememory leak example.
2. Either analyze the core dump created by the application if it aborted, or use gcore as seen above.
3. Use MDB to analyze the application core for the memory corruption using the MDB commands listed in a previous section.
%mdb core.1095 Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ] > ::umem_verify Cache Name Addr Cache Integrity umem_magazine_1 3c008 clean umem_magazine_3 3c1c8 clean umem_magazine_7 3c388 clean umem_magazine_15 3c548 clean umem_magazine_31 3c708 clean umem_magazine_47 3c8c8 clean umem_magazine_63 3ca88 clean umem_magazine_95 3cc48 clean umem_magazine_143 3ce08 clean umem_slab_cache 3cfc8 clean umem_bufctl_cache 3d188 clean umem_bufctl_audit_cache 3d348 clean umem_alloc_8 3d508 clean umem_alloc_16 3d6c8 clean umem_alloc_24 3d888 1 corrupt buffer ... snip ...
Using the umem_verify command we can see that one of the umem caches has a corrupted buffer.
> 3d888::umem_verify Summary for cache 'umem_alloc_24' buffer 49fc0 (allocated) has a corrupt redzone size encoding
This provides more detail about the type of corruption that has taken place within the 24 byte umem cache.
> 49fc0/10X 0x49fc0: 18 3a10bfe8 0 1 2 3 4 1789 50000 a115c8ed
When we dump out the buffer we can see that the size of the original allocation was 16 bytes. This can be calculated by decoding the redzone value of 1789 by dividing it by 251 and then subtracting the result by 8 bytes. Once we know the size of the allocation we can look for the 0xfeedface boundary 16 bytes from where the user data section starts. Scrutinizing the buffer above reveals that the user section is filled with 0 through 4, and there is no redzone boundary tag (that is, 0xfeedface). We find the value 4 where the redzone value should be!
>50000$Getting the stack trace for the last memory transaction will allow the developer to narrow down where in the code the memory corruption is taking place.
%cat test.c #include #include #include void test_sleep(int); void main(){ int *test; test = malloc(16); for(int i = 0; i <= 4; i++){ // CORRUPTION! The for loop should only test[i] = i; // be traversed three times instead } // of four due to the buffer size test_sleep(24); } void test_sleep( int interval ){ printf("Starting to sleep for %d seconds...\n", interval); sleep(interval); printf("Stopped sleeping...\n\n"); }
After looking at the code it is obvious that the memory corruption is due to invalid conditions in the for loop.
 
References

The Slab Allocator: An Object-Caching Kernel Memory Allocator, by Jeff Bonwick
Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources, by Jeff Bonwick and Jonathan Adams
Rate and Review
Tell us what you think of the content of this page.
Excellent   Good   Fair   Poor
Comments:
Your email address (no reply is possible without an address):
Sun Privacy Policy
Note: We are not able to respond to all submitted comments.


Oracle is reviewing the Sun product roadmap and will provide guidance to customers in accordance with Oracle's standard product communication policies. Any resulting features and timing of release of such features as determined by Oracle's review of roadmaps, are at the sole discretion of Oracle. All product roadmap information, whether communicated by Sun Microsystems or by Oracle, does not represent a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. It is intended for information purposes only, and may not be incorporated into any contract.



About Sun  | About This Site  | Newsletters  | Contact Us  | Employment  | How to Buy  | Licensing  | Terms of Use  | Privacy  | Trademarks
© 2010, Oracle Corporation and/or its affiliates
A Sun Developer Network Site

Unless otherwise licensed, code in all technical manuals herein (including articles, FAQs, samples) is provided under thisLicense.
 Sun Developer RSS Feeds

Feedback
Click here to
rate this page