Saturday 26 October 2013

KMEMLEAK

KMEMLEAK

Kmemleak is a feature of kernel, that helps in finding the memory leaks in kernel. It only reports the list of memory leaks in the kernel. There is a similar tool to find out memory leaks in user space which is called as Valgrind.

Memory leaks happen when some memory is allocated but not freed. In precise, there is no way to de-reference a particular portion of memory or there is no pointer pointing to that memory using which it can freed.

REQUIREMENTS

To use this particular tool, the kernel has to be built with kmemleak support. In this post, the steps to enable a kernel and ways of identifying and analyzing a leak will be discussed by writing a module that intentionally introduces kernel memory leak.

In the kernel CONFIG_DEBUG_KMEMLEAK is to be enabled. This can be done by doing following.
    From the source directory,
            make menuconfig


Once this configuration is enabled, Set the early log values in Maximum kmemleak early log entries. By default this value is set to 400. But this might not be sufficient in some cases and hence kmemleak might not work as expected. It is better to set this value to a safe 4000.

After setting these kernel configurations, save and quit. In order to detect kernel memory leak, we can write a small piece of code with a memory allocated but not freed. For reference, please use mm/kmemleak-test.c

Then the kernel can be built and booted by using the following steps.
    a) make
    b) make modules
    c) make modules_install
    d) make install
    e) update menu.lst or grub.cfg if make install doesn't automatically do it.
    f) Boot your machine with this particular kernel.

Once the board comes  up you can verify that this is your kernel by the following,
    a) uname -r (You can check the kernel version, if you have used source of a different version)
    b) zcat /proc/config.gz | grep KMEMLEAK (can be verified to check if kmemleak is enabled or not)
    c) If /proc/config.gz is not available you can check cat /boot/config-<kernel version> | grep KMEMLEAK
    d) dmesg | grep -i kmem

After confirming that the deployed kernel is kmemleak enabled, we can start our kmemleak testing and analysis.

To begin with we need to mount debugfs file system. (mount -t debugfs nodev /sys/kernel/debug). KMEMLEAK enabling in kernel will by default do this step and the mount can be done if it is not done automatically. /sys/kernel/debug/kmemleak file will be automatically created which will be our single point of reference for testing and detecting kernel memory leaks.
Multiple options can be passed to this particular file. But most commonly used ones are "scan"and "clear".
Upon doing echo clear > /sys/kernel/debug/kmemleak, all the data will be cleared off. echo scan > /sys/kernel/debug/kmemleak will scan the entire memory and will give information of the kernel memory leaks.
After a scan is performed, cat /sys/kernel/debug/kmemleak will list all the kernel memory leaks at that system state. It also gives a back trace which helps the programmer identify the point where leak happens and helps in fixing the same.
This gives following information
    a)The insmod created memory leak.
    b) The unreference object (leaked memory) is at 0xffff880012f20480
    c) The back trace shows that the leak happens in init function of the module that was inserted.
    d) The kmem_cache_alloc_trace in the bbacktrace shows that a memory is being allocated using one of the malloc variants and is not freed.
This gives all necessary information to fix this leak.

The scan will be performed every 10 minutes by default. But this value of 10 minutes can be modified using echo scan=<secs> > /sys/kernel/debug/kmemleak. All other options supported are
  off           - disable kmemleak (irreversible)
  stack=on      - enable the task stacks scanning (default)
  stack=off     - disable the tasks stacks scanning
  scan=on       - start the automatic memory scanning thread (default)
  scan=off      - stop the automatic memory scanning thread
  scan=<secs>   - set the automatic memory scanning period in seconds
                  (default 600, 0 to stop the automatic scanning)
  scan          - trigger a memory scan
  clear         - clear list of current memory leak suspects, done by
                  marking all current reported unreferenced objects grey
  dump=<addr>   - dump information about the object found at <addr>

To disable kmemleak during boot up, kmemleak=off should be passed in the kernel command line parameter. 

HOW IT DETECTS

The following things happen when kmemleak is enabled in the kernel. 
Whenever a kernel memory is allocated, the pointers are added to a prio search tree and when free is performed this is removed. During a scan, the memory is verified if there is any pointer is available that can free it. If not the memory is considered to be orphan or leaked. 

The best place to get reference is Documentation/kmemleak.txt inside the kernel source.

Thanks,
Gomathi

2 comments:

  1. Hi, I followed your steps for Android kernel and successfully enabled kmemleak under kernel Hacking mode. But after build when I push that kernel to my mobile and grep dmesg it gives me kmemleak=off. Do you have any comment/Idea about this? I want to enable kmemleak tool on Android kernel. Please let me know.

    Thank you.

    ReplyDelete
    Replies
    1. You can check if the kernel cmdline parameters have kmemleak=off

      Delete