Linux Test Project - LTP - A Brief

LTP

LTP is an open source project used for testing a linux kernel. The test suites can be written in c, shell or perl. In this post, I will try to explain my understanding of LTP and a sample test case that I will write and execute using LTP.

Download the source from
http://sourceforge.net/projects/ltp/files/LTP%20Source/ltp-20130904/
Upon going to the parent directory of this, You can find out more recen sources of LTP. As of now the above link is the latest suite.

WHAT LTP CONTAINS
There are test cases inside LTP and a test driver called as "pan".
Once the source is extracted, LTP can be viewed as follows.
SCRIPTS
a) IDcheck.sh -> Creates the groups and users required for running LTP
b) runltp -> main script that is used for executing LTP. It  has ability to execute tests related to Sub systems of linux kernel, commands, system calls.
c) Networktest.sh -> To verify features related to network
d) diskio.sh -> verifies disk IO related tests.

DIRECTORIES
a) scenario_groups ->has files which has groups of SCENARIOS to be tested like fs, mm, io, commands, syscalls etc.
b) runtest -> For each scenario described in scenario_groups, these is a corresponding file inside runtest directory which has the Test cases. The format of this file is <TEST CASE TAG> <TEST EXECUTABLE NAME> <Arguments if any>
c) testcases/bin -> This directory consists of each <TEST EXECUTABLE> defined in previous point.
d) testscripts -> contains the scripts that can be used for usage of runltp. Eg : ltpstress.sh etc.

COMPILE AND INSTALL LTP
Once the source is downloaded, perform the following.
1. untar
2. ./configure
3. make
4. sudo make install (will be installed in /opt/ltp)

EXECUTING DEFAULT TEST CASES
1. cd /opt/ltp
2. ./IDcheck.sh
3. sudo ./runltp -p -o output.log -f commands

Point 3 illustrates a sample execution of a particular sub sytem/feature in LTP.

WRITING CUSTOM TEST CASE AND EXECUTION

This is a sample program.
The customized test script can be added in /opt/ltp/testcases/bin/ folder
To execute this script it has to be added in one of the run test files Eg : runtest/commands and the format of this file is

Then this runtest can be executed using
         sudo ./runltp -p -o output.log -f commands

The output will be as shown below

and the output log will be of the following format

In Similar way .c file also can be compiled, added and executed.

Thanks,
Gomathi

KMEMLEAK

KMEMLEAK

Kmemleak is a feature of kernel, that helps in finding the memory leaks in kernel. It only reports the list of memory leaks in the kernel. There is a similar tool to find out memory leaks in user space which is called as Valgrind.

Memory leaks happen when some memory is allocated but not freed. In precise, there is no way to de-reference a particular portion of memory or there is no pointer pointing to that memory using which it can freed.

REQUIREMENTS

To use this particular tool, the kernel has to be built with kmemleak support. In this post, the steps to enable a kernel and ways of identifying and analyzing a leak will be discussed by writing a module that intentionally introduces kernel memory leak.

In the kernel CONFIG_DEBUG_KMEMLEAK is to be enabled. This can be done by doing following.
    From the source directory,
            make menuconfig

Once this configuration is enabled, Set the early log values in Maximum kmemleak early log entries. By default this value is set to 400. But this might not be sufficient in some cases and hence kmemleak might not work as expected. It is better to set this value to a safe 4000.

After setting these kernel configurations, save and quit. In order to detect kernel memory leak, we can write a small piece of code with a memory allocated but not freed. For reference, please use mm/kmemleak-test.c

Then the kernel can be built and booted by using the following steps.
    a) make
    b) make modules
    c) make modules_install
    d) make install
    e) update menu.lst or grub.cfg if make install doesn't automatically do it.
    f) Boot your machine with this particular kernel.

Once the board comes  up you can verify that this is your kernel by the following,
    a) uname -r (You can check the kernel version, if you have used source of a different version)
    b) zcat /proc/config.gz | grep KMEMLEAK (can be verified to check if kmemleak is enabled or not)
    c) If /proc/config.gz is not available you can check cat /boot/config-<kernel version> | grep KMEMLEAK
    d) dmesg | grep -i kmem

After confirming that the deployed kernel is kmemleak enabled, we can start our kmemleak testing and analysis.

To begin with we need to mount debugfs file system. (mount -t debugfs nodev /sys/kernel/debug). KMEMLEAK enabling in kernel will by default do this step and the mount can be done if it is not done automatically. /sys/kernel/debug/kmemleak file will be automatically created which will be our single point of reference for testing and detecting kernel memory leaks.
Multiple options can be passed to this particular file. But most commonly used ones are "scan"and "clear".
Upon doing echo clear > /sys/kernel/debug/kmemleak, all the data will be cleared off. echo scan > /sys/kernel/debug/kmemleak will scan the entire memory and will give information of the kernel memory leaks.
After a scan is performed, cat /sys/kernel/debug/kmemleak will list all the kernel memory leaks at that system state. It also gives a back trace which helps the programmer identify the point where leak happens and helps in fixing the same.

This gives following information
    a)The insmod created memory leak.
    b) The unreference object (leaked memory) is at 0xffff880012f20480
    c) The back trace shows that the leak happens in init function of the module that was inserted.
    d) The kmem_cache_alloc_trace in the bbacktrace shows that a memory is being allocated using one of the malloc variants and is not freed.
This gives all necessary information to fix this leak.

The scan will be performed every 10 minutes by default. But this value of 10 minutes can be modified using echo scan=<secs> > /sys/kernel/debug/kmemleak. All other options supported are
  off           - disable kmemleak (irreversible)
  stack=on      - enable the task stacks scanning (default)
  stack=off     - disable the tasks stacks scanning
  scan=on       - start the automatic memory scanning thread (default)
  scan=off      - stop the automatic memory scanning thread
  scan=<secs>   - set the automatic memory scanning period in seconds
                  (default 600, 0 to stop the automatic scanning)
  scan          - trigger a memory scan
  clear         - clear list of current memory leak suspects, done by
                  marking all current reported unreferenced objects grey
  dump=<addr>   - dump information about the object found at <addr>

To disable kmemleak during boot up, kmemleak=off should be passed in the kernel command line parameter. 

HOW IT DETECTS

The following things happen when kmemleak is enabled in the kernel. 

Whenever a kernel memory is allocated, the pointers are added to a prio search tree and when free is performed this is removed. During a scan, the memory is verified if there is any pointer is available that can free it. If not the memory is considered to be orphan or leaked. 

The best place to get reference is Documentation/kmemleak.txt inside the kernel source.

Thanks,

Gomathi

Interview Experience - PART II

Questions in C

1. What are static variables and static functions?
2. How to use a function in a file which is defined as a static function in a different file? (Function pointer)
3. A header file is written and included in two .c files. What is the problem in it? - Multiple definition of functions.
How can this be resolved?? (There are no problems when stdio.h is included multiple times. How is that possible?)
4. Given a string, find the number of occurrences of a sub string and print only those words that contain this sub string.
5. Write a program for pre-order traversal of a binary tree using recursion.
6. write a program for binary search in a tree.
7. Assign all the alternate bits of an unsigned short to an unsigned char.
8. What is the different stages of compilation of a c file?
9. What is the content of a .o file?
10. How to pass a command line argument to change a macro?
11. What is the difference between a MACRO and inline function?

Questions in LINUX

1. Write a simple Makefile
2. What all files will be packed in run time rpm, devel rpm.
3. How to add run time rpm as dependency to devel rpm in rpm spec file?
4. Why and when use cross compiler?
5. What is the difference between printf and write? (between library functions and system calls)