Systemtap

Targeted Audience

 * This page provides all the necessary details to enable systemtap on OMAP platforms (Android or Poky).
 * The information given below is the result of the integration on OMAP platform (ARM) of systemtap rel 1.0 with linux kernel rel 2.6.27 (and later releases). However, it is kept up to date with latest kernel and systemtap releases that have been tested

Systemtap Documentation

 * SystemTap: Instrumenting the Linux Kernel for Analyzing Performance and Functional Problems


 * Systemtap language reference

Systemtap Source Code

 * links to systemtap GIT or HTTP or FTP

OMAP Systemtap Scripts/Tapsets

 * OMAP tapsets and scripts releases (available soon)

Systemtap and Cross Compilation

 * As shown in the figure [[Image:Cross compilingsystemtap.jpg|thumb|widthpx|Systemtap & cross compilation]] systemtap execution consists in several phases:


 * 1. Script ".stp" parsing, script analysis to find probes and translation in C code (Pass 1, 2 and 3)
 * 2. Compilation of C code to kernel module * .ko (Pass 4)
 * 3. The load and execution of the *.ko module


 * Originally, systemtap was designed so that all 3 steps take place on the same system (for eg Ubuntu PC). Indeed, the system translating/building the scripts was also the one whose performance had to be analyzed.


 * For embedded systems, the 4 steps are shared between a host platform (Ubuntu or other linux based environment) and the target platform (OMAP ARM).


 * To use systemtap for embedded systems, the HOST system shall now support:


 * 1. Script parsing, script analysis to find probes and translation in C code (Pass 1, 2 and 3) => stap binary
 * 2. * .ko module build (Pass 4) => staprun/stapio binaries


 * These 3 steps are performed by the stap command execution on the host platform.


 * On the other hand, the target platform shall support:


 * 3. The load and execution of the *.ko module


 * This 3rd step is performed by the staprun command execution on the target platform under analysis.


 * Naturally, the *.ko module(s) shall be copied from the HOST to the TARGET platform between step 2 and step 3.


 * It may be worth noting that once step 2 is completed and *.ko files are transferred, step 3 execution and completion solely relies on the TARGET platform; the HOST platform is not needed anymore

Systemtap Install
Following steps are needed:
 * get elfutils package
 * patch kernel code
 * enable some kernel features and rebuild kernel
 * get systemtap code and build Host/Target tools
 * Build systemtap script into a kernel module
 * Copy target tool and kernel module on target. Execute it

Pre-requisite

 * In order to compile the tool and scripts, elfutils package shall be available on the host.


 * Older revisions of elfutils packages can be taken from following location, where code is fullu packaged (autoconf was run). git clone https://git.fedorahosted.org/git/elfutils.git would be a cleaner solution but you need to run autoconf, which can be quite tricky.


 * No need to install elfutils on the host; only the untar'ed source is required.


 * For the compilation examples given below, elfutils was untar'ed on the home directory
 * Therefore, elfutils path on the system taken as an example in this wiki page is /home/xxx/elfutils-0.143

Kernel Code
To run, systemtap requires the activation of the following kernel compilation flags:


 * CONFIG_DEBUG_FS: Kernel Hacking -> Debug Filesystem
 * CONFIG_KPROBES: General Setup -> Kprobes
 * CONFIG_DEBUG_INFO: Kernel hacking -> Compile the kernel with debug info
 * CONFIG_RELAY: General Setup -> user space relay support
 * CONFIG_PROFILING: General Setup -> Profiling support (EXPERIMENTAL) + General Setup -> Oprofile system profiling (EXPERIMENTAL)
 * CONFIG_TRACEPOINTS (via CONFIG_SLAB), generally already enabled

To do so, it is suggested to modify the kernel settings directly accessing menuconfig (make ARCH=arm menuconfig)

Furthermore, before launching the kernel compilation, one kernel file shall be updated so that systemtap release works for ARM architecture. Patch is no longer needed in recent kernels and can even crash it (Gingerbread)

The file is ptrace.h located in kernel/arch/arm/include/asm/ directory
 * 1) define interrupts_enabled(regs) \

(!((regs)->ARM_cpsr & PSR_I_BIT))


 * 1) define fast_interrupts_enabled(regs) \

(!((regs)->ARM_cpsr & PSR_F_BIT))

++ #define condition_codes(regs) \

++        ((regs)->ARM_cpsr & (PSR_V_BIT|PSR_C_BIT|PSR_Z_BIT|PSR_N_BIT))

/* Are the current registers suitable for user mode?

* (used to maintain security in signal handlers)

*/


 * Once the extra lines are added, the kernel shall be recompiled as per the OMAP release build instructions.

Systemtap tool
Systemtap can allow you to trace and probe any part of the kernel. Different version of systemtap should be used according to you kernel version. the list under give the basic reference that have been done on TI kernel * systemtap 1.0 : 2.6.32 * systemtap 1.3 : 2.6.35 * systemtap 1.5 : 2.6.39/3.0 * systemtap 1.6/1.7/1.8 : 2.6.39 -> 3.3 * systemtap 2.0: -> 3.6

Systemtap 1.0 code update for OMAP ARM platforms
Following patches are required to make the tool operate properly:
 * /systemtap-1.0/Translate.cxx
 * /systemtap-1.0/runtime/Sym.h
 * /systemtap-1.0/runtime/Sym.c


 * Systemtap compilation for the host
 * To compile systemtap executable used on the host side, proceed as follows:

./configure --exec-prefix=/home/xxx/systemtap-1.0-ubuntu --prefix=/home/xxx/systemtap-1.0-ubuntu --with-elfutils=/home/xxx/elfutils-0.143

make install
 * /home/xxx/systemtap-1.0-ubuntu will then be the default directory where executables will be generated.


 * Systemtap compilation for the target
 * In order to compile the executable run on the target side (staprun), proceed with the following commands:

ac_cv_file__usr_include_nspr=no ac_cv_file__usr_include_nspr4=no ac_cv_file__usr_include_nss=no ac_cv_file__usr_include_nss3=no ac_cv_func_malloc_0_nonnull=yes ./configure --exec-prefix=/home/xxx/systemtap-1.0-macharm --prefix=/home/xxx/systemtap-1.0-macharm --host=arm-none-linux-gnueabi CXXFLAGS=-static CFLAGS=-static --disable-translator --disable-docs --disable-refdocs --disable-grapher --without-rpm

make install
 * In the example above, the directory containing the target executable will then be /home/xxx/systemtap-1.0-macharm

Systemtap 1.3 code update for OMAP ARM platforms
Since Kernel 2.6.35 Systemtap version needs to be updated to 1.3 in order to be able to run some of the scripts. New compilation procedure is described under. You need first to download EFL utils package like for Systemtap 1.0.


 * Systemtap compilation for the host

To compile systemtap executable used on the host side, proceed as follows:

(You need to have codesourcery inside your PATH)

git clone git://sources.redhat.com/git/systemtap.git cd systemtap git checkout release-1.3 ./configure --exec-prefix=/home/xxx/systemtap-1.3-ubuntu --prefix=/home/xxx/systemtap-1.3-ubuntu --with-elfutils=/home/xxx/elfutils-0.143

make install


 * /home/xxx/systemtap-1.3-ubuntu will then be the default directory where executables will be generated.


 * Systemtap compilation for the target

In order to compile the executable run on the target side (staprun), proceed with the following commands: make clean

ac_cv_file__usr_include_avahi_common=no ac_cv_file__usr_include_avahi_client=no ac_cv_file__usr_include_nspr=no ac_cv_file__usr_include_nspr4=no ac_cv_file__usr_include_nss=no ac_cv_file__usr_include_nss3=no ac_cv_func_malloc_0_nonnull=yes ./configure --exec-prefix=/home/xxx/systemtap-1.3-macharm --prefix=/home/xxx/systemtap-1.3-macharm --host=arm-none-linux-gnueabi CXXFLAGS=-static CFLAGS=-static --disable-translator --disable-docs --disable-refdocs --disable-grapher --without-rpm

make install
 * In the example above, the directory containing the target executable will then be /home/xxx/systemtap-1.3-macharm
 * Post compilation patch

Change line 59 of /home/xxx/systemtap-1.3-macharm/share/systemtap/runtime/stack-arm.c (you will get an error later otherwise pointing to this file) --		_stp_print_addr((unsigned long)pc, verbose, task); ++		_stp_print_addr((unsigned long)pc, verbose, tsk);

Systemtap 1.5/1.6/1.7/1.8 code update for OMAP ARM platforms
Since Kernel 2.6.39 and 3.0 Systemtap version needs to be updated to 1.5 in order to be able to run some of the scripts. v1.7 is preferred and described below (use older versions by replacing 1.7 by right version number)

v1.8 is required for kernel 3.4. v1.8 requires elfutils v 0.148

New compilation procedure is described under. You need first to download ELF utils package like for Systemtap 1.0.


 * Systemtap compilation for the host

To compile systemtap executable used on the host side, proceed as follows:

(You need to have codesourcery inside your PATH)

git clone git://sources.redhat.com/git/systemtap.git cd systemtap git clean -dfx git checkout release-1.8 git checkout release-1.7 or git checkout 2cf32f06fb5d880ce4f5a9abbd4f9d3874a0b852 or git checkout release-1.5 ./configure --exec-prefix=/home/xxx/systemtap-1.7-ubuntu --prefix=/home/xxx/systemtap-1.7-ubuntu --with-elfutils=/home/xxx/elfutils-0.143 (or 0.148)

make install


 * /home/xxx/systemtap-1.7-ubuntu will then be the default directory where executables will be generated.


 * Systemtap compilation for the target

In order to compile the executable run on the target side (staprun), proceed with the following commands: make clean

ac_cv_file__usr_include_avahi_common=no ac_cv_file__usr_include_avahi_client=no ac_cv_file__usr_include_nspr=no ac_cv_file__usr_include_nspr4=no ac_cv_file__usr_include_nss=no ac_cv_file__usr_include_nss3=no ac_cv_func_malloc_0_nonnull=yes ./configure --exec-prefix=/home/xxx/systemtap-1.7-macharm --prefix=/home/xxx/systemtap-1.7-macharm --host=arm-none-linux-gnueabi CXXFLAGS=-static CFLAGS=-static --disable-translator --disable-docs --disable-refdocs --disable-grapher --without-rpm

make install
 * In the example above, the directory containing the target executable will then be /home/xxx/systemtap-1.7-macharm

Systemtap installation on the target

 * The two previous steps consisted in generating systemtap executables for the host and the target. The tool directory generated for the target (/systemtap-1.x-macharm) shall by definition be transferred to the target.
 * One shall pay attention that the paths indicated for the tool compilation remain hard coded. In the example above, the tool running on the target is generated in a directory called /home/xxx/systemtap-1.0-macharm. It is important to keep the directory location and structure on the target or the tool will not solve all its code dependencies.
 * For example on Android:


 * copy the folder in /data
 * at each boot execute: ln -s /data/systemtap-1.x-macharm /home/xxx/systemtap-1.x-macharm (you may need to make "/" writable through mount -o remount,rw rootfs / then create /home/xxx)


 * Next, you need to copy kernel modules on the target. There are 2 solutions:


 * copy modules anywhere (/.../systemtap-1.x-macharm/bin recommended) and launch them through staprun /.../zzz.ko
 * by default, staprun program looks for *.ko modules in the /lib/modules/xxx/systemtap directory, where xxx is related to your kernel. Copy .ko there and launch them through staprun zzz without .ko suffix
 * (To get xxx, simply execute "staprun xyz" on the target, and an error will be displayed indicating the right path as: "ERROR: Unable to canonicalize path "/lib/modules/2.6.29-omap1-00034-g8b1c197-dirty/systemtap/xyz.ko": No such file or directory". Create this directory or a link to another directory with this name before copying your modules there)


 * Note that you may need to update file permissions to make staprun work

Systemtap Scripts compilation

 * Scripts compilation is performed through the command below:

export SYSTEMTAP_RUNTIME=SYSTEMTAPCFILEPATH export SYSTEMTAP_TAPSET=FULLTAPSETPATH ./stap -gv -a arm -B CROSS_COMPILE=arm-none-linux-gnueabi- -r FULLKERNELPATH -m SYSTEMTAPMODULENAME SCRIPTNAME
 * Where:


 * FULLKERNELPATH is the Kernel full path


 * FULLTAPSETPATH is the reference tapset path. It shall not be necessary if you did "make install" and run from configured prefix. It is necessary if tool is at another location


 * SYSTEMTAPCFILEPATH is the path to systemtap runtime directory. Same comment as above


 * SYSTEMTAPMODULENAME is the name of the output module to be generated


 * SCRIPTNAME is the script file (stp file)


 * On recent SW, typically a command example could be (from dir /systemtap-1.7-ubuntu/bin):

export SYSTEMTAP_RUNTIME=/home/xxx/systemtap-1.7-macharm/share/systemtap/runtime export SYSTEMTAP_TAPSET=/home/xxx/systemtap-1.7-macharm/share/systemtap/tapset ./stap -gv -a arm -B CROSS_COMPILE=arm-none-linux-gnueabi- -r custom_path/mydroid/kernel/android-2.6.29 -m iotop .../iotop.stp


 * When completed, the command generates the *.ko file in the current directory (iotop.stp comes from http://sourceware.org/systemtap/examples/io/iotop.stp)


 * Additional compilation debug information can be accessed by adding several “v” in the compilation command (-gvvv for eg)


 * As explained earlier, the script file is translated into a C file prior to compilation. All files generated during these steps can be accessed in a temporary directory (location given during the compilation phase), providing the –k option is applied (-gvk considering the example above). If not working, use "-p 3" option

Systemtap Scripts Execution

 * See above section Systemtap installation on the target

Mounting RAM file system

 * Although the tool by default generates the traces on the open terminal, these traces can be redirected to the filesystem (-o option when invoking staprun).


 * However, experience has shown that these accesses to the filesystem could interfere with the assessment of the performance indicators gravitating around the filesystem (MMC card access for eg if filesystem is mounted onto an SD card).


 * Therefore, to alleviate the interference caused by the tool accesses to the filesystem, it is suggested to mount a filesystem onto the RAM as explained below. When redirected to the filesystem mounted onto the RAM, systemtap traces output will not interfere with the typical MMC activity running on the platform.


 * Quick instructions :


 * Create a mount point, e.g. ‘mkdir /mnt/tmp’
 * Mount the RAM disk: ‘mount -t tmpfs nodev /mnt/tmp’
 * Optionally, you can add ‘-o size=32m’ to specify the maximum size reserved for this ‘disk’
 * Run systemtap scripts typing: staprun –o /mnt/tmp/output-trace.txt …

Time measurements

 * By default, today on ARM architecture, the only references to time measurement are the gettimeofday_s, gettimeofday_ms functions.


 * Indeed, the default implementation of get_cycle function can not report any cycle counter value.


 * In order to read the 32k clock value on OMAP3430/3630/4430 platforms, a function has been implemented and is defined in the reference script file omap_timestamp.stp: get_32k


 * This function can be called in any other functions/probes to provide a time resolution of ~ 30µs (1/32768 s)


 * For higher resolution time measurements a general purpose timer can be used. The Systemtap OMAP Profiler tapset provides an example on how this can be done.