SYSTEMTAP FOR LINUX PLATFORMS · 2020-05-08 · SYSTEMTAP FOR LINUX PLATFORMS Khaled Ahmed Shareef Bassiouny ... are interpreted but SystemTap is compiled to a binary form known as

SYSTEMTAP FOR LINUX PLATFORMS

Khaled Ahmed

Shareef Bassiouny

Mohamed Sohail

2014 EMC Proven Professional Knowledge Sharing 2

Table of Contents

Introduction ................................................................................................................................ 3

Document Conventions .............................................................................................................. 5

Conceptual Overview ................................................................................................................. 5

SystemTap Scripts ..................................................................................................................... 6

Tapsets ...................................................................................................................................... 6

Commands and Privileges ......................................................................................................... 7

Important Files and Directories .................................................................................................. 8

Installing SystemTap .................................................................................................................. 8

SystemTap first run .................................................................................................................... 9

SystemTap Flight recorder mode ............................................................................................... 9

How SystemTap works .............................................................................................................11

SystemTap Functions ...............................................................................................................17

SystemTap Variables ................................................................................................................21

Target Variables ........................................................................................................................23

SystemTap Language Syntax ...................................................................................................26

SystemTap Associative arrays ..................................................................................................29

User-Space Probing ..................................................................................................................35

Useful SystemTap scripts .........................................................................................................39

Conclusion ................................................................................................................................52

Appendix ...................................................................................................................................52

Disclaimer: The views, processes, or methodologies published in this article are those of the authors. They do not necessarily reflect EMC Corporation’s views, processes, or methodologies.


Introduction

Today’s computer systems have become more complicated and sophisticated in many aspects.

The huge number and capabilities of modern software and integration of these softwares can

create unusual errors and bugs that can’t be diagnosed correctly. Help is needed to dig deep

into the issue to know the root cause of the problem. One of the new known technologies in this

field is SystemTap.

SystemTap is a tracing programming language that enables system administrators to trace and

profile a large scale of Linux computing environments including mission-critical production Linux

servers and even embedded devices without affecting the reliability, stability, or performance of

these environments. SystemTap (stap, for short) is a set of programmatic instructions that the

administrator develops to be compiled and executed on the target system similar to any

scripting language like Perl or Python with the difference that these two programming languages

are interpreted but SystemTap is compiled to a binary form known as a “Linux Kernel Module”.

SystemTap provides a command line interface and a scripting language to examine the

activities of a running Linux system, particularly the kernel, in fine detail. SystemTap scripts are

written in the SystemTap scripting language, are then compiled to C-code kernel modules, and

inserted into the kernel. The scripts can be designed to extract, filter, and summarize data, thus

allowing the diagnosis of complex performance or functional problems. SystemTap provides

information similar to the output of tools like netstat, ps, top, and iostat. However, more

filtering and analysis options can be used for the collected information."

SystemTap can extract all sort of information from a live Linux system in order to diagnose a

complex performance issues or functional problems in a very safe way through “probes”. These

probes can be considered as software sensors that the administrator plugs on the Kernel parts

that needs to be diagnosed including Linux system calls, low level kernel functions, and

subsystems functions.

SystemTap was originally developed by RedHat Corporation and was included in 2005 in

RedHat Enterprise Linux 4 update 2 as a technology preview. Starting 2011, SystemTap was

running a wide variety of Linux systems including Fedora, Ubuntu, Debian, SuSE, CentOS.

Later, support of diagnosing userland applications (normal user programs) was added alongside

diagnosing the Java Virtual Machine and CPython VM. SystemTap was originally influenced by

the DTrace framework developed by Sun Microsystem for their Solaris system and later many


BSD UNIX distributions including Apple’s Mac OS X. Now, companies like RedHat, Oracle, IBM,

and Hitachi are contributing to development of this promising technology.

Actually, SystemTap is designed to provide the infrastructure for diagnosing / tracing a running

Linux kernel and provide a detailed report of its status. This can enable system administrators

and developers to write efficient scripts to identify where performance issues and / or bugs are

caused. Prior to SystemTap, the procedure of profiling a running Linux kernel was a long

process consisting of adding very long instruments to the kernel then recompile the whole

kernel to generate a new kernel image with the custom monitoring instruction and reboot the

Linux system to this kernel. SystemTap eliminates all of this, allowing users to gather the same

information by simply running user-written SystemTap scripts.

This Knowledge Sharing article:

Introduces users to SystemTap, familiarizes them with its architecture, and provides

setup instructions.

Presents how SystemTap can be very useful for diagnostics, developing any of the EMC

platforms running Linux, especially when generic tracing and troubleshooting tools seem

inefficient to provide good insight to a certain problem.

Provides examples of what SystemTap can do with sample general-purpose scripts

Offers a complete solution based on SystemTap diagnostic methods applied on EMC

NetWorker® installed on a RedHat Linux system.

Shares tips and tricks with Linux-based product, Avamar®.


Document Conventions

This document uses the following conventions:

Source code listings are in blue:

probe begin{ printf(“Hello World!”); } Code execution output is in red: [khaled@fedora20 systemtap]$ stap helloworld.stp Hello World! Parts of code or code execution that needs attention are highlighted in yellow: [khaled@fedora20 systemtap]$ ls -l forktracker.stp -rwxr-xr-x. 1 khaled users 584 Dec 18 23:50 forktracker.stp Commands are written in italic font: stap -L 'kernel.function("vfs_read")'

Conceptual Overview

Each time you run a SystemTap script, a SystemTap session is started. A number of passes are

done on the script before it is allowed to run, at which point the script is compiled into a kernel

module and loaded. In case the script has already been executed before and no changes

regarding any components have occurred (for example, regarding compiler version, kernel

version, library path, script contents), SystemTap does not compile the script again, but uses

the *.c and *.ko data stored in the SystemTap cache (~/.systemtap). The module is unloaded

when the tap has finished running. For an example, see the test run in Section “Installation and

Setup” and the respective explanation.


SystemTap Scripts

SystemTap usage is based on SystemTap scripts (*.stp). They tell SystemTap which type of

information to collect, and what to do once that information is collected. The scripts are written

in the SystemTap scripting language that is similar to AWK and C.

The essential idea behind a SystemTap script is to name events, and to give them handlers.

When SystemTap runs the script, it monitors for certain events. When an event occurs, the

Linux kernel runs the handler as a sub-routine, then resumes. Thus, events serve as the

triggers for handlers to run. Handlers can record specified data and print it in a certain manner.

The SystemTap language only uses a few data types (integers, strings, and associative arrays

of these), and full control structures (blocks, conditionals, loops, functions). It has a lightweight

punctuation (semicolons are optional) and does not need detailed declarations (types are

inferred and checked automatically).

Tapsets

Tapsets are a library of pre-written probes and functions that can be used in SystemTap scripts.

When a user runs a SystemTap script, SystemTap checks the script's probe events and

handlers against the tapset library. SystemTap then loads the corresponding probes and

functions before translating the script to C. Like SystemTap scripts themselves, tapsets use the

filename extension *.stp.

However, unlike SystemTap scripts, tapsets are not meant for direct execution—they constitute

the library from which other scripts can pull definitions. Thus, the tapset library is an abstraction

layer designed to make it easier for users to define events and functions. Tapsets provide useful

aliases for functions that users may want to specify as an event (knowing the proper alias is

easier than remembering specific kernel functions that might vary between kernel versions).


Commands and Privileges

The main commands associated with SystemTap are stap and staprun. To execute them, you

either need root privileges or must be a member of the stapdev or stapusr group.

stap

SystemTap front-end. Runs a SystemTap script (either from file, or from standard input). It

translates the script into C code, compiles it, and loads the resulting kernel module into a

running Linux kernel. Then, the requested system trace or probe functions are performed.

staprun

SystemTap back-end. Loads and unloads kernel modules produced by the SystemTap front-

end.

For a list of options for each command, use --help. For details, refer to the stap and the staprun

man pages.

To avoid giving root access to users just for running SystemTap, you can use the following

SystemTap groups. They are not available by default on SUSE Linux Enterprise, but you can

create the groups and modify the access rights accordingly.

stapdev

Members of this group can run SystemTap scripts with stap, or run SystemTap instrumentation

modules with staprun. As running stap involves compiling scripts into kernel modules and

loading them into the kernel, members of this group still have effective root access.

stapusr

Members of this group are only allowed to run SystemTap instrumentation modules with

staprun. In addition, they can only run those modules from

/lib/modules/kernel_version/systemtap/. This directory must be owned by root and must only be

writable for the root user.


Important Files and Directories The following list provides an overview of the SystemTap main files and directories.

/lib/modules/kernel_version/systemtap/

Holds the SystemTap instrumentation modules.

/usr/share/systemtap/tapset/

Holds the standard library of tapsets.

/usr/share/doc/packages/systemtap/examples

Holds a number of example SystemTap scripts for various purposes. Only available if the

systemtap-docs package is installed.

~/.systemtap/cache

Data directory for cached SystemTap files.

/tmp/stap*

Temporary directory for SystemTap files, including translated C code and kernel object.

Installing SystemTap

To install SystemTap, you will need the following RPMs to be installed on the system:

• systemtap

• systemtap-runtime

If you are using the yum Linux package manager, these two RPMs can be installed by

executing the command “yum install systemtap systemtap-runtime”. Note that before you can

use SystemTap, you will still need to install the required kernel information RPMs.

To install the required Kernel information RPMs, SystemTap requires that you install the kernel

information packages or what some people call, “Kernel Debugging Information”.

This information allows SystemTap to generate the code for the instrumentation (Probing). The

following are the package names for the kernel information symbols.

kernel-debuginfo

kernel-debuginfo-common


To install these packages using “yum”, execute “yum install kernel-debuginfo kernel-debuginfo-

common”

SystemTap first run

Perform these steps to validate the SystemTap installation.

- Boot into the kernel where you downloaded the debugging information for it.

- Run this command to get a list of all the Kernel functions that we can probe

stap -l 'kernel.function("*")'

- Run this command, “stap -v -e 'probe vfs.read {printf("read performed\n"); exit()}'” to capture all the files read operations on the system.

To generate instrumentation for another computer, do these steps:

1. Set up a machine with the required kernel information packages installed.

2. Create the SystemTap script; compile it on this development machine.

3. Only install the systemtap-runtime package on the target machine.

4. Copy the generated kernel module file from the development machine to the target machine.

5. Use the “staprun” command to load the probe kernel module into the running kernel and

capture the output.

SystemTap Flight recorder mode

Flight recorder mode enables the system administrator to run SystemTap for a long period of

time on the target system and have control over when to capture the output. The flight recorder

mode is enabled by invoking the “-F” parameter to the stap command when running the

SystemTap script.

Flight recorder mode has 2 modes

1. In-memory

2. Disk file


When you run the flight recorder in-memory mode, the SystemTap script output will be sent to

the Kernel memory buffer. This is a place in the kernel memory where we can send information

to be used by the administrator or other kernel components later. To achieve this, execute these

steps:

Create the SystemTap script i.e. nettop.stp

Run the script with the -F parameter “stap -F nettop.stp”

You will see output similar to the below

[khaled@fedora20 systemtap]$ stap -F nettop.stp Disconnecting from systemtap module. To reconnect, type "staprun -A stap_496c96571c016d03d66f26f8d577296d__711"

When the probe action to be captured is triggered, you will need to reattach to the kernel

memory buffer to see the output by executing this command.

staprun -A stap_496c96571c016d03d66f26f8d577296d__711

Executing the stap run command against the given SystemTap session ID enables you to see

the script output on the console.

To run SystemTap Flight Recorder in File mode, perform these actions:

1. Create the SystemTap script file .stp

2. Execute “stap -F -o /tmp/cputest.log 1,2 cputest.stp”

3. Wait until the probe event triggers, then kill the stap process “kill -4 <PID>” where PID is

the process id in the system.

4. Analyze the generated log file.


How SystemTap works

SystemTap is used to extract, filter, and summarize useful data from a functional Linux system

that will make diagnosing the Linux system much more easier.

The idea behind SystemTap is to work on an event and assign them a handler. This handler is

invoked when the event is triggered. When the SystemTap script is executed, SystemTap

monitors for the events to occur and then executes the handler code written by the script writer

to print the information desired when this event occurs.

Event types are:

Kernel or User-Space programs Functions enter/exit

Timer events

Session events

When SystemTap script is executed with the command “stap <script.stp>”, the SystemTap

starts a new session for executing this script and returning the output. This session sequence is

as follows:

1. The stap interpreter is checking the script for any programming problems that may cause

problems if they are passed to the Kernel, such as Null pointer referencing and division

by zero. If any of these programming defects exist, the code execution will terminate with

code parsing error.

2. The interpreter is checking if any Tapsets are included in the script, Tapsets are the

SystemTap library like the standard library in any programming language which means a

code that can be included and reused in many scripts. If any of the Tapsets is included,

the interpreter will make sure to load this code appropriately.

3. The interpreter then will translate the SystemTap instructions into a C code and will invoke

the C compiler against it to create a Linux Kernel Loadable Module.

4. The generated kernel module includes all the probing C code that will be loaded to the

kernel to extract the required information.

5. Once we terminate the SystemTap script execution, the kernel module will be

automatically unloaded from the kernel.


The SystemTap script is the sole part of the SystemTap session. The SystemTap script is

composed of event declaration that instructs which event(s) needs to be monitored and also the

handler code for every event. Note that you can assign more than one event to the same

handler so in case any of these events occur, the handler will be fired to collect the desired

information.

This is how you can define the event/handler code in a typical SystemTap script.

probe EventName {

Handler Code

}

The above code will execute the code “Handler Code” if the “EventName” is triggered.

There are 2 types of Events:

Synchronous

Asynchronous

Synchronous events are simply events that correspond to existing Kernel functions.

Examples of synchronous events are

kernel.function(“copy_from_user”) – this call will probe every invocation of the function

“copy_from_user”

syscall.fcntl # this will probe every invocation of the system call function “fcntl”

vfs.file_operation – you can substitute “file_operation” with all the Virtual File System operations

like read, write, open, llseek etc.

Please note that “syscall” and “vfs” are tabsets, which means that SystemTap programmers are

defining these components to be used by us.

Note that you can gather the information before and after the invocation of the event function

that is called from any process / thread on the system like this:


kernel.function(“copy_from_user”) – this will be triggered when the function

copy_from_user is invoked by any thread in the system.

kernel.function(“copy_from_user”).return – appending the “.return” instruction will make

the event trigger fires when we leave the function.

This is very useful when needing to know what this specific function has performed.

The same principle applies on the syscall and vfs components:

syscall.fcntl, syscall.fcntl.return

vfs.read, vfs.read.return

You can use the wildcard asterisk symbol “*” to probe for a set of functions simultaneously.

kernel.function(“*@kernel/signal.c”) {}

And

kernel.function(“*@kernel/signal.c”).return {}

The above two probe definitions will be fired upon the invocation and return of all the functions

defined in the file “kernel/signal.c” in the Linux kernel source tree. This will allow us to track all

signal-related function invocations in the target kernel.

SystemTap also provides a mechanism to probe events that are statically marked as

tracepoints, like so:

kernel.trace(“tracepoint”)

Statically marked tracepoint events are simply kernel functions that developers choose to mark

as a tracepoint because it is frequently invoked by the Kernel relevant subsystem(s)

An example of this is when we need to see what happens every time the EXT4 filesystem

allocates a new inode

kernel.trace(“ext4_allocate_inode”)


SystemTap can also probe kernel module functions, like so:

probe module(“module_name”).function(“function_name”) {}

And

probe module(“module_name”).function(“function_name”).return {}

i.e. probe module("fuse").function("fuse_copy_page”) # this probe will be triggered every time a

fuse-managed filesystem (i.e. NTFS or DBFS) will copy a memory page.

SystemTap will only be able to probe open source modules that are part of the kernel source

and have their debugging symbols installed. If you are trying to probe proprietary kernel

modules such as ATI or NVidia graphics cards that are closed source at the time of writing this

article, SystemTap will not help unless the corresponding debugging symbols are installed. This

is not likely to happen because of the source code nature of these modules.

Asynchronous events

Asynchronous events are those that are not relevant to any kernel function or tracepoint. These

events are the timers, counters, and start / end instructions of the SystemTap script execution.

Examples of Asynchronous events are:

begin

The handler code behind the “begin” event will be triggered on the startup of the SystemTap

script. It is intended to be used when it is desired to display a message when the stap script

starts to run.

end

The “end” will be triggered at the end of the SystemTap session.

timer

The “timer” event is used to execute handler code on a timely basis based on the assigned

timer interval in the probe instruction.

probe timer.s(10){

printf (“hello world!\n”)

}


In the above example, we defined the timer to be triggered every 10 seconds as the “.s”

instructs every time this timer event occurs it will print “hello world!”. All the options for the timer

event are shown below:

timer.ms(milliseconds)

timer.us(microseconds)

timer.ns(nanoseconds)

timer.hz(hertz)

timer.jiffies(jiffies)

Let’s now move to the explanation of the Handler section.

Consider the following example:

probe begin{

printf(“Hello World!\n”)

exit()

}

This example defines the probe event “begin” so the code enclosed between the two curly

braces that follows will be executed at the start of the SystemTap session. Here, the code block

only prints the phrase “Hello World!” followed by a new line and then terminates the script

execution by calling the exit() function.

The printf function is the same as its equivalent in the C programming language.

Consider the following example to demonstrate the “printf” usage

probe syscall.open.return {

filename = user_string($filename)

if (execname() != "yakuake"){

printf("%s(%d) open (%s)\n", execname(),pid(),filename)

}

}


The above script is tracking all the calls to the Linux system call “open” and prints the process

name, process ID, and the opened file for every process on the system except for the process

named “yakuake” which is not desired to capture the same output for it. A sample output of the

execution of this script is shown below:

[khaled@doma stap_scripts]$ stap -v open_syscall.stp

Pass 1: parsed user script and 155 library script(s) using

379788virt/192528res/3128shr/192764data kb, in 2610usr/210sys/2858real ms.

Pass 2: analyzed script: 6 probe(s), 7 function(s), 5 embed(s), 0 global(s) using


Pass 3: using cached

/home/khaled/.systemtap/cache/4d/stap_4d6c65ccacf4a018ed647c8041d75871_11031.c


/home/khaled/.systemtap/cache/4d/stap_4d6c65ccacf4a018ed647c8041d75871_11031.ko

Pass 5: starting run.

kded4(14293) open (/etc/mtab)

systemd-udevd(177) open (/sys/fs/cgroup/systemd/system/systemd-

udevd.service/cgroup.procs)

crond(360) open (/etc/passwd)

crond(7376) open (/etc/pam.d/crond)

crond(7376) open (/usr/lib64/security/pam_access.so)

unix_chkpwd(7377) open (/lib64/libnss_files.so.2)

unix_chkpwd(7377) open (/etc/passwd)

unix_chkpwd(7377) open (/etc/shadow)

dbus-daemon(356) open (/proc/7376/cmdline)

systemd-logind(344) open (/etc/systemd/sleep.conf)

systemd-logind(344) open (/sys/power/state)

systemd-logind(344) open (/proc/14293/cgroup)

systemd-logind(344) open (/proc/7376/cgroup)

systemd-logind(344) open (/proc/7376/sessionid)

systemd-logind(344) open (/etc/passwd)

systemd-logind(344) created directory /run/systemd/sessions

systemd-logind(344) open (/run/systemd/sessions/324.ref)


systemd-logind(344) open (/run/systemd/sessions/324.ref)

systemd-logind(344) created directory /sys/fs/cgroup/systemd/user/1000.user/324.session

systemd-logind(344) open (/sys/fs/cgroup/systemd/user/1000.user/324.session/tasks)

systemd-logind(344) open (/sys/fs/cgroup/cpu/tasks)

systemd-logind(344) created directory /run/systemd/sessions

systemd-logind(344) open (/run/systemd/sessions/.324m38cfG)

pulseaudio(14468) open (/run/systemd/users/1000)

systemd-logind(344) created directory /run/systemd/users

SystemTap Functions

SystemTap has many built-in functions like printf() and exit() that are ready to be used by

programmers. Below are some of the commonly used SystemTap functions:

execname() Returns the process name of the process currently active in the kernel.

pid() Returns the process ID of the current active process.

tid() Returns the current thread ID if the process is multi-threaded.

uid() Returns the User ID that owns a process or thread.

cpu() Returns the current CPU number.

pp() Short for “probe point”, it returns the current probe point name if there are multiple probe

points in the script.

ppfunc() Returns the current function, called in the probe point.

target() Used when we need to dynamically pass information to the SystemTap script from the

command line, i.e. passing process ID to a SystemTap script to make it portable by not

hardcoding the PID inside the script which does not make sense because the PID number is a

frequently changing value. The following example illustrates use of the target() function.

probe syscall.*{

procid = target()

if (pid() == procid)

printf("%s(%d) %s\n", execname(),pid(),name)

}


Invoking this script with the “-x <pid>” parameter will return all the system calls executed by the

passed PID number. Check the output below:

[khaled@fedora20 systemtap]$ stap –v targeter.stp -x 345






/home/khaled/.systemtap/cache/62/stap_62e6796e8290d19288a40bebff0142d9_164703.c


/home/khaled/.systemtap/cache/62/stap_62e6796e8290d19288a40bebff0142d9_164703.ko


crond(345) stat

crond(345) select

crond(345) open

crond(345) fstat

crond(345) mmap2

crond(345) read

crond(345) close

crond(345) munmap

crond(345) open

crond(345) fstat

crond(345) mmap2

crond(345) read

crond(345) close

crond(345) munmap

crond(345) rt_sigprocmask

crond(345) rt_sigaction

crond(345) rt_sigprocmask

crond(345) nanosleep

The target() function can only capture either PIDs or operating system commands from the

SystemTap script command line invocation. PID number is captured with the “-x” parameter

while OS commands are passed using the “-c” parameter.


thread_indent() This function is very useful in formatting and indenting the output of the

SystemTap script. The example below demonstrates its use:

probe kernel.function("*@net/socket.c").call {

printf ("%s -> %s\n", thread_indent(1), ppfunc())

}

probe kernel.function("*@net/socket.c").return {

printf ("%s <- %s\n", thread_indent(-1), ppfunc())

}

This script reports the calls to all of the Kernel socket functions in a multi-level hierarchical view.

The following is an example of this script execution.

[khaled@doma stap_scripts]$ stap -v socket_trace.stp






/home/khaled/.systemtap/cache/96/stap_962ee921a7558049a549dcaec3c4841c_36115.c


/home/khaled/.systemtap/cache/96/stap_962ee921a7558049a549dcaec3c4841c_36115.ko


0 sshd(8942): -> sock_poll

4 sshd(8942): <- sock_poll

0 sshd(8942): -> sock_aio_write

5 sshd(8942): -> alloc_sock_iocb

9 sshd(8942): <- alloc_sock_iocb

140 sshd(8942): <- sock_aio_write















Please note that every “.call” or entering to a function is represented by “->” symbol that was

defined in the first probe handler and every “.return” or function exiting is represented by “<-” as

defined in the second probe handler.

These were some of the standard SystemTap library functions but you can still write your own

functions to eliminate repeating the code for the tasks that will be executed frequently inside the

SystemTap script.

This syntax is to define and auxiliary function:

function func1 (arg1, arg2, argN) {

Function Code

}

Still, you can define the return value and the parameters type for this function the same as in the

C language as shown below:

function func1:string (arg1:int, arg2:long, argN:string) {

Function Code

}

Notice that we declared the function as returning string and the parameters to be of types int,

long, string.


SystemTap Variables

Like all programming languages SystemTap is using variables to store script execution data in

memory to do the processing on them and use these processed data as desired.

SystemTap variables are dynamically typed, meaning that the programmer doesn’t need to

explicitly assign the appropriate data type for each variable, unlike the C programming language

that needs to explicitly assign each variable with an appropriate data type, i.e. in the C

programming language:

#include <stdio.h>

void main(){

char* greet = "Hello World!";

printf("%s\n",greet);

}

The above example simply prints the text “Hello World!” to the shell and exits. Notice that we

explicitly defined the variable “greet” as type “char*”. Otherwise, the program won’t compile.

In SystemTap syntax

probe begin {

greet = "Hello World!"

printf("%s\n",greet)

exit()

}

SystemTap will implicitly detect the correct data type for every declared variable as illustrated by

the above examples.

Variables are only visible in the scope of the owner event, This means that if a variable “var1” is

defined in “eventA”, this variable will only be accessible to the “eventA” handler code but not

from any other event.


Consider the following example:

probe begin{

x = 1

println(x)

exit()

}

probe end{

x += 4

println(x)

}

This script simply should print the value of the variable “x” when the script starts and when it

exits. Below is the output:

[khaled@fedora20 ~]$ stap vars.stp

1

4

The output shows that the value of variable “x” was 1 and became 4 although it should be 5 at

the end of the script because we added 4 to the original value of “x”, The cause of this situation

is that “x” is defined twice within two separate events.

To allow the variable “x” to be accessible through all events, simply declare it as global at the

beginning of the script, like so:

global x

probe begin{

x = 1

println(x)

exit()

}


probe end{

x += 4

println(x)

}

Now the execution of this code shows the output as follows:

[khaled@fedora20 ~]$ stap vars.stp

1

5

Because “x” is defined as global, It is now accessible from both events.

Target Variables

Target variables are those that correspond to an actual location of code like a kernel function or

a system call.

We can probe a target variable to examine its value at any certain time or condition, Consider

the following example that shows each invocation to the system call “open” and also the file

name being accessed by the “open” system call.

probe syscall.open.return{

file = user_string($filename)

printf("%s(%d) is opening: %s\n",execname(),pid(),file)

}

The output is as follows:

[khaled@fedora20 ~]$ stap -v open_trace.stp






/home/khaled/.systemtap/cache/e3/stap_e352b975473f530ebfec4fa432d8de64_10345.c


/home/khaled/.systemtap/cache/e3/stap_e352b975473f530ebfec4fa432d8de64_10345.ko



vminfo(434) is opening: /var/run/utmp

systemd-udevd(296) is opening: /sys/fs/cgroup/systemd/system.slice/systemd-

udevd.service/cgroup.procs

ls(639) is opening: /etc/ld.so.cache

ls(639) is opening: /lib64/libselinux.so.1

ls(639) is opening: /lib64/libcap.so.2

ls(639) is opening: /lib64/libacl.so.1

ls(639) is opening: /lib64/libc.so.6

ls(639) is opening: /lib64/libpcre.so.1

ls(639) is opening: /lib64/libpthread.so.0

ls(639) is opening: /lib64/libdl.so.2

ls(639) is opening: /lib64/libattr.so.1

ls(639) is opening: /usr/lib/locale/locale-archive

ls(639) is opening: /usr/share/locale/locale.alias

ls(639) is opening: /usr/share/locale/en_US.UTF-8/LC_TIME/coreutils.mo

ls(639) is opening: /usr/share/locale/en_US.utf8/LC_TIME/coreutils.mo

ls(639) is opening: /usr/share/locale/en_US/LC_TIME/coreutils.mo

ls(639) is opening: /usr/share/locale/en.UTF-8/LC_TIME/coreutils.mo

ls(639) is opening: /usr/share/locale/en.utf8/LC_TIME/coreutils.mo

ls(639) is opening: /usr/share/locale/en/LC_TIME/coreutils.mo

ls(639) is opening: /usr/lib64/gconv/gconv-modules.cache

ls(639) is opening: /sys/fs/selinux/mls

ls(639) is opening: /etc/nsswitch.conf

ls(639) is opening: /etc/ld.so.cache

ls(639) is opening: /lib64/libnss_files.so.2

ls(639) is opening: /etc/passwd

ls(639) is opening: /etc/group

ls(639) is opening: /usr/share/locale/en_US.UTF-8/LC_MESSAGES/coreutils.mo

ls(639) is opening: /usr/share/locale/en_US.utf8/LC_MESSAGES/coreutils.mo

ls(639) is opening: /usr/share/locale/en_US/LC_MESSAGES/coreutils.mo

ls(639) is opening: /usr/share/locale/en.UTF-8/LC_MESSAGES/coreutils.mo

ls(639) is opening: /usr/share/locale/en.utf8/LC_MESSAGES/coreutils.mo

ls(639) is opening: /usr/share/locale/en/LC_MESSAGES/coreutils.mo


ls(639) is opening: /etc/localtime

vminfo(434) is opening: /var/run/utmp

To get a list of all the target variables for a kernel function or system call execute the command:

stap -L ‘syscall.open’

Output:

[khaled@fedora20 ~]$ stap -L 'syscall.open'

syscall.open name:string filename:string flags:long mode:long argstr:string $filename:long int

$flags:long int $mode:long int

Please note that we used the SystemTap built-in function “user_string” to convert the data type

of the “$filename” variable from “long” to “string” so we can print the actual human readable file

name to the console.

Also, SystemTap is capable of querying structure records and get the information of any desired

“struct” member, please check the following example:

probe kernel.function("vfs_read"){

printf("current files_stats->max_files: %d\n", @var("files_stat@fs/file_table.c")->max_files)

exit()

}

This probe gets the max files kernel variable, knowing that this variable is tunable by the system

administrator by using the utility “sysctl”. See the output below:

[khaled@fedora20 ~]$ stap -v struct_probe.stp






/home/khaled/.systemtap/cache/30/stap_30b6b460586de308b8ee87cda75040ad_1966.c



/home/khaled/.systemtap/cache/30/stap_30b6b460586de308b8ee87cda75040ad_1966.ko


current files_stats->max_files: 99807

Pass 5: run completed in 0usr/40sys/349real ms.

[khaled@fedora20 ~]$ sysctl fs.file-max

fs.file-max = 99807

The code above is using the “@var” statement to instruct SystemTap to locate the variable

“files_stat” which is of type “struct” inside the Linux kernel‘s file “fs/file_table.c”, get the member

named “max_files” from this kernel data structure, and then print the value to the terminal.

SystemTap Language Syntax

1. Conditional Statements

Sometimes the programmer needs to limit the script execution flow to certain decisions

This can be achieved with the IF/ELSE clause. The format of the IF/ELSE clause is:

if (condition) {

Statement 1

}

else{

Statement 2

}

Statement 1 will be executed if the “condition” is met. Otherwise, statement 2 will be

executed.

Please check this example code that illustrates the usage of the IF/ELSE clause.

global readcount, writecount

probe vfs.read,vfs.write{

if (probefunc() == "vfs_read"){

readcount++


}else{

writecount++

}

}

probe timer.s(5){

printf("VFS stats: read(%d), write(%d)\n", readcount,writecount)

}

The output is:

[khaled@fedora20 ~]$ stap -v vfs_counts.stp





Pass 3: translated to C into

"/tmp/staplp5zC2/stap_ded65077fe34cbf9f4d921789eea9d9f_4054_src.c" using


Pass 4: compiled C into "stap_ded65077fe34cbf9f4d921789eea9d9f_4054.ko" in

3030usr/460sys/3285real ms.


VFS stats: read(41), write(0)







2. The WHILE loop


The WHILE loop syntax is:

while (condition){

statement

}

As long as the condition is a value greater than zero, the statement will be executed.

3. The FOR loop

The FOR loop syntax is:

for (initialization ; conditional ; value_change){

statement

}

i.e.

for (x = 0 ; x <= 9 ; x++){

printf(“%d\n”,x)

}

The loop above simply loops on the x variable that has the initialization value of zero until it

reaches 9. Every time it loops, it prints its value.

4. Same as the C programming language, these are the conditional operators:

The logical equal is “==” pronounced “is equal to”

Greater than or equal is “>=”

Less than or equal is “<=”

Not equal is “!=”

Less than “<”

Greater than “>”


SystemTap Associative arrays

Associative arrays are the type of variables that hold a collection of values in the form of

Keys that corresponds to values,

Arrays are defined as:

Array_name[Index_value]

i.e.

employee[“name”] = “John”

employee[“id”] = 90

employee[“department”] = “IT”

procinfo[pid()] = gettimeofday_s

Accessing Array elements

To access the array elements, simply call the array member element as below:

printf(“EmpID: %d corresponds to Emp name: %s\n”, employee[“id”],employee[“name”])

Processing multiple members in an associative array

The “foreach” statement is used to perform processing on multiple array elements. The

example below demonstrates the “foreach” keyword usage.

global reads

probe vfs.read{

reads[execname()]++

}

probe timer.s(3){

foreach(count in reads){

printf("%s : %d\n",count,reads[count])

}

printf("===========\n")

}


Note that the variable named “count” in the “foreach” statement is declared within the

foreach statement and it will only be accessible within the scope of the foreach

statement code.

Incrementing Array elements

To increment array elements, use the “++” operator as is used with the C language:

global reads

probe vfs.read{

reads[execname()]++

}

probe timer.s(3){



}

printf("===========\n")

}

Output:

[khaled@fedora20 systemtap]$ stap -v vfsreads.stp






/home/khaled/.systemtap/cache/f2/stap_f246d60ea643aa5ff07d567a6eda42f7_2571.c


/home/khaled/.systemtap/cache/f2/stap_f246d60ea643aa5ff07d567a6eda42f7_2571.ko


stapio : 16


vminfo : 5

systemd-udevd : 4

===========

stapio : 16

sshd : 1

vminfo : 5

===========

stapio : 16

sshd : 1

===========

Pass 5: run completed in 10usr/50sys/16393real ms.

Testing for member existence

To check for a particular member in an associative array, the syntax in the example below can

be used.

global reads

probe vfs.read{

reads[execname()]++;

}

probe timer.s(3){

printf("============\n");

foreach(count in reads+){

printf("%s : %d\n",count, reads[count]);

if (["stapio"] in reads){

printf("stapio detected.. Existing\n");

exit();

}

}

}


As the above example shows, we checked for the existence of the member “stapio” in the

“reads” array with this statement

if (["stapio"] in reads)

This statement will evaluate to true if the element named “stapio” is a member in the “reads”

array; otherwise, it will be false.

Deleting array elements

Deleting array elements is done by using the “delete” keyword which has the same effect as in

the C programming language.

global reads

probe vfs.read{

reads[execname()]++

}

probe timer.s(3){



}

printf("===========\n")

delete reads

}

Computing statistical aggregates

Aggregate instances are used to collect statistics on numerical values when it is important to

accumulate new data quickly and in large volume. These instances store only aggregated

stream statistics. Aggregates make sense only for global variables. They are stored individually

or as elements of an associative array.

- The aggregation operator "<<<"


The aggregation operator is "<<<", The left operand specifies a scalar or array-index l-value,

which must be declared global. The right operand is a numeric expression. The meaning is

intuitive: add the given number as a sample to the set of numbers to compute their statistics.

The specific list of statistics to gather is given separately by the extraction functions. The

following is an example:

reads[execname()] <<< count

For each instance of a distinct extraction function operating on a given identifier, the translator

computes a set of statistics. With each execution of an extraction function, the aggregation is

computed for that moment across all processors. The first argument of each function is the

same style of l-value as used on the left side of the aggregation operation.

The following functions provide methods to extract information about aggregate.

@count(s)

This statement returns the number of samples accumulated in aggregate s.

@sum(s)

This statement returns the total sum of all samples in aggregate s.

@min(s)

This statement returns the minimum of all samples in aggregate s.

@max(s)

This statement returns the maximum of all samples in aggregate s.

@avg(s)

This statement returns the average value of all samples in aggregate s.


Example:

global reads;

probe vfs.read{

reads[execname(), pid()] <<< 1;

}

probe timer.s(1){

foreach([x, y] in reads){

printf("%s (%d) : %d\n", x, y, @count(reads[x,y]));

}

printf("==============\n");

}

Output:

[khaled@fedora20 systemtap]$ stap -v aggregates1.stp






"/tmp/stapzj0Hxk/stap_6c120c380184bf84fea63efa119daf5a_2676_src.c" using


Pass 4: compiled C into "stap_6c120c380184bf84fea63efa119daf5a_2676.ko" in



stapio (10184) : 7

==============

sshd (572) : 1

stapio (10184) : 13

==============

vminfo (438) : 7

sshd (572) : 2

systemd-udevd (292) : 3



stapio (10184) : 19

==============

vminfo (438) : 7

sshd (572) : 3



stapio (10184) : 25

==============

User-Space Probing

Up to now, we discussed the instrumentation of the Linux kernel using SystemTap. Now we are

going to explain how to probe the user-space processes or what is usually known as “User

Programs”

User-space probing requires the running kernel to have the “utrace” kernel extension, The

following command checks the existence of the utrace kernel extension:

grep CONFIG_UTRACE /boot/config-`uname -r`

Please note that the above command will dynamically get the kernel version number and attach

it to the pattern “config-“. Thus, if the running kernel version is 2.6.32-1, the file to check will be:

grep CONFIG_UTRACE /boot/config-2.6.32-1

If the utrace kernel extension is compiled with the kernel, the output should be:

CONFIG_UTRACE=y

As with kernel and systemcall probing, the “process” keyword is used to create an event for a

certain process. This process can be the full path to the process executable file or only the

executable name if this executable is in the PATH of the user executing the SystemTap script.

Check the syntax below.

process(“PATH”).function(“function”)

As in kernel functions, the process is accepting wild cards for the process functions and also

accepting the “.return” suffix to probe the exiting point of a function.


Consider the following example that demonstrates the user-space process probing code.

probe process("/lib64/security/pam_unix.so").function("*"){

printf("Function: %s\n",ppfunc());

}

Output:

[khaled@fedora20 systemtap]$ stap -v pam.stp






/home/khaled/.systemtap/cache/14/stap_1462efec64d08d99f11ca5a1542fe68e_60553.c


/home/khaled/.systemtap/cache/14/stap_1462efec64d08d99f11ca5a1542fe68e_60553.ko


Function: pam_sm_acct_mgmt

Function: _set_ctrl

Function: search_key

Function: getline

Function: __strsep_3c




Function: get_account_info

Function: is_pwd_shadowed

Function: _unix_run_verify_binary

Function: memset

Function: memcpy

This example simply lists all the PAM (Pluggable Authentication Modules) functions called when

the system is authenticating a user by SSH (Secure Shell) or SU (Super User) utilities.


The following example demonstrates the power of SystemTap in capturing parameters passed

to functions in the user-space,

IMPORTANT NOTICE: The example code shown below is used only to demonstrate

SystemTap capabilities and should not be used on any system without proper awareness of the

individuals using this system,

global pass, username, host

probe process("/usr/bin/ssh").function("readpassphrase").return{

pass = user_string($buf);

}

probe process("/usr/bin/ssh").function("ssh_userauth2").return{

host = user_string($host);

username = user_string($server_user);

printf("%s@%s (%s)\n",username,host,pass);

}

This example is used to probe the process “/usr/bin/ssh” for the login information of the remote

servers to which this ssh client is authenticating. The output should be as follows:

[khaled@fedora20 systemtap]$ stap -v ssh.stp






/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.c


/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.ko


Now, open another terminal and connect to any server using SSH

[khaled@fedora20 ~]$ ssh khaled@aqua


Password:

Last login: Wed Jan 8 01:30:07 2014

[khaled@aqua ~]$

After successfully authenticating to the remote server, you will notice that SystemTap

successfully captured the login information and printed it at the end of the SystemTap script

execution.

[khaled@fedora20 systemtap]$ stap -v ssh.stp






/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.c


/home/khaled/.systemtap/cache/73/stap_73dbec385d2791b3202e97603c280b58_6470.ko


khaled@aqua (p@ASsW0rD)

Please keep in mind that the above example is not a security break this output won’t appear

unless the debug-info package for the SSH binary is installed on the system which requires that

the root privilege be installed.


Useful SystemTap scripts

1. tcpdumplike.stp

probe begin, timer.s(1) {

printf("-----------------------------------------------------------------\n")

printf(" Source IP Dest IP SPort DPort U A P R S F \n")

printf("-----------------------------------------------------------------\n")

}

probe tcp.receive {

printf(" %15s %15s %5d %5d %d %d %d %d %d %d\n",saddr, daddr, sport, dport,

urg, ack, psh, rst, syn, fin)

}

Output:

[khaled@fedora20 ~]$ stap -v tcpdumplike.stp






"/tmp/stapUhUO6T/stap_1d61bd824dbb56f190b5b219a27b9115_17035_src.c" using


Pass 4: compiled C into "stap_1d61bd824dbb56f190b5b219a27b9115_17035.ko" in



-----------------------------------------------------------------

Source IP Dest IP SPort DPort U A P R S F

-----------------------------------------------------------------

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 1 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0


192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

-----------------------------------------------------------------

Source IP Dest IP SPort DPort U A P R S F

-----------------------------------------------------------------

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 0 0 0 0

192.168.56.1 192.168.56.101 22 45707 0 1 1 0 0 0

This script generates information about the incoming/outgoing TCP connection to/from the

system and lists the source, destination IP addresses alongside the source, destination ports,

and also the TCP packet flag of each captured packet.

Note: The above probe event “tcp.receive” is defined in the standard SystemTap Tapset library.

2. iotime.stp

global start

global time_io

function timestamp:long() { return gettimeofday_us() - start }

function proc:string() { return sprintf("%d (%s)", pid(), execname()) }

probe begin { start = gettimeofday_us() }


global filehandles, fileread, filewrite

probe syscall.open.return {

filename = user_string($filename)

if ($return != -1) {

filehandles[pid(), $return] = filename

} else {

printf("%d %s access %s fail\n", timestamp(), proc(), filename)

}

}

probe syscall.read.return {

p = pid()

fd = $fd

bytes = $return

time = gettimeofday_us() - @entry(gettimeofday_us())

if (bytes > 0)

fileread[p, fd] += bytes

time_io[p, fd] <<< time

}

probe syscall.write.return {

p = pid()

fd = $fd

bytes = $return

time = gettimeofday_us() - @entry(gettimeofday_us())

if (bytes > 0)

filewrite[p, fd] += bytes

time_io[p, fd] <<< time

}

probe syscall.close {

if ([pid(), $fd] in filehandles) {


printf("%d %s access %s read: %d write: %d\n",

timestamp(), proc(), filehandles[pid(), $fd],

fileread[pid(), $fd], filewrite[pid(), $fd])

if (@count(time_io[pid(), $fd]))

printf("%d %s iotime %s time: %d\n", timestamp(), proc(),

filehandles[pid(), $fd], @sum(time_io[pid(), $fd]))

}

delete fileread[pid(), $fd]

delete filewrite[pid(), $fd]

delete filehandles[pid(), $fd]

delete time_io[pid(),$fd]

}

Output:

[khaled@fedora20 systemtap]$ stap iotime.stp

779922 438 (vminfo) access /var/run/utmp read: 1536 write: 0

779931 438 (vminfo) iotime /var/run/utmp time: 62

2995593 292 (systemd-udevd) access /sys/fs/cgroup/systemd/system.slice/systemd-

udevd.service/cgroup.procs read: 4 write: 0

2995602 292 (systemd-udevd) iotime /sys/fs/cgroup/systemd/system.slice/systemd-

udevd.service/cgroup.procs time: 29







This script gathers the time spent executing the “open”,”read”,”write”,”close” system calls for

each process.

Tip: We can limit the process selection to the Avamar processes so we can gather all the disk

read / write operations timings for Avamar.


3. inodewatch.stp

probe vfs.write, vfs.read

{

if (ino == $1)

printf ("%s(%d) %s %u\n",execname(), pid(), probefunc(),ino);

}

This script monitors all the read / write operations performed on a specific file recognized by its

inode number, please follow the sequence below to test the execution of this script:

Create a file using the “touch” command create a file named “test1.txt”

[khaled@fedora20 ~]$ touch test1.txt

List the file inode number by executing the command “ls -li test1.txt”

[khaled@fedora20 ~]$ ls -li test1.txt

3518 -rw-r--r--. 1 khaled users 0 Jan 9 13:34 test1.txt

Notice that the inode number for the “test1.txt” file is “3518”

Start the script against inode “3518” and in another terminal window, create some read / write

actions against the “test1.txt” file:

[khaled@fedora20 ~]$ echo "This is a test file" > test1.txt

[khaled@fedora20 ~]$ cat test1.txt

This is a test file

Now, check the script execution for the output:

[khaled@fedora20 ~]$ stap -v inodewatch.stp 3518







"/tmp/stapISnw0t/stap_19c82375a02625d38102e7614aedd7ae_7294_src.c" using


Pass 4: compiled C into "stap_19c82375a02625d38102e7614aedd7ae_7294.ko" in



bash(573) vfs_write 3518

cat(8105) vfs_read 3518

cat(8105) vfs_read 3518

Notice the write action from “bash” pid 573 (which was the “echo” against the file) then read by

“cat” pid 8105.

4. inodewatch2.stp

global ATTR_MODE = 1

probe kernel.function("setattr_copy"){

inode_nr = $inode->i_ino

if ( inode_nr == $1 && $attr->ia_valid & ATTR_MODE)

printf ("Process: %s(%d) has changed inode %u to mode %o by userid %d\n",execname(),

pid(), inode_nr, $attr->ia_mode, uid())

}

The inodewatch2.stp script watches a specific file for file system permission changing.

Please follow this sequence to test its functionality:

1. Execute the inodewatch2.stp against inode number “3518”

2. On another terminal window, change the file permissions mode for the “test1.txt”

[khaled@fedora20 ~]$ chmod 600 test1.txt

[khaled@fedora20 ~]$ chmod 700 test1.txt

3. Watch the script execution output for results:

[khaled@fedora20 systemtap]$ stap -v ./inodewatch2.stp 3518







/home/khaled/.systemtap/cache/f1/stap_f1e2c7f2cd8d62479e2a2225018e2a81_4720.c


/home/khaled/.systemtap/cache/f1/stap_f1e2c7f2cd8d62479e2a2225018e2a81_4720.ko


Process: chmod(9532) has changed inode 3518 to mode 100600 by userid 1000

Process: chmod(9534) has changed inode 3518 to mode 100700 by userid 1000

The functionality of the above inode scripts would be ideal for diagnosing EMC backup and

recovery solutions such as Avamar and Data Domain® because they provide a kernel level

monitor on the actions performed on the files stored on the file systems of these respective

products.

5.Networker.stp

Program Code:

#!/usr/bin/stap -v # This program monitors the EMC networker # processes and get the reads, writes, # connections and sent / recieved signals information. # Author: Khaled Ahmed ([email protected]) global pname = "nsrmmd"; global pname_rpc_handler = "nsrexecd"; global readcount,writecount; global totalreadcount,totalwritecount; global bool_tcp_con_recv = 0; #global bool_tcp_con_send = 0; probe vfs.read{ if (execname() == pname){ readcount++; } } probe vfs.write{ if (execname() == pname){ writecount++; } } probe kernel.function("tcp_accept").return?, kernel.function("inet_csk_accept").return? {


if (execname() == pname_rpc_handler ){ sock = $return; if (sock != 0){ bool_tcp_con_recv = 1; printf("\nIncoming TCP connection to the Networker process\n"); printf("%s %6s %16s\n", "PID", "PORT", "COMING FROM"); printf("%d %6d %16s\n\n", pid(), inet_get_local_port(sock), inet_get_ip_source(sock)); } } } probe signal.send{ if (execname() == pname || execname() == pname_rpc_handler){ printf("\nSIGNAL SENT\n"); printf("%-8s %-16s %-5s %-16s %6s %-16s\n", "SPID", "SNAME", "RPID", "RNAME", "SIGNUM", "SIGNAME"); printf("%-8d %-16s %-5d %-16s %-6d %-16s\n\n", pid(), execname(), sig_pid, pid_name, sig, sig_name); }else{ if (pid_name == pname || pid_name == pname_rpc_handler){ printf("\nSIGNAL RECIEVED\n"); printf("%-8s %-16s %-5s %-16s %6s %-16s\n", "SPID", "SNAME", "RPID", "RNAME", "SIGNUM", "SIGNAME"); printf("%-8d %-16s %-5d %-16s %-6d %-16s\n\n", pid(), execname(), sig_pid, pid_name, sig, sig_name); } } } probe timer.s(3){ printf("(%s): reads = %d, write = %d\n",pname, readcount, writecount); totalreadcount += readcount; totalwritecount += writecount; readcount = 0; writecount = 0; } probe timer.s(10){ printf("Total Counts: READS(%d) WRITES(%d)\n", totalreadcount, totalwritecount); }

Sample Output:

[khaled@centos6 ~]$ stap -v networker.stp Pass 1: parsed user script and 96 library script(s) using 200576virt/26776res/3152shr/24072data kb, in 210usr/60sys/416real ms. Pass 2: analyzed script: 7 probe(s), 18 function(s), 30 embed(s), 7 global(s) using 439148virt/130892res/6780shr/122216data kb, in 2570usr/4320sys/20351real ms. Pass 3: using cached /home/khaled/.systemtap/cache/e1/stap_e1c1c67f347eda5734bdd60cd7a9cb0f_25284.c Pass 4: using cached /home/khaled/.systemtap/cache/e1/stap_e1c1c67f347eda5734bdd60cd7a9cb0f_25284.ko Pass 5: starting run. (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(0) WRITES(0) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(0) WRITES(0) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0


(nsrmmd): reads = 0, write = 0 Total Counts: READS(0) WRITES(0) (nsrmmd): reads = 0, write = 0 SIGNAL RECIEVED SPID SNAME RPID RNAME SIGNUM SIGNAME 20629 kill 20448 nsrexecd 15 SIGTERM

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20474 nsrmmd 15 SIGTERM

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20473 nsrmmd 15 SIGTERM SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20472 nsrmmd 15 SIGTERM SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20471 nsrmmd 15 SIGTERM

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20458 nsrsnmd 15 SIGTERM

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM


Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 (nsrmmd): reads = 4, write = 3 SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20472 nsrmmd 20458 nsrsnmd 17 SIGCHLD

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20473 nsrmmd 20458 nsrsnmd 17 SIGCHLD SIGNAL RECIEVED SPID SNAME RPID RNAME SIGNUM SIGNAME 20458 nsrsnmd 20448 nsrexecd 17 SIGCHLD


SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 20448 nsrexecd 15 SIGTERM Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001


Incoming TCP connection to the Networker process PID PORT COMING FROM 20448 7938 0000:0000:0000:0000:0000:ffff:7f00:0001





SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20471 nsrmmd 1 init 17 SIGCHLD

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20474 nsrmmd 1 init 17 SIGCHLD


SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20448 nsrexecd 1 init 17 SIGCHLD (nsrmmd): reads = 0, write = 5 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001



Incoming TCP connection to the Networker process


PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001

SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20682 nsrexecd 20679 nsrexecd 15 SIGTERM SIGNAL SENT SPID SNAME RPID RNAME SIGNUM SIGNAME 20679 nsrexecd 20677 networker 17 SIGCHLD

Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b



Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b

Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b

Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b

Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b

Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process


PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:0a4c:c52b



Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:0a4c:c52b Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:0a4c:c52b



Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001 (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7937 0000:0000:0000:0000:0000:ffff:7f00:0001








Incoming TCP connection to the Networker process


PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001


Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001





Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 Total Counts: READS(4) WRITES(8) Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 (nsrmmd): reads = 0, write = 0 Total Counts: READS(4) WRITES(8) (nsrmmd): reads = 0, write = 0 Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 7938 0000:0000:0000:0000:0000:ffff:7f00:0001

Incoming TCP connection to the Networker process PID PORT COMING FROM 20682 9294 0000:0000:0000:0000:0000:ffff:7f00:0001 Pass 5: run completed in 20usr/380sys/106690real ms.


Conclusion

At the end of this article, I can confidently say that there is a limitless amount of information that

could be retrieved from either the operating system or the user applications that can assist in

gaining a deep understanding of how the OS is interacting with its components and also with the

user space applications that run on top of it. I couldn’t provide intensive examples on the

Avamar or NetWorker user space processes as the debugging information is not available for

either product because of the nature of their source code,

If the engineering team wishes to play around with these products and probe the product

functions on the runtime using SystemTap, it will be important to modify the build “Makefile” of

those products and add the option “-g” to the “GCC” C compiler in order to generate the

required debugging information to be used with SystemTap. Otherwise, SystemTap will be

unable to diagnose either the OS kernel or a user-space applications if the debugging

information is not installed on the system.

Appendix

1- SystemTap Language Reference https://sourceware.org/systemtap/langref 2- SystemTap Beginners Guide https://sourceware.org/systemtap/SystemTap_Beginners_Guide 3- http://doc.opensuse.org/products/draft/SLES/SLEStuning_sd_draft/cha.tuning.systemtap.html

https://sourceware.org/systemtap/langref

https://sourceware.org/systemtap/SystemTap_Beginners_Guide

http://doc.opensuse.org/products/draft/SLES/SLEStuning_sd_draft/cha.tuning.systemtap.html


EMC believes the information in this publication is accurate as of its publication date. The

information is subject to change without notice.

THE INFORMATION IN THIS PUBLICATION IS PROVIDED “AS IS.” EMC CORPORATION

MAKES NO RESPRESENTATIONS OR WARRANTIES OF ANY KIND WITH RESPECT TO

THE INFORMATION IN THIS PUBLICATION, AND SPECIFICALLY DISCLAIMS IMPLIED

WARRANTIES OF MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE.

Use, copying, and distribution of any EMC software described in this publication requires an

applicable software license.

Documents

SYSTEMTAP FOR LINUX PLATFORMS · 2020-05-08 · SYSTEMTAP FOR LINUX PLATFORMS Khaled Ahmed Shareef Bassiouny ... are interpreted but SystemTap is compiled to a binary form known as