SystemTap SystemTap

http://d3s.mff.cuni.czCrash Dump Analysis 2014/2015

CHARLES UNIVERSITY IN PRAGUE

faculty of mathematics and physicsfaculty of mathematics and physics

SystemTapSystemTap

Crash Dump Analysis 2014/2015 2SystemTap

SystemTapSystemTap

Dynamic analysis frameworkInspired by DTraceLinux, GPLUnder heavy developmentSuitable for production usage

Motto: “Painful to use, but more painful not to”


TracingTracing

Capture information about executionOften specialized tools (strace)Limited filteringOverwhelming amount of data


ProfilingProfiling

Sampling during executionBoth targeted and system-wideOften just one metric: time


DebuggingDebugging

Full context (memory, registers, backtrace)BreakpointsTargeted


EnvironmentsEnvironments

DevelopmentRich environment: tools, debuginfoExclusive access and control

ProductionDifferent from developmentUnder loadValuableVirtualized / containerized


SystemTap to rule them all, in productionSystemTap to rule them all, in production

Tracing, profiling, debuggingSystem-wideMonitoring multiple event types (and layers)SafeLow overheadControlled dependencies


Basic principlesBasic principles

EventsInteresting points in running systemDifferent abstraction levels: code line/model eventKernel and user space

ActionsScripting languageAccess to program state, capturing, processingEven state modification if you are bold


EventsEvents

Low-levelFunction entry and returnCode lines

High-levelTimersCode tracepointsTapsets


Events: ProbesEvents: Probes

Probe: Basic SystemTap elementBind actions to events

Probe specification: Defines interesting eventsProbe hit: Specific event occurrenceProbe handler: Action to perform when hit


Language: ProbesLanguage: Probes

probe kernel.function(“vfs_read”) {

log(“vfs_read was called!”)

}

keyword providerevent

specification

event handler


Language: ProbesLanguage: Probes

Probes


Probes: GenericProbes: Generic

beginend errortimer.jiffies()timer.ms()


Probes: System callsProbes: System calls

Needs DWARF debuginfosyscall.NAMEsyscall.NAME.return

Do not need DWARF debuginfond_syscall.NAMEnd_syscall.NAME.return


Probes: Kernel DWARFProbes: Kernel DWARF

kernel.function(PATTERN)module(PATTERN).function(PATTERN)kernel.statement(PATTERN)module(PATTERN).statement(PATTERN)

Provides access to context variables, arguments, reachable memory etc.


Probes: Modifiers and variantsProbes: Modifiers and variants

function.returnfunction.call / function.inlinefunction.callee(NAME)function.callees(DEPTH)


Probes: Kernel DWARF-lessProbes: Kernel DWARF-less

kprobe.function(FUN) + .returnkprobe.module(MOD).function(FUN) + .return

No access to variables etc.


Probes: Kernel tracepointsProbes: Kernel tracepoints

kernel.trace(PATTERN)

Static markers in kernelFaster and more reliable mechanismAccess to macro arguments


Probes: Userspace DWARFProbes: Userspace DWARF

process(PATH/PID).function(NAME)process(PATH/PID).statement(PATTERN)process(P).library(P).function(NAME)process(P).library(P).statement(PATTERN)

Provides access to context variables, arguments, reachable memory etc.


Probes: Userspace via utraceProbes: Userspace via utrace

process({,PATH,PID}).{begin,end}process({,PATH,PID}).thread.{begin,end}process({,PATH,PID}).syscall + .return


Probes: Userspace static markersProbes: Userspace static markers

process(PID/PATH).mark(LABEL)process(P).provider(PROVIDER).mark(LABEL)

Static markers put by developers to applicationSTAP_PROBE/DTRACE_PROBE macrosAccess to macro arguments


Probes: PythonProbes: Python

Markers in Python runtimeTapset to work with Python structure


Probes: JavaProbes: Java

Prototype featureByteman backend

Example:probe java(PNAME/PID).class(NAME).method(METHOD)


Probes: Find available probesProbes: Find available probes

stap -l 'probe definition'

Example:$ stap -l 'kernel.trace(“kmem:*”)'


ActionsActions

Actions


Language: Elements [1/2]Language: Elements [1/2]

Functions: built-in, user defined, tapsetsProbe aliasesControl structures: conditionals, loops, ternary opVariables

Scalar: string, integers, no declarationsMulti-key associative arrays (global only)Global or local scope

Aggregates


Language: Elements [2/2]Language: Elements [2/2]

Operators: Usual ops, member operatorPointer typecastingConditional compilationSimple preprocessor macrosEmbedded C


Language: ReadingLanguage: Reading

Context and tapset functions pid(), execname(), print_backtrace()

Context variablesPrefixed by '$'$ stap -L 'probe kernel.DEF'

Tapset variablesSet by a tapset in a prologue

Pretty printers and groups$$vars, $$locals, $$parms, $$parms$, $$parms$$


Language: ParametersLanguage: Parameters

$ stap script.stp param1 param2 ...Unquoted: $1, $2...Strings: @1, @2...


Language: OutputLanguage: Output

Functionslog()printf()sprintf()...more...


Language: AggregatesLanguage: Aggregates

Aggregates: Support for data accumulation and statistics

aggregate <<< valuearray[execname(), pid()] <<< value@count, @sum, @min, @max, @avg@hist_linear(agg, LOW, HIGH, WIDTH)


Language: QuirksLanguage: Quirks

Often quirky syntactic shortcuts@entry(expression): saves expression value in entry probe for usage in return probeforeach(v = [i,j] in array)

Read language reference...


TapsetsTapsets

Tapset = “Library”Collection of SystemTap functions and aliasesEncapsulates internals via aliasesProvides convenience functions

Example: /usr/share/systemtap/tapset/linux/syscalls2.stp


UsageUsage

Usage


Using SystemTapUsing SystemTap

Write a scriptOne-shot script executionLong-term result collectionFlight-recorder mode


SystemTap passesSystemTap passes

Pass 1: ParserPass 2: Process probes, data structures etc.Pass 3: Translate to C (cached)Pass 4: Compile as a kernel module (cached)

Pass 5: Insert module


SystemTap translator / driverSystemTap translator / driver

Command: stapHigh-level driver commandAll or some passesSystem-wide or targeted

Examples:$ stap -e 'probe kernel.trace(“kmem:*”)

$ stap script.stp

$ stap -p4 script.stp

$ stap (…) -c 'command'

$ stap (…) -x PID


SystemTap runtimeSystemTap runtime

Command: staprunLoads a SystemTap probe kernel modulestap = stap -p4 + staprun

Examples:$ staprun stap_ee1d4debf0add5c64f0b095_1991.ko


Permission modelPermission model

Privileged operationsTherefore, SystemTap needs either:

root useruser in groups: stapusr+stapdevuser in groups: stapusr+stapsysuser in groups: stapusr


Permission modelPermission model

stapusr + stapdevCan run SystemTap as if root

stapusr + stapsysCan run prebuilt, signed modulesCan run limited functionality scripts

Avoid damage to system

stapusrCan run prebuilt and/or signed modulesCan run limited functionality scripts

Disallow access to other user informationDisallow harming other user process performance


Flight recorder and SystemTap serviceFlight recorder and SystemTap service

Flight recorder$ stap -F …Load module and detach......or run staprun as a daemon

SystemTap service/etc/systemtap/script.d/script.stp/etc/systemtap/conf.d/script.conf$ service systemtap start script


Guru modeGuru mode

Fun, useful and a little bit perilous!Bypass security limitsEmbedded CWrite to context variablesUsage: hotfixes, hacks, showcases, fun

Example:$ stap -g -e 'probe syscall.kill {if (pid==$1) { $pid = pid() } }' PID


SystemTap compile serverSystemTap compile server

ServerHas SystemTap translator and module builderHas DWARF debuginfoBuilds and signs SystemTap modules

ClientHave just SystemTap client and runtime


Final wordsFinal words

Final words


SystemTap: Problem solving processSystemTap: Problem solving process

Determine involved componentsKernel, executables, libraries

Investigate possible probe pointsWhat events are interesting?What events carry interesting data?

Determine desired outputHistogram, specific knowledge, data set

Start with generic probes, then limit


SystemTap: PatternsSystemTap: Patterns

Event logger: simple output on eventEvent detector: complicated filtering in probesTop: data collection, printing in timer.msData collection: runs until cancel, end probe


SystemTap: ResourcesSystemTap: Resources

man pagesstap, stapprobes

Exampleshttps://sourceware.org/systemtap/examples/

Tutorials and articleshttps://sourceware.org/systemtap/wiki

Language, tapset referenceshttps://sourceware.org/systemtap/documentation.html

https://sourceware.org/systemtap/examples/

https://sourceware.org/systemtap/wiki

https://sourceware.org/systemtap/documentation.html

Documents

SystemTap SystemTap