SystemTap SystemTap

http://d3s.mff.cuni.czCrash Dump Analysis 2014/2015

CHARLES UNIVERSITY IN PRAGUE

faculty of mathematics and physicsfaculty of mathematics and physics

SystemTapSystemTap

Crash Dump Analysis 2014/2015 2SystemTap

SystemTapSystemTap

Dynamic analysis frameworkInspired by DTraceLinux, GPLUnder heavy developmentSuitable for production usage

Motto: “Painful to use, but more painful not to”

TracingTracing

Capture information about executionOften specialized tools (strace)Limited filteringOverwhelming amount of data

ProfilingProfiling

Sampling during executionBoth targeted and system-wideOften just one metric: time

DebuggingDebugging

Full context (memory, registers, backtrace)BreakpointsTargeted

EnvironmentsEnvironments

DevelopmentRich environment: tools, debuginfoExclusive access and control

ProductionDifferent from developmentUnder loadValuableVirtualized / containerized

SystemTap to rule them all, in productionSystemTap to rule them all, in production

Tracing, profiling, debuggingSystem-wideMonitoring multiple event types (and layers)SafeLow overheadControlled dependencies

Basic principlesBasic principles

EventsInteresting points in running systemDifferent abstraction levels: code line/model eventKernel and user space

ActionsScripting languageAccess to program state, capturing, processingEven state modification if you are bold

EventsEvents

Low-levelFunction entry and returnCode lines

High-levelTimersCode tracepointsTapsets

Events: ProbesEvents: Probes

Probe: Basic SystemTap elementBind actions to events

Probe specification: Defines interesting eventsProbe hit: Specific event occurrenceProbe handler: Action to perform when hit

Language: ProbesLanguage: Probes

probe kernel.function(“vfs_read”) {

log(“vfs_read was called!”)

keyword providerevent

specification

event handler

Language: ProbesLanguage: Probes

Probes

Probes: GenericProbes: Generic

beginend errortimer.jiffies()timer.ms()

Probes: System callsProbes: System calls

Needs DWARF debuginfosyscall.NAMEsyscall.NAME.return

Do not need DWARF debuginfond_syscall.NAMEnd_syscall.NAME.return

Probes: Kernel DWARFProbes: Kernel DWARF

kernel.function(PATTERN)module(PATTERN).function(PATTERN)kernel.statement(PATTERN)module(PATTERN).statement(PATTERN)

Provides access to context variables, arguments, reachable memory etc.

Probes: Modifiers and variantsProbes: Modifiers and variants

function.returnfunction.call / function.inlinefunction.callee(NAME)function.callees(DEPTH)

Probes: Kernel DWARF-lessProbes: Kernel DWARF-less

kprobe.function(FUN) + .returnkprobe.module(MOD).function(FUN) + .return

No access to variables etc.

Probes: Kernel tracepointsProbes: Kernel tracepoints

kernel.trace(PATTERN)

Static markers in kernelFaster and more reliable mechanismAccess to macro arguments

Probes: Userspace DWARFProbes: Userspace DWARF

process(PATH/PID).function(NAME)process(PATH/PID).statement(PATTERN)process(P).library(P).function(NAME)process(P).library(P).statement(PATTERN)

Provides access to context variables, arguments, reachable memory etc.

Probes: Userspace via utraceProbes: Userspace via utrace

process({,PATH,PID}).{begin,end}process({,PATH,PID}).thread.{begin,end}process({,PATH,PID}).syscall + .return

Probes: Userspace static markersProbes: Userspace static markers

process(PID/PATH).mark(LABEL)process(P).provider(PROVIDER).mark(LABEL)

Static markers put by developers to applicationSTAP_PROBE/DTRACE_PROBE macrosAccess to macro arguments

Probes: PythonProbes: Python

Markers in Python runtimeTapset to work with Python structure

Probes: JavaProbes: Java

Prototype featureByteman backend

Example:probe java(PNAME/PID).class(NAME).method(METHOD)

Probes: Find available probesProbes: Find available probes

stap -l 'probe definition'

Example:$ stap -l 'kernel.trace(“kmem:*”)'

ActionsActions

Actions

Language: Elements [1/2]Language: Elements [1/2]

Functions: built-in, user defined, tapsetsProbe aliasesControl structures: conditionals, loops, ternary opVariables

Scalar: string, integers, no declarationsMulti-key associative arrays (global only)Global or local scope

Aggregates

Language: Elements [2/2]Language: Elements [2/2]

Operators: Usual ops, member operatorPointer typecastingConditional compilationSimple preprocessor macrosEmbedded C

Language: ReadingLanguage: Reading

Context and tapset functions pid(), execname(), print_backtrace()

Context variablesPrefixed by '$'$ stap -L 'probe kernel.DEF'

Tapset variablesSet by a tapset in a prologue

Pretty printers and groups$$vars, $$locals, $$parms, $$parms$, $$parms$$

Language: ParametersLanguage: Parameters

$ stap script.stp param1 param2 ...Unquoted: $1, $2...Strings: @1, @2...

Language: OutputLanguage: Output

Functionslog()printf()sprintf()...more...

Language: AggregatesLanguage: Aggregates

Aggregates: Support for data accumulation and statistics

aggregate <<< valuearray[execname(), pid()] <<< value@count, @sum, @min, @max, @avg@hist_linear(agg, LOW, HIGH, WIDTH)

Language: QuirksLanguage: Quirks

Often quirky syntactic shortcuts@entry(expression): saves expression value in entry probe for usage in return probeforeach(v = [i,j] in array)

Read language reference...

TapsetsTapsets

Tapset = “Library”Collection of SystemTap functions and aliasesEncapsulates internals via aliasesProvides convenience functions

Example: /usr/share/systemtap/tapset/linux/syscalls2.stp

UsageUsage

Using SystemTapUsing SystemTap

Write a scriptOne-shot script executionLong-term result collectionFlight-recorder mode

SystemTap passesSystemTap passes

Pass 1: ParserPass 2: Process probes, data structures etc.Pass 3: Translate to C (cached)Pass 4: Compile as a kernel module (cached)

Pass 5: Insert module

SystemTap translator / driverSystemTap translator / driver

Command: stapHigh-level driver commandAll or some passesSystem-wide or targeted

Examples:$ stap -e 'probe kernel.trace(“kmem:*”)

$ stap script.stp

$ stap -p4 script.stp

$ stap (…) -c 'command'

$ stap (…) -x PID

SystemTap runtimeSystemTap runtime

Command: staprunLoads a SystemTap probe kernel modulestap = stap -p4 + staprun

Examples:$ staprun stap_ee1d4debf0add5c64f0b095_1991.ko

Permission modelPermission model

Privileged operationsTherefore, SystemTap needs either:

root useruser in groups: stapusr+stapdevuser in groups: stapusr+stapsysuser in groups: stapusr

Permission modelPermission model

stapusr + stapdevCan run SystemTap as if root

stapusr + stapsysCan run prebuilt, signed modulesCan run limited functionality scripts

Avoid damage to system

stapusrCan run prebuilt and/or signed modulesCan run limited functionality scripts

Disallow access to other user informationDisallow harming other user process performance

Flight recorder and SystemTap serviceFlight recorder and SystemTap service

Flight recorder$ stap -F …Load module and detach......or run staprun as a daemon

SystemTap service/etc/systemtap/script.d/script.stp/etc/systemtap/conf.d/script.conf$ service systemtap start script

Guru modeGuru mode

Fun, useful and a little bit perilous!Bypass security limitsEmbedded CWrite to context variablesUsage: hotfixes, hacks, showcases, fun

Example:$ stap -g -e 'probe syscall.kill {if (pid==$1) { $pid = pid() } }' PID

SystemTap compile serverSystemTap compile server

ServerHas SystemTap translator and module builderHas DWARF debuginfoBuilds and signs SystemTap modules

ClientHave just SystemTap client and runtime

Final wordsFinal words

Final words

SystemTap: Problem solving processSystemTap: Problem solving process

Determine involved componentsKernel, executables, libraries

Investigate possible probe pointsWhat events are interesting?What events carry interesting data?

Determine desired outputHistogram, specific knowledge, data set

Start with generic probes, then limit

SystemTap: PatternsSystemTap: Patterns

Event logger: simple output on eventEvent detector: complicated filtering in probesTop: data collection, printing in timer.msData collection: runs until cancel, end probe

SystemTap: ResourcesSystemTap: Resources

man pagesstap, stapprobes

Exampleshttps://sourceware.org/systemtap/examples/

Tutorials and articleshttps://sourceware.org/systemtap/wiki

Language, tapset referenceshttps://sourceware.org/systemtap/documentation.html

SystemTap SystemTap

Documents

Dynamic Tracing with DTrace & SystemTap - GitHub Pagesmyaut.github.io/dtrace-stap-book/dtrace-stap-book.pdf · Thanks DTrace SystemTap community for creating such great instruments,

Advanced OS performance tools and their more basic alternatives … · 2019. 12. 6. · Systemtap safety “In practice, there are several weak points in systemtap and the underlying

SystemTap Beginners Guideir.archive.ubuntu.com/redhat/RHEL_7.2/Docs/Red_Hat...Red Hat Enterprise Linux 7 SystemTap Beginners Guide Introduction to SystemTap William Cohen Red Hat Software

ThermTap: An Online Power Analyzer and Thermal Simulator for … · 2015. 6. 9. · SystemTap which is an industry-standard kernel debugging and performance monitoring tool [7]. SystemTap

Red Hat Enterprise Linux 7 · Red Hat Enterprise Linux 7 SystemTap Beginners Guide Introduction to SystemTap William Cohen Red Hat Software Engineering wcohen@redhat.com Don Domingo

Monitoring MySQL with DTrace/SystemTap

Linux SystemTap Beginners Guide

SystemTap Beginners Guide - Introduction to SystemTap · SystemTap Beginners Guide Introduction to SystemTap ... SystemTap 3.0 SystemTap Beginners Guide Introduction to SystemTap

Red Hat Enterprise Linux-5-SystemTap Language Reference-En-US

HOW I FELL IN LOVE WITH SYSTEMTAP AND PERF · PDF filehow i fell in love with systemtap and perf ralph bÖhme, sernet, samba team understanding and improving samba fileserver performance

SystemTap Beginners Guide - Introduction ... - sourceware.org

Tracing and profiling production code with SystemTapfruli.krunch.be/~krunch/systemtap-osdcfr-20101010.pdf · 2010. 10. 10. · SystemTap as to what’s going on in the upper layers

Observing HotSpot with SystemTap

SystemTap Beginners Guide Red Hat Enterprise Linux 5 ...people.redhat.com/wcohen/systemtap-doc/SystemTap... · Preferences. In the Buttons tab, click the Left-handed mouse check box

Red Hat Enterprise Linux 5 SystemTap Language Reference

SYSTEMTAP FOR LINUX PLATFORMS · 2020-05-08 · SYSTEMTAP FOR LINUX PLATFORMS Khaled Ahmed Shareef Bassiouny ... are interpreted but SystemTap is compiled to a binary form known as

Debugging Gluster with Wireshark and SystemTap - Red …people.redhat.com/ndevos/talks/fisl15/debugging-gluster-with... · Debugging Gluster with Wireshark and SystemTap ... The GlusterFS-client

SystemTap: Instrumenting the Linux Kernel for Analyzing ... · 6 SystemTap: Instrumenting the Linux Kernel for Analyzing Performance and Functional Problems 2.1 Architectural overview

SystemTap Beginners Guide

SystemTap Beginners Guide - Introduction to SystemTapsourceware.org/systemtap/SystemTap_Beginners_Guide.pdf · Mono-spaced Bold Italic or Proportional Bold Italic Whether mono-spaced