Crash Dump AnalysisSystem Debugging in Linux

Vtzslav BtrlaMartin DckJakub Jerm

CrashDumpAnalysisMFFUKSystemDebugginginLinux 2

Overview

Userspacedebuggingtools

Mostlyalreadymentioned gdbandfriends

Kerneldebuggingtools

Historicallylessofficialysupportedtools(invanillakernel) Differentdeveloper'sattitude printk()astheultimatedebuggingtool

CrashDumpAnalysisMFFUKSystemDebugginginLinux 3

LinuxKernelCrashDump

LKCDtoolchain(http://lkcd.sourceforge.net/)

Kernelpatchesforsimplecrashdumpgenerationonkernelpanic SupportedonlyonSCSIdevices

Inkerneldebuggerandpostmortemdebugger SimilarfeaturesasmdbK

Heuristicalanalysis

Activedevelopmentduring2.5.xtimeframe Lastsupportedreleasefrom2006

CrashDumpAnalysisMFFUKSystemDebugginginLinux 4

Standarddebuggingmeans

MagicSysRqhotkeys

Fordealingwithhangsandsecurityissues Operator'sinterventiontotherunningsystem Canbeenabled/disabledby/proc/sys/kernel/sysrq

Alt+SysRq+0..9 setconsolelogginglevel Alt+SysRq+C activatekexecandgenerateacrashdump Alt+SysRq+B immediatereboot Alt+SysRq+O immediateshutdown Alt+SysRq+S syncallmountedfilesystems Alt+SysRq+U remountallfilesystemsreadonly

CrashDumpAnalysisMFFUKSystemDebugginginLinux 5

Standarddebuggingmeans(2) Alt+SysRq+P dumpregisterstoconsole Alt+SysRq+T dumpprocessinformationtoconsole Alt+SysRq+L dumpstacktracesofrunningthreads Alt+SysRq+M dumpmemorystatisticstoconsole Alt+SysRq+D dumplockedlockstoconsole Alt+SysRq+K killallprocessesonthecurrentconsole Alt+SysRq+E terminateallprocessesexceptinit Alt+SysRq+I killallprocessesexceptinit Alt+SysRq+F executetheOOMkiller Alt+SysRq+N resetnicelevelofallrealtimeprocesses Alt+SysRq+X switchoffrawkeyboardmode

CrashDumpAnalysisMFFUKSystemDebugginginLinux 6

Standarddebuggingmeans(3) Activatefromcommandlinebywritinginto/proc/sysrqtrigger

Activateovernetworkbyaspecialsysrqdserver

RaisingElephantsIsSoUtterlyBoringRebootEvenIfSystemUtterlyBroken

Rawkeyboard SendSIGTERMtoallprocesses SendSIGKILLtoallprocesses Syncdatatodisk Remountallfilesystemsreadonly Reboot

CrashDumpAnalysisMFFUKSystemDebugginginLinux 7

Standarddebuggingmeans(4)

kprobes

Basickernelinstrumentation Anexecutionofanykernelfunctioncantriggeracallbackfunction

kexec

Bootinganew(Linux)kernelwithoutphysicalreboot Physicalmemorycanbe(moreorless)retainedandanalyzedbythenewkernel(asacrashdump)

CrashDumpAnalysisMFFUKSystemDebugginginLinux 8

UserModeLinux

UML

Specialpseudohardwarearchitecture Otherwisecompatiblewiththetargetarchitecture RunningLinuxkernelasauserspaceprocess

Originallyavirtualizationeffort

Greatfordebuggingandkerneldevelopment Aplainstandardgdbcanbeusedtoattachtotherunningkernel GuestthreadsarethreadsoftheUMLprocess

Slightlymorecomplicatedtofollowprocesses

CrashDumpAnalysisMFFUKSystemDebugginginLinux 9

KDB

Linuxkerneldebugger

http://oss.sgi.com/projects/kdb Asetofpatchestothevanillakernel(veryunprobablytobeintegratedsoon)

Forlocaldebugging

PossibilitiesandfeaturesverysimilartomdbK Activationin/proc/sys/kernel/kdb BreakinbythePausekey(Ctrl+Aonserialconsole)

md memorydisplay mm memorymodify

CrashDumpAnalysisMFFUKSystemDebugginginLinux 10

KDB(2) rd registerdump rm registermodify bt stacktrace bp setbreakpoint...

CrashDumpAnalysisMFFUKSystemDebugginginLinux 11

KGDB

KernelGDB

http://kgdb.sourceforge.net/

Since2.6.26integratedinthevanillakernel(finally!) Client/serverdebuggerforremotekerneldebugging

Overseriallineorethernet Client:Plaingdb

Assemblyandsourcelinedebugging Symbolinformationfromvmlinuxbinaryontheclient

Activation AfterloadingtheI/Omoduleandconnectingtotheserver Justafterbootwhenkgdbwaitkerneloptionisused

CrashDumpAnalysisMFFUKSystemDebugginginLinux 12

KGDB(2)

Features Memoryandregisteranalysisonkerneloopsorpanic Limitedsinglesteppingoftherunningkernel

%gdb./vmlinux(gdb)setremotebaud115200(gdb)targetremote/dev/ttyS0

%gdb./vmlinux(gdb)targetremoteudp:192.168.1.1.:6443

Slide 1Slide 2Slide 3Slide 4Slide 5Slide 6Slide 7Slide 8Slide 9Slide 10Slide 11Slide 12