Embed Size (px)
Citation preview
Woo-Sun Yang!NERSC User Services Group!!NUG Training!February 3, 2014
Debugging Tools
Debuggers on NERSC machines • Parallel debuggers with a graphical user interface
– DDT (Distributed Debugging Tool) – TotalView
• Specialized debuggers on Hopper and Edison – STAT (Stack Trace Analysis Tool)
• Collect stack backtraces from all (MPI) tasks – ATP (Abnormal Termina@on Processing)
• Collect stack backtraces from all (MPI) tasks when an applica@on fails – CCDB (Cray Compara@ve Debugger)
• Compara@ve debugging
• Valgrind – Suite of debugging and profiler tools
-‐ 2 -‐
DDT and TotalView • GUI-‐based tradi>onal parallel debuggers
– Control program’s execu@on pace by it advance to a desired loca@on – Set breakpoints, watchpoints and tracepoints – Display the values of variables and expressions, and visualize arrays
• To check whether the program is execu@ng as expected – Memory debugging – And more…
• Works for C, C++, Fortran programs with MPI, OpenMP, pthreads – DDT supports CAF (Coarray Fortran) and UPC (Unified Parallel C), too
• Maximum applica>on size to use the debuggers at NERSC – DDT: up to 8192 MPI tasks – TotalView: up to 512 MPI tasks – Licenses shared among users
• For info – $ALLINEA_TOOLS_DOCDIR/userguide.pdf (a]er loading the ‘allineatools’ module) – h_p://www.nersc.gov/users/so]ware/debugging-‐and-‐profiling/ddt/ – h_p://www.roguewave.com/products/totalview – h_p://www.nersc.gov/users/so]ware/debugging-‐and-‐profiling/totalview/
-‐ 3 -‐
DDT
-‐ 4 -‐
% ftn -g -O0 -o jacobi_mpi jacobi_mpi.f90!!% qsub -IV -lmppwidth=24,walltime=30:00 -q debug!% cd $PBS_O_WORKDIR!% module load allineatools!% ddt ./jacobi_mpi!
Load the allineatools module to use DDT Start DDT
Compile with -‐g to have debugging symbols Include -‐O0 for the Intel compiler
Start an interac@ve batch session
DDT (cont’d)
-‐ 5 -‐
Navigate using these bu_ons
Parallel stack frame view is helpful in quickly finding out where each process is execu@ng
To check the value of a variable, right-‐click on a variable or check the pane on the right
Sparklines to quickly show varia@on over MPI tasks
TotalView
-‐ 6 -‐
% qsub -IV -lmppwidth=24,walltime=30:00 -q debug!% cd $PBS_O_WORKDIR!% module load totalview!% totalview aprun -a -n 4 ./jacobi_mpi!! Then, • Click OK in the ‘Startup Parameters -‐ aprun’ window • Click ‘Go’ bu_on in the main window
• Click ‘Yes’ to the ques@on ‘Process aprun is a parallel job. Do you want to stop the job
now?’
TotalView (cont’d)
-‐ 7 -‐
To see the value of a variable, right-‐click on a variable to “dive” on it or just hover mouse over it
Navigate using these bu_ons
STAT (Stack Trace Analysis Tool) • Gathers stack backtraces (showing the func>on calling sequences
leading up to the ones in the current stack frames) from all (MPI) processes and merges them into a single file (*.dot) – Results displayed graphically as a call tree showing the loca@on in the
code that each process is execu@ng and how it got there – Can be useful for debugging a hung applica@on – With the info learned from STAT, can inves@gate further with DDT or
TotalView • Works for MPI, CAF and UPC, but not OpenMP • STAT commands (aSer loading the ‘stat’ module)
– stat (STAT or stat-‐cl): invokes STAT to gather stack backtraces – statview (STATview or stat-‐view): a GUI to view the results – statgui (STATGUI or stat-‐gui): a GUI to run STAT or view results
• For more info: – ‘intro_stat’, ‘STAT’, ‘statview’ and ‘statgui’ man pages – h_ps://[email protected]/code/STAT/stat_userguide.pdf – h_p://www.nersc.gov/users/so]ware/debugging-‐and-‐profiling/stat-‐2/
-‐ 8 -‐
Hung application with STAT
-‐ 9 -‐
% qstat -f 722272!...! login_node_id = nid02051!!% ssh -XY nid02051!% ps -f!UID PID PPID C STIME TTY TIME CMD!wyang 23953 16045 0 Feb01 pts/0 00:00:00 aprun -n 4 ./jacobi_mpi!wyang 23961 23953 0 Feb01 pts/0 00:00:00 aprun -n 4 ./jacobi_mpi!…!% module load stat!% stat -i 23953!…!Attaching to application...!Attached!!Application already paused... ignoring request to pause!Sampling traces...!Traces sampled!!…!Resuming the application...!Resumed!!Merging traces...!Traces merged!!Detaching from application...!Detached!!!Results written to /scratch1/scratchdirs/wyang/parallel_jacobi/stat_results/jacobi_mpi.0010!
% ls -l stat_results/jacobi_mpi.0010/*.dot!-rw-r----- 1 wyang wyang 2227 Feb 2 00:09 stat_results/jacobi_mpi.0010/jacobi_mpi.0010.3D.dot!% statview stat_results/jacobi_mpi.0010/jacobi_mpi.0010.3D.dot!
Find the MOM node that launched the app.
Find pid Log into the MOM node
Run ‘stat’ for the process 10921; -‐i to get source line numbers STAT samples stack backtraces a few @mes
Hung application with STAT (Cont’d)
-‐ 10 -‐
Rank 3 is here
Ranks 1 & 2 are here
Rank 0 is here
ATP (Abnormal Termination Processing) • ATP gathers stack backtraces from all processes of a failing applica>on
– Invokes STAT underneath – Output in atpMergedBT.dot and atpMergedBT_line.dot (which shows source code
line numbers), which are to be viewed with statview • By default, the atp module is loaded on Hopper and Edison, but ATP is
not enabled; to enable: – setenv ATP_ENABLED 1 # csh/tcsh – export ATP_ENABLED=1 # sh/bash/ksh Include this in your dot file (e.g., .tcshrc.ext) to enable ATP by default
• Can make core dumps (core.atp.apid.rank), too, by seZng coredumpsize unlimited: – unlimit coredumpsize # for csh/tcsh – ulimit -‐c unlimited # for sh/bash/ksh but they do not represent the exact same moment in >me (therefore the loca>on of a failure can be inaccurate)
• For more info – ‘intro_atp’ man page – h_p://www.nersc.gov/users/so]ware/debugging-‐and-‐profiling/gdb-‐and-‐atp/
-‐ 11 -‐
Running an application with ATP
-‐ 12 -‐
% cp $ATP_HOME/demos/testMPIApp.c .!% cc -o testMPIApp testMPIApp.c!% cat runit!#!/bin/csh!#PBS -l mppwidth=24!#PBS -l walltime=5:00!#PBS -q debug!#PBS -j oe!! cd $PBS_O_WORKDIR! setenv ATP_ENABLED 1! aprun -n 8 ./testMPIApp 1 4!!% qsub runit!714152.edique02!!% cat runit.o714152!…!Application 2885291 is crashing. ATP analysis proceeding...!…!Process died with signal 4: 'Illegal instruction'!View application merged backtrace tree with: statview atpMergedBT.dot!…!% module load stat!% statview atpMergedBT.dot # or statview atpMergedBT_line.dot!
Enable ATP
Hung application with ATP • Force to generate backtraces from a hung applica>on • For the following to work, must have used
– ‘setenv ATP_ENABLED 1’ in batch script – ‘setenv FOR_IGNORE_EXCEPTION true’ in batch script for Intel Fortran – ‘-‐f no-‐backtrace’ at compile/link @me for GNU Fortran
-‐ 13 -‐
% apstat!…! Apid ResId User PEs Nodes Age State Command!2885161 140092 wyang 4 1 0h02m run jacobi_mpi!…!% apkill 2885161!% cat runit.o714080!…!aprun: Apid 2885161: Caught signal Terminated, sending to application!…!Process died with signal 15: 'Terminated'!View application merged backtrace tree with: statview atpMergedBT.dot!…!% module load stat!% statview atpMergedBT.dot # or statview atpMergedBT_line.dot!
Find apid
Kill the applica@on
CCDB (Cray Comparative Debugger) • Find a bug introduced in a version, by running two
versions side by side and comparing data between them
• GUI • It runs the command line mode version, lgdb (Cray Line
Mode Parallel Debugger), underneath • Supports MPI; doesn’t support threading • For info:
– ccdb man page and help pages – lgdb man page and help pages – ‘Using the lgdb Compara3ve Debugging Feature’, h_p://docs.cray.com/books/S-‐0042-‐22/S-‐0042-‐22.pdf
– h_p://www.nersc.gov/users/so]ware/debugging-‐and-‐profiling/ccdb-‐lgdb/ (work in progress)
-‐ 14 -‐
CCDB (Cont’d) • To compare something between two applica>ons, need to specify – Variable name – Loca@on in a source file – How the global data for the variable is distributed over MPI processes
– Set of MPI processes (“PE set”) for the distribu@on • 3 en>>es used in CCDB (and lgdb) – PE set: A set of MPI processes – Decomposi@on: How an array is distributed over PEs – Asser@on script: A collec@on of mathema@cal rela@onships (e.g., equality) to be tested
-‐ 15 -‐
Running CCDB
-‐ 16 -‐
PE set for 1st app PE set for 2nd app
1st app 2nd app
% qsub -IV -lmppwidth=48,walltime=30:00 -q debug!% cd $PBS_O_WORKDIR!% module load cray-ccdb!% ccdb
Request enough nodes to run two apps. simultaneously
CCDB assertion script
-‐ 17 -‐
• This script tests whether the 6 variables have the same values between the applica>ons, at line 418 of HPL_pdtest.c; resid0 and XnormI don’t
Valgrind
• Suite of debugging and profiler tools • Tools include – memcheck: memory error and memory leaks detec@on – cachegrind: a cache and branch-‐predic@on profiler – callgrind: a call-‐graph genera@ng cache and branch predic@on profiler
– massif, dhat (exp-‐dhat): heap profilers – helgrind, drd: pthreads error detectors
• For info: – h_p://valgrind.org/docs/manual/manual.html
-‐ 18 -‐
Valgrind’s memcheck
-‐ 19 -‐
% qsub -IV -lmppwidth=24,walltime=30:00 -q debug!% cd $PBS_O_WORKDIR!% module load valgrind!% ftn -dynamic -g -O0 memory_leaks.f $VALGRIND_MPI_LINK!% aprun -n 2 valgrind --leak-check=full ./a.out >& report!% awk ‘/^==/ {print $1}’ report | sort -u!==46886==!==46887==!==46888==!==46889==!!
Found four sub-‐reports, each star@ng a line with its PID (-‐-‐log-‐file with %p doesn’t seem to work properly with aprun…)
% awk ‘/^==46888/ {print}’ report!…!==46888== 8,000,000 bytes in 2 blocks are possibly lost in loss record 33 of 37!==46888== at 0x4C27F9E: malloc (vg_replace_malloc.c:291)!==46888== by 0x424A93: for_allocate (in /scratch1/scratchdirs/wyang/valgrind/a.out)!==46888== by 0x408269: sub_bad_ (memory_leaks.f:37)!==46888== by 0x407DF5: MAIN__ (memory_leaks.f:14)!==46888== by 0x407D55: main (in /scratch1/scratchdirs/wyang/valgrind/a.out)!…!
• Out of 4 sub-‐reports, two are for the applica>on’s 2 MPI tasks; let’s look at the one for process ID 46888
• Can suppress spurious error messages by using a suppression file (-‐-‐suppressions=/path/to/directory/file)
Could have explicitly added ‘-‐-‐tool=memcheck’; save res in ‘report’
Show the result for PID 46888
Valgrind’s cachegrind
-‐ 20 -‐
% ftn -g -O2 memory_leaks.f!% aprun -n 2 valgrind --tool=cachegrind ./a.out!% ls -lrt!…!-rw------- 1 wyang wyang 14112 Jan 31 08:55 cachegrind.out.46848!-rw------- 1 wyang wyang 34532 Jan 31 08:55 cachegrind.out.46849!-rw------- 1 wyang wyang 33926 Jan 31 08:55 cachegrind.out.46850!-rw------- 1 wyang wyang 15251 Jan 31 08:55 cachegrind.out.46847!
• It generates 4 separate reports just like before, and two are for the applica>on’s 2 MPI tasks; let’s look at the one for process ID 46849
% cg_annotate cachegrind.out.46849 --------------------------------------------------------------------------------!I1 cache: 32768 B, 64 B, 8-way associative!D1 cache: 32768 B, 64 B, 8-way associative!LL cache: 31457280 B, 64 B, 30-way associative!Command: ./a.out!Data file: cachegrind.out.46849!…!--------------------------------------------------------------------------------! Ir I1mr ILmr Dr D1mr DLmr Dw D1mw DLmw file:function!--------------------------------------------------------------------------------!220,000,000 4 4 28,000,000 12 2 20,000,000 0 0 ???:for_random_number_single! 25,250,536 43 40 500,069 250,016 2 8,000,073 250,020 187,498 /some/path/memory_leak.f:MAIN__!
…!
Use -‐g but keep the usual op@miza@on level
Valgrind’s callgrind
-‐ 21 -‐
% ftn -g -O2 memory_leaks.f!% aprun -n 2 valgrind --tool=callgrind ./a.out!% ls -lrt!…!-rw------- 1 wyang wyang 38527 Feb 1 13:16 callgrind.out.43223!-rw------- 1 wyang wyang 117581 Feb 1 13:17 callgrind.out.43225!-rw------- 1 wyang wyang 124967 Feb 1 13:17 callgrind.out.43224!-rw------- 1 wyang wyang 47409 Feb 1 13:17 callgrind.out.43222!
• It generates 4 separate reports just like before, and two are for the applica>on’s 2 MPI tasks; let’s look at the one for process ID 43224
% callgrind_annotate callgrind.out.43224 memory_leaks.f!…!--------------------------------------------------------------------------------!-- User-annotated source: memory_leaks.f!--------------------------------------------------------------------------------!…! . subroutine sub_ok(val,n) ! no memory leak! . integer n! . real val! 40 real, allocatable :: a(:)! 240 allocate (a(n))! 2,425 => ???:for_alloc_allocatable (10x)! 810 => ???:for_check_mult_overflow64 (10x)!50,000,070 call random_number(a)!550,000,000 => ???:for_random_number_single (10000000x)! 3,125,926 val = val + sum(a)! . ! deallocate(a) ! ok not to deallocate! 180 end! 1,379 => ???:for_dealloc_allocatable (10x)!
…!
Use -‐g but keep the usual op@miza@on level
Valgrind’s heap profilers
-‐ 22 -‐
% ifort -g -O2 memory_leaks_serial.f!% valgrind --tool=massif ./a.out!% ls -lrt!…!-rw------- 1 wyang wyang 12008 Feb 1 12:40 massif.out.1384!% ms_print massif.out.1384!...! MB!114.5^ #! | :::#! | ::: :#! | ::::: : :#! | :: : : : :#! | :::::: : : : :#! | ::: : :: : : : :#! | ::@: : : :: : : : :#! | ::: @: : : :: : : : :#! | ::::: : @: : : :: : : : :#! | :: : : : @: : : :: : : : :#! | :::::: : : : @: : : :: : : : :#! | ::: : :: : : : @: : : :: : : : :#! | @::: : : :: : : : @: : : :: : : : :#! | ::@: : : : :: : : : @: : : :: : : : :#! | ::::: @: : : : :: : : : @: : : :: : : : :#! | ::: : : @: : : : :: : : : @: : : :: : : : :#! | :::: : : : @: : : : :: : : : @: : : :: : : : :#! | ::: :: : : : @: : : : :: : : : @: : : :: : : : :#! | ::::: : :: : : : @: : : : :: : : : @: : : :: : : : :#! 0 +----------------------------------------------------------------------->Gi! 0 2.248!…!--------------------------------------------------------------------------------! n time(i) total(B) useful-heap(B) extra-heap(B) stacks(B)!--------------------------------------------------------------------------------!…! 67 2,414,128,664 120,000,280 120,000,032 248 0!
!
• massif and exp-‐dhat for profiling heap memory usage Doesn’t seem to work with compiler wrappers at the moment, so let’s try a serial code
@ for detailed snapshot where detailed info is provided # for peak snapshot where the peak heap usage is This example strongly suggests memory leaks
Thank you.
-‐ 23 -‐
