Upload
gayora
View
22
Download
0
Tags:
Embed Size (px)
DESCRIPTION
UPC CHECK: A scalable tool for detecting run-time errors in Unified Parallel C. Indranil Roy High Performance Computing (HPC) group. Segmentation error. Core dumped. A good error message. - PowerPoint PPT Presentation
Citation preview
UPC CHECK: A scalable tool for detecting run-time errors in Unified Parallel CIndranil RoyHigh Performance Computing (HPC) group
Segmentation error. Core dumped.
A good error messageThread 0 encountered invalid arguments in function upc all broadcast at line 26 in file /home/jjc/ex1.upc.
Error: Parameter (sizeof(int ) * sh val) passes non-positive value of 0 to nbytes argument
Variable sh val was declared at line 10 in file /home/jjc/ex1.upc.
Outline▫Understanding a Unified Parallel C▫UPC-CHECK 1.0 tool
How does it work? Usability Error coverage and quality of error reports generated Testing Overheads Scalability Known limitations
▫Challenges in argument error detection▫Deadlock detection algorithm▫Demo
Understanding Unified Parallel C
•Shared memory model
•Distributed memory model
Understanding Unified Parallel C•Unified Parallel C
▫Distributed Shared Memory Model or Partitioned Global Address Space Model
UPC-CHECK v1.0•Source to source translator•Pre-compiler•Error handling
▫Argument errors ▫Deadlocks
UPC-CHECK: Usability •Portable
▫Machine independent▫Compiler independent
•Ease of use▫Easy to install
install_UPC-CHECK
▫Easy to run •Freely available
wget http://hpcgroup.public.iastate.edu/UPC-CHECK/UPC-CHECK.tar.gz
UPC-CHECK 1.0: Usability•Usage
upc-check [compiler options] [--upccheck:flag [--upccheck:flag] ...] -c sourcefile.upc
-a|-d_argument_check disables argument checking (enabled by
default)-d|-d_deadlock_check disables deadlock checking (enabled by
default)-s|-e_track_func_call_stack enables tracing of function call stack (disabled by default)-h|--h|-help prints help for UPC-CHECK
• Just replace your compile-command with upc-check.
Quality of error reports generated• Coyle, J., Hoekstra, J., Kraeva, M., Luecke, G. R., Kleiman, R.,
Srinivas, V., Tripathi, A., Weiss, O., Wehe, A., Xu, Y., Yahya, M. (2008). UPC Run-Time Error Detection Test Suite. http://kraeva.public.iastate.edu/rted/UPC.TestPlan.pdf,
Iowa State University, High Performance Computing Group.▫ A score of 5 is given for a detailed error message that will assist a
programmer to x the error.▫ A score of 4 is given for error messages with more information than a
score of 3 and less than 5. This is tailored for each test.▫ A score of 3 is given for error messages with the correct error name,
line number and the name of the file where the error occurred.▫ A score of 2 is given for error messages with the correct error name and
line number where the error occurred but not the file name where the error occurred.
▫ A score of 1 is given for error messages with the correct error name.▫ A score of 0 is given when the error was not detected.
Run-time environments
Argument errors Deadlocks
Cray 0.38 0Berkeley 0.04 0.58HP 0 0.36GNU 0 0.27
UPC-CHECK 4.89 5
UPC-CHECK 1.0: Testing
•400 error test-cases•1800 false-positive cases•Additional testing for deadlocks•Testing across application programs
UPC-CHECK 1.0: Overhead• Base memory requirement
▫~ 128 KB per thread▫With every acquired or requested shared memory
lock, requirement goes by around 256 B ▫while tracking function call stack, with every
level of nested function call, memory requirement goes by around 512 B
• Increase of code section▫~ 100 lines of instrumentation per UPC operation▫~12000 lines from support files
Efficiency overheadBerkeley UPC Cray UPC
Original Instumented Overhead Original Instumented OverheadCG-S 7.329 7.393 1.009 7.34 7.924 1.080CG-W 7.554 7.613 1.008 7.576 8.344 1.101CG-A 8.531 9.378 1.099 8.372 9.133 1.091CG-B 73.619 74.222 1.008 56.376 63.239 1.122CG-C 171.36 173.036 1.010 132.997 140.317 1.055EP-S 8.048 8.581 1.066 5.319 5.307 0.998EP-W 8.944 10.179 1.138 6.039 6.019 0.997EP-A 19.71 25.193 1.278 14.755 14.743 0.999EP-B 57.366 92.385 1.610 44.706 46.567 1.042EP-C 211.214 349.248 1.654 164.289 163.929 0.998FT-S 7.529 7.74 1.028 4.97 4.918 0.990FT-W+ 7.651 7.68 1.004 5.135 5.151 1.003FT-A*+ 15.34 14.312 0.933 9.173 9.084 0.990FT-B*+ 83.981 77.339 0.921 50.621 50.613 1.000FT-C*+ 0.000 200.947 220.111 1.095IS-S 7.257 7.389 1.018 4.954 4.894 0.988IS-W 7.409 7.441 1.004 5.099 5.006 0.982IS-A 8.526 8.435 0.989 5.787 5.799 1.002IS-B 12.038 12.115 1.006 8.604 8.54 0.993IS-C*+ 25.397 25.69 1.012 19.662 21.655 1.101MG-S 7.298 7.45 1.021 4.798 4.79 0.998MG-W 7.631 7.499 0.983 5.239 5.815 1.110MG-A*+ 11.32 10.73 0.948 6.979 12.038 1.725MG-B*+ 19.083 19.1 1.001 11.718 16.88 1.441MG-C*+ 118.651 118.965 1.003 68.33 107.597 1.575
Maximum 1.654 Maximum 1.725Average 1.073 Average 1.099
Maximum 1.654
Average 1.073
Maximum 1.725
Average 1.099
UPC-CHECK 1.0: ScalabilityOriginal Instrumented Slowdown
CG-S 4.742 4.942 1.042CG-W 15.664 15.708 1.003CG-A 4.912 4.99 1.016CG-B 54.183 54.239 1.001CG-C 58.309 58.281 1.000EP-S 1.145 1.145 1.000EP-W 6.247 6.243 0.999EP-A 1.417 1.427 1.007EP-B 7.116 7.128 1.002EP-C 11.19 11.17 0.998FT-S DNR DNR 0.000FT-W DNR DNR 0.000FT-A DNR DNR 0.000FT-B 15.528 15.556 1.002FT-C* 22.855 22.735 0.995IS-S 3.541 3.594 1.015IS-W 10.422 10.961 1.052IS-A 3.56 3.658 1.028IS-B 8.752 8.776 1.003IS-C 10.089 10.073 0.998MG-S DNR DNR 0.000MG-W 8.288 8.308 1.002MG-A DNR DNR 0.000MG-B 9.293 9.341 1.005MG-C 13.551 13.579 1.002
Maximum 1.052Average 1.008
Maximum 1.052
Average 1.008
• CROW cluster• Cray compiler• Cray run-time environment• 128 threads
UPC-CHECK v1.0: Known limitations• UPC-CHECK will not test the single-valued
requirement of upc forall statements.• Since UPC-CHECK works on UPC source
programs, it will be unable to handle any deadlocks which are created in a library that a user might be using.
• UPC-CHECK should not be used for programs where the ‘main' function lies within a headerfile▫Best effort will be made, but may lead to memory
leaks at end of execution.
Challenges in checking argument errors•Engineering challenges
▫Exhaustiveness▫Argument checks against multiple
functions▫Handling vector arguments▫Dependency of one argument on another
argument▫Data-structures used ▫Displaying the errors
A novel Deadlock Detection Algorithm•Dynamic•Optimal
▫O(1) for deadlocks created by collective routines
▫O(n) for deadlocks created by locks•Distributed•Scalable
A few more terms:“collective” operations
• “Collective” is a constraint placed on some language operations which requires evaluation of such operations to be matched across all threads. The behavior of collective operations is undefined unless all threads execute the same sequence of collective operations.
• “Single valued” refers to an operand to a collective operation, which has the same value on every thread. The behavior of the operation is otherwise undefined.
Central idea•The collective requirement simply states a
relative ordering property of calls to collective operations that must be maintained in the parallel execution trace for all executions of any legal program.
time
threads
Deadlocks in UPC1. Not all threads are waiting at the same collective routine
time
threads
0 1 2 … i T-1… T-2j …
2. Some threads are waiting at the same collective routine when at least one of the threads has reached end-of-execution
time
threads
0 1 2 … i T-1… T-2j …
End-of-execution
3. One of the threads at a collective routine is holding a lock that at least
one of the threads are trying to acquire.
time
threads
0 1 2 … i T-1… T-2j …
5. Circular dependency for acquiring locks amongst threads
time
threads
0 1 2 … i T-1… T-2j …
Definition: A thread i is dependent on another thread j if the thread i is trying to acquire a lock held by thread j
6. Chain of dependency for acquiring locks leads to a thread which is
waiting at a collective routine.
time
threads
0 1 2 … i T-1… T-2j …
6. Chain of dependency for acquiring locks leads to a thread which is
reached end of execution.
time
threads
0 1 2 … i T-1… T-2j …
End-of-execution
Algorithm: Get all the threads in the picture
i+2
13 i-1 i
jT-3T-2
0
T-1
2i+1
… …
…
Validation method: A basic block
time
threads
i-1
i
Rtim
ethreads
i-1
i
R
Implementation: Algorithm 1shared [1] deadlock_ctxt_t unified_deadlock_ctxt[THREADS];
0 1 … i-1 i i+1 … T-1… nk nk nk …… un un un …
statedesired_stat
e
i-1 i i+1
0 1 … i-1 i i+1 … T-1… nk …… un un un …
0 1 … i-1 I i+1 … T-1… nk nk …… un un un …
0 1 … i-1 i i+1 … T-1… …… un un un …
shared [1] deadlock_ctxt_t unified_deadlock_ctxt[THREADS];
statedesired_stat
e
i-1 i i+1
0 1 … i-1 i i+1 … T-1… nk nk …… un un …
shared [1] deadlock_ctxt_t unified_deadlock_ctxt[THREADS];
statedesired_stat
e
i-1 i i+1
0 1 … i-1 i i+1 … T-1… nk …… un un un …
shared [1] deadlock_ctxt_t unified_deadlock_ctxt[THREADS];
statedesired_stat
e
i-1 i i+1
0 1 … i-1 i i+1 … T-1… nk …… un un …
shared [1] deadlock_ctxt_t unified_deadlock_ctxt[THREADS];
statedesired_stat
e
i-1 i i+1
0 1 … i-1 i i+1 … T-1… …… un un …
shared [1] deadlock_ctxt_t unified_deadlock_ctxt[THREADS];
statedesired_stat
e
i-1 i i+1
0 1 … i-1 i i+1 … T-1… …… un …
Atomicity and serialization of status checks•One centralized lock solution
▫Efficiency hit – complete serialization•Decentralized lock solution –one lock per
thread▫ shared [1] upc_lock_t upc_check_deadlock_detection_lock[THREADS];
…i i+1
… T-3 T-2 T-10 1 2
Avoiding deadlocks created by the checks
…i i+1
… T-3 T-2 T-10 1 2
Scheme 1 of acquiring locks
…i i+1
… T-3 T-2 T-10 1 2
Legend: : First lock acquired : Second lock acquired
Even thread: lock[i] then lock[(i+1) %THREADS]Odd thread: lock[(i+1) %THREADS] then lock[i]
Scheme 1: Maximum latency of acquiring locks for even number of threads
i-2 i-1 i i+1
1 2 2 1 1 2 2 1
i-1 i i+1 i+2
1 2 2 1 1 2 2 1
Longest dependency chains when i is even
Longest dependency chains when i is odd
Maximum latency is 3 or O(1)
Maximum latency: when total number of threads are odd
Maximum latency is 4 or O(1)
Efficiency •The number of threads for which any
thread has to wait before entering its critical section is is O(1).
•The number of remote memory access is O(1) as any thread i only accesses memory related to the state of only thread I and thread (i+1)%THREADS.
•Optimal!
When thread reaches a upc_lock• Track requested locks and acquired locks• Look out cyclical hold-and-wait conditions• Look out for chain of hold-and-wait conditions
which lead to a thread blocked at a collective routine▫ If a thread has reached a collective routine, check if
there is a request for a lock that the thread is holding
• Look out for chain of hold-and-wait conditions which lead to a thread which has reached end-of-execution▫ If a thread is exiting without freeing all locks held by it,
then check if there is a request for a lock that the thread is holding
Papers1. Coyle, J., Hoekstra, J., Kraeva, M., Luecke, G. R., Kleiman, R.,
Roy, I. (2009). UPC Compile-Time Error Detection Test Suite. http://kraeva.public.iastate.edu/rted/UPCct.TestPlan.pdf, Iowa State University High Performance Computing Group.
2. Roy, I., Luecke, G. R., Coyle, J., Kraeva, M., Hoekstra, J. (2011). UPC-CHECK: A run-time error detection tool for programs written in UPC. Preprint
3. Roy, I., Luecke, G. R., Coyle, J., Kraeva, M., Hoekstra, J. (2011). An O(1) algorithm to detect deadlocks in collective routines in the distributed shared memory model. Preprint
Thank you