View
1
Download
0
Category
Preview:
Citation preview
1
Program Testing and Analysis:
Testing Concurrent Programs(Part 2)
Dr. Michael Pradel
Software Lab, TU Darmstadt
2
Warm-up Quiz
var x = 5;var y = Number(5);var z = new Number(5);x.foo = "bar"; y.foo = "bar"; z.foo = "bar";console.log(x.foo);console.log(y.foo);console.log(z.foo);
What does the following code print?
barundefinedbar
undefinedundefinedundefined
barbarbar
Some-thingelse
2
Warm-up Quiz
var x = 5;var y = Number(5);var z = new Number(5);x.foo = "bar"; y.foo = "bar"; z.foo = "bar";console.log(x.foo);console.log(y.foo);console.log(z.foo);
What does the following code print?
barundefinedbar
undefinedundefinedundefined
barbarbar
Some-thingelse
”undefined” (x and y areprimitive values, whichcannot have properties)
”bar” (z is an object)
3
Outline
1. Introduction
2. Dynamic Data Race Detection
3. Testing Thread-Safe Classes
4. Exploring Interleavings
Mostly based on these papers:
� Eraser: A Dynamic Data Race Detector for MultithreadedPrograms, Savage et al., ACM TOCS, 1997
� Fully Automatic and Precise Detection of Thread SafetyViolations, Pradel and Gross, PLDI 2012
� Finding and Reproducing Heisenbugs in ConcurrentPrograms, Musuvathi et al., USENIX 2008
4
Thread Safety
� Popular way to encapsulate the challenges ofconcurrent programming: Thread-safe classes
� Class ensures correct synchronization
� Clients can use instances as if they were alone
� Rest of program can treat implementation ofthread-safe class as a blackbox
5
Thread Safety (2)
“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code” page 18
5
Thread Safety (2)
“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code” page 18
5
Thread Safety (2)
“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code” page 18
5
Thread Safety (2)
“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”
“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”
page 18
StringBuffer API documentation, JDK 6
5
Thread Safety (2)
“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”
“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”
page 18
StringBuffer API documentation, JDK 6
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
Quiz: What can be the content of b ifStringBuffer is thread-safe?
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7
6
Example from JDK
StringBuffer b = new StringBuffer()
b.append("a")
b.append("b")
b.append("c")
Thread 1 Thread 2
"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7
7
Testing Thread-Safe Classes
� Correctness of program relies on thread safety ofspecific classes
� But: What if the class is actually not thread-safe?
� ConTeGe = Concurrent Test Generator
� Creates multi-threaded unit tests
� Detects thread safety violations by comparingconcurrent behavior against linearizations
8
Example Bug from JDK
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
8
Example Bug from JDK
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
IndexOutOfBoundsException
Issue #7100996 in Oracle’s bug tracker
!
9
ConTeGe
Bug
Classundertest(CUT)
Execute
Thread safetyoracle
Generate aconcurrent test
9
ConTeGe
Bug
Classundertest(CUT)
Execute
Thread safetyoracle
Generate aconcurrent test
9
ConTeGe
Bug
Classundertest(CUT)
Execute
Thread safetyoracle
Generate aconcurrent test
10
Generating Concurrent Tests
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Example:
10
Generating Concurrent Tests
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Sequential prefix:
Create and set upCUT instance
Example:
10
Generating Concurrent Tests
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Concurrent suffixes:
Use shared CUTinstance
Example:
11
Test Generation Algorithm
1. Create prefix� Instantiate CUT
� Call methods
2. Create suffixes for prefix� Call methods on shared CUT instance
3. Prefix + two suffixes = test
Selection of methods similar tofeedback-directed test generation
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Randomlyselect aconstructor
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Randomlyselect aconstructor
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
After adding a call:Execute
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
After adding a call:Execute
3
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Randomlyselect amethod
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Randomlyselect amethod
b.append(/* String */)
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
b.append(/* String */)
Arguments:a) Take available objectb) Call method returning
required typec) Random value
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Arguments:a) Take available objectb) Call method returning
required typec) Random value
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
After adding a call:Execute
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
After adding a call:Execute
3
12
Creating a Prefix
1. Create prefix� Instantiate CUT
� Call methods
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
Randomlyselect amethod
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
Randomlyselect amethod
b.insert(/* int */, /* CharSequence */)
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)b.insert(/* int */, /* CharSequence */)
Arguments:a) Take available objectb) Call method returning
required typec) Random value
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)b.insert(-5, b)
Arguments:a) Take available objectb) Call method returning
required typec) Random value
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)b.insert(-5, b)
After adding a call:Execute
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)b.insert(-5, b)
After adding a call:Execute
!
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)b.insert(/* int */, /* CharSequence */)
Arguments:a) Take available objectb) Call method returning
required typec) Random value
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
Arguments:a) Take available objectb) Call method returning
required typec) Random value
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
After adding a call:Execute
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
After adding a call:Execute
3
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b)
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
After adding a call:Execute
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
After adding a call:Execute
3
13
Creating Suffixes
2. Create suffixesfor prefix
� Call methods onshared CUT instance
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
14
Creating a Test
3. Prefix + two suffixes = test
14
Creating a Test
3. Prefix + two suffixes = test
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
14
Creating a Test
3. Prefix + two suffixes = test
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
Spawn new threadfor each suffix
15
Approach
Bug
Classundertest(CUT)
Execute
Thread safetyoracle
Generate aconcurrent test
15
Approach
Bug
Classundertest(CUT)
Execute
Thread safetyoracle
Generate aconcurrent test
16
Thread Safety Oracle
Does the test executionexpose a thread safetyviolation?
� Focus on exceptionsand deadlocks
� Compare concurrentexecution tolinearizations
17
Linearizations
� Put all calls into one thread� Preserve order of calls within a thread
17
Linearizations
� Put all calls into one thread� Preserve order of calls within a thread
21 3
17
Linearizations
213
2
13
21
3
� Put all calls into one thread� Preserve order of calls within a thread
21 3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Execute concurrently
3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Exception ordeadlock? 3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Execute linearization
3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Samefailure?
3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Execute linearization
3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Samefailure?
3
3
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
3
3
All linearizationschecked
18
The Oracle
Exception ordeadlock?
Execute concurrently
No
Yes
Yes
No Thread safetyviolation
Samefailure?
Execute linearization All linearizationschecked
Thread safetyviolation
3
3
19
Example
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
19
Example
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
Thread 1 Thread 2
!
19
Example
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
StringBuffer b = ..
b.append("abc")
b.insert(1, b)
b.deleteCharAt(1) 3
Thread 1 Thread 2
!
19
Example
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
StringBuffer b = ..
b.append("abc")
b.insert(1, b)
b.deleteCharAt(1) 3
Thread 1 Thread 2
StringBuffer b = ..
b.append("abc")
b.deleteCharAt(1)
b.insert(1, b) 3
!
19
Example
StringBuffer b = new StringBuffer()
b.append("abc")
b.insert(1, b) b.deleteCharAt(1)
StringBuffer b = ..
b.append("abc")
b.insert(1, b)
b.deleteCharAt(1) 3
Thread 1 Thread 2
StringBuffer b = ..
b.append("abc")
b.deleteCharAt(1)
b.insert(1, b) 3
!Thread safety violation
20
Properties of the Oracle
Sound but incomplete *
� All reported violations are real� Cannot guarantee thread safety
Independent of bug type� Data races� Atomicity violations� Deadlocks
* with respect to incorrectness
21
Implementation & Results
� Implemented for Java classes
� Applied to popular thread-safe classesfrom JDK, Apache libraries, etc.
� Found 15 concurrency bugs, includingpreviously unknown problems in JDK
� Takes between several seconds andseveral hours (worst case: 19 hours)
22
Open Challenges
� How to generate tests that are likely to triggerbugs? (Currently: random decisions)
� Static analysis to find potential bugs; focus on thoseduring test generation
� Use feedback from test execution to steer testgeneration towards not yet explored behavior
� How to generate tests for larger pieces ofconcurrent software, e.g., entire libraries orprograms?
22
Open Challenges
� How to generate tests that are likely to triggerbugs? (Currently: random decisions)
� Static analysis to find potential bugs; focus on thoseduring test generation
� Use feedback from test execution to steer testgeneration towards not yet explored behavior
� How to generate tests for larger pieces ofconcurrent software, e.g., entire libraries orprograms?
Hint: Opportunities for master theses
23
Outline
1. Introduction
2. Dynamic Data Race Detection
3. Testing Thread-Safe Classes
4. Exploring Interleavings
Mostly based on these papers:
� Eraser: A Dynamic Data Race Detector for MultithreadedPrograms, Savage et al., ACM TOCS, 1997
� Fully Automatic and Precise Detection of Thread SafetyViolations, Pradel and Gross, PLDI 2012
� Finding and Reproducing Heisenbugs in ConcurrentPrograms, Musuvathi et al., USENIX 2008
24
Scheduling Non-Determinism
� A single program executed with a single inputmay have many different interleavings
� Scheduler decides interleavingsnon-deterministically
� Some interleavings may expose bugs, othersexecute correctly (”Heisenbugs”)
� Challenge: How to explore different interleavings?How to detect buggy interleavings?
25
CHESS in a Nutshell
� A user mode scheduler that controlsall scheduling non-determinism
� Guarantees:� Every program run takes a new thread
interleaving� Can reproduce the interleaving for every run
� Systematic but non-exhaustiveexploration of the set of possibleinterleavings
26
Tree of Interleavings
� Search space of possibleinterleavings: Represent as a tree
� Nodes = points of scheduling decision
� Edges = decisions taken
� Each path = one possible schedule
27
Example
// bank account
int balance = 10;
// deposit money
int tmp1 = balance;
balance = tmp1 + 5;
// withdraw money
int tmp2 = balance;
balance = tmp2 - 7;
Thread 1 Thread 2
1
29
State Space Explosion
Thread 1:
instr. 1instr. 2...instr. k
n threads
k instructions
� Number ofinterleavings: O(nn·k)
� Exponential in both n
and k
� Typically:n < 10, k > 100
� Exploring allinterleavings does notscale to largeprograms (i.e., large k)
Thread 2:
instr. 1instr. 2...instr. k
30
Preemption Bounding
� Limit exploration to schedules with a smallnumber c of preemptions� Preemption = Context switches forced by the
scheduler
� Number of schedules: O((n2 · k)c · n!)� Exponential in c and n, but not in k
� Based on empirical observation: Mostconcurrency bugs can be triggered with few (< 2)preemptions
31
Implementation and Results
� Implemented via binary instrumentation
� Applied to eight mid-size and large systems (upto 175K lines of code),
� Found a total of 27 bugs
� Major benefit over stress testing: Once a failure isdetected, can easily reproduce and debug it
32
Summary
� Concurrent programming is inevitable
� Writing correct concurrent programsis hard
� Techniques to detect concurrencybugs� Dynamic data race detection� Test generation and thread safety checking� Systematic exploration of interleavings
Recommended