(Part 2) Testing Concurrent Programs Program Testing and ... · Thread Safety (2) “behaves correctly when accessed from multiple threads ... with no additional synchronization

1

Program Testing and Analysis:

Testing Concurrent Programs(Part 2)

Dr. Michael Pradel

Software Lab, TU Darmstadt

2

Warm-up Quiz

var x = 5;var y = Number(5);var z = new Number(5);x.foo = "bar"; y.foo = "bar"; z.foo = "bar";console.log(x.foo);console.log(y.foo);console.log(z.foo);

What does the following code print?

barundefinedbar

undefinedundefinedundefined

barbarbar

Some-thingelse

2

Warm-up Quiz

var x = 5;var y = Number(5);var z = new Number(5);x.foo = "bar"; y.foo = "bar"; z.foo = "bar";console.log(x.foo);console.log(y.foo);console.log(z.foo);

What does the following code print?

barundefinedbar

undefinedundefinedundefined

barbarbar

Some-thingelse

”undefined” (x and y areprimitive values, whichcannot have properties)

”bar” (z is an object)

3

Outline

1. Introduction

2. Dynamic Data Race Detection

3. Testing Thread-Safe Classes

4. Exploring Interleavings

Mostly based on these papers:

� Eraser: A Dynamic Data Race Detector for MultithreadedPrograms, Savage et al., ACM TOCS, 1997

� Fully Automatic and Precise Detection of Thread SafetyViolations, Pradel and Gross, PLDI 2012

� Finding and Reproducing Heisenbugs in ConcurrentPrograms, Musuvathi et al., USENIX 2008

4

Thread Safety

� Popular way to encapsulate the challenges ofconcurrent programming: Thread-safe classes

� Class ensures correct synchronization

� Clients can use instances as if they were alone

� Rest of program can treat implementation ofthread-safe class as a blackbox

5

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code” page 18

5

Thread Safety (2)


5

Thread Safety (2)


5

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”

“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”

page 18

StringBuffer API documentation, JDK 6

5

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”

“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”

page 18

StringBuffer API documentation, JDK 6

6

Example from JDK

StringBuffer b = new StringBuffer()

b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

6

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

Quiz: What can be the content of b ifStringBuffer is thread-safe?

6

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

6

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

6

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

6

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

6

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

7

Testing Thread-Safe Classes

� Correctness of program relies on thread safety ofspecific classes

� But: What if the class is actually not thread-safe?

� ConTeGe = Concurrent Test Generator

� Creates multi-threaded unit tests

� Detects thread safety violations by comparingconcurrent behavior against linearizations

8

Example Bug from JDK


b.append("abc")

b.insert(1, b) b.deleteCharAt(1)

Thread 1 Thread 2

8

Example Bug from JDK


b.append("abc")


Thread 1 Thread 2

IndexOutOfBoundsException

Issue #7100996 in Oracle’s bug tracker

!

9

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle

Generate aconcurrent test

9

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


9

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


10

Generating Concurrent Tests


b.append("abc")


Thread 1 Thread 2

Example:

10



b.append("abc")


Thread 1 Thread 2

Sequential prefix:

Create and set upCUT instance

Example:

10



b.append("abc")


Thread 1 Thread 2

Concurrent suffixes:

Use shared CUTinstance

Example:

11

Test Generation Algorithm

1. Create prefix� Instantiate CUT

� Call methods

2. Create suffixes for prefix� Call methods on shared CUT instance

3. Prefix + two suffixes = test

Selection of methods similar tofeedback-directed test generation

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect aconstructor

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect aconstructor

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

After adding a call:Execute

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


3

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect amethod

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


b.append(/* String */)

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

b.append(/* String */)

Arguments:a) Take available objectb) Call method returning

required typec) Random value

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2



12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


3

12

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

13

Creating Suffixes

2. Create suffixesfor prefix

� Call methods onshared CUT instance

13

Creating Suffixes




b.append("abc")

b.insert(1, b)

13

Creating Suffixes




b.append("abc")

b.insert(1, b)


13

Creating Suffixes




b.append("abc")

b.insert(1, b)


b.insert(/* int */, /* CharSequence */)

13

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(/* int */, /* CharSequence */)



13

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(-5, b)



13

Creating Suffixes




b.append("abc")



13

Creating Suffixes




b.append("abc")



!

13

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(/* int */, /* CharSequence */)



13

Creating Suffixes




b.append("abc")

b.insert(1, b)



13

Creating Suffixes




b.append("abc")

b.insert(1, b)


13

Creating Suffixes




b.append("abc")

b.insert(1, b)


3

13

Creating Suffixes




b.append("abc")

b.insert(1, b)

13

Creating Suffixes




b.append("abc")


13

Creating Suffixes




b.append("abc")



13

Creating Suffixes




b.append("abc")



3

13

Creating Suffixes




b.append("abc")


14

Creating a Test


14

Creating a Test



b.append("abc")


14

Creating a Test



b.append("abc")


Thread 1 Thread 2

Spawn new threadfor each suffix

15

Approach

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


15

Approach

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


16

Thread Safety Oracle

Does the test executionexpose a thread safetyviolation?

� Focus on exceptionsand deadlocks

� Compare concurrentexecution tolinearizations

17

Linearizations

� Put all calls into one thread� Preserve order of calls within a thread

17

Linearizations


21 3

17

Linearizations

213

2

13

21

3


21 3

18

The Oracle

Exception ordeadlock?

Execute concurrently

No

Yes

Yes

No Thread safetyviolation

Samefailure?

Execute linearization All linearizationschecked

3

3

18

The Oracle



No

Yes

Yes


Samefailure?



3

3

18

The Oracle



No

Yes

Yes


Samefailure?


Exception ordeadlock? 3

3

18

The Oracle



No

Yes

Yes


Samefailure?


Execute linearization

3

3

18

The Oracle



No

Yes

Yes


Samefailure?


Samefailure?

3

3

18

The Oracle



No

Yes

Yes


Samefailure?


Execute linearization

3

3

18

The Oracle



No

Yes

Yes


Samefailure?


Samefailure?

3

3

18

The Oracle



No

Yes

Yes


Samefailure?


3

3

All linearizationschecked

18

The Oracle



No

Yes

Yes


Samefailure?


Thread safetyviolation

3

3

19

Example


b.append("abc")


Thread 1 Thread 2

19

Example


b.append("abc")


Thread 1 Thread 2

!

19

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

!

19

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

StringBuffer b = ..

b.append("abc")

b.deleteCharAt(1)

b.insert(1, b) 3

!

19

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

StringBuffer b = ..

b.append("abc")

b.deleteCharAt(1)

b.insert(1, b) 3

!Thread safety violation

20

Properties of the Oracle

Sound but incomplete *

� All reported violations are real� Cannot guarantee thread safety

Independent of bug type� Data races� Atomicity violations� Deadlocks

* with respect to incorrectness

21

Implementation & Results

� Implemented for Java classes

� Applied to popular thread-safe classesfrom JDK, Apache libraries, etc.

� Found 15 concurrency bugs, includingpreviously unknown problems in JDK

� Takes between several seconds andseveral hours (worst case: 19 hours)

22

Open Challenges

� How to generate tests that are likely to triggerbugs? (Currently: random decisions)

� Static analysis to find potential bugs; focus on thoseduring test generation

� Use feedback from test execution to steer testgeneration towards not yet explored behavior

� How to generate tests for larger pieces ofconcurrent software, e.g., entire libraries orprograms?

22

Open Challenges

� How to generate tests that are likely to triggerbugs? (Currently: random decisions)

� Static analysis to find potential bugs; focus on thoseduring test generation

� Use feedback from test execution to steer testgeneration towards not yet explored behavior

� How to generate tests for larger pieces ofconcurrent software, e.g., entire libraries orprograms?

Hint: Opportunities for master theses

23

Outline

1. Introduction

2. Dynamic Data Race Detection

3. Testing Thread-Safe Classes

4. Exploring Interleavings

Mostly based on these papers:

� Eraser: A Dynamic Data Race Detector for MultithreadedPrograms, Savage et al., ACM TOCS, 1997

� Fully Automatic and Precise Detection of Thread SafetyViolations, Pradel and Gross, PLDI 2012

� Finding and Reproducing Heisenbugs in ConcurrentPrograms, Musuvathi et al., USENIX 2008

24

Scheduling Non-Determinism

� A single program executed with a single inputmay have many different interleavings

� Scheduler decides interleavingsnon-deterministically

� Some interleavings may expose bugs, othersexecute correctly (”Heisenbugs”)

� Challenge: How to explore different interleavings?How to detect buggy interleavings?

25

CHESS in a Nutshell

� A user mode scheduler that controlsall scheduling non-determinism

� Guarantees:� Every program run takes a new thread

interleaving� Can reproduce the interleaving for every run

� Systematic but non-exhaustiveexploration of the set of possibleinterleavings

26

Tree of Interleavings

� Search space of possibleinterleavings: Represent as a tree

� Nodes = points of scheduling decision

� Edges = decisions taken

� Each path = one possible schedule

27

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

1

29

State Space Explosion

Thread 1:

instr. 1instr. 2...instr. k

n threads

k instructions

� Number ofinterleavings: O(nn·k)

� Exponential in both n

and k

� Typically:n < 10, k > 100

� Exploring allinterleavings does notscale to largeprograms (i.e., large k)

Thread 2:

instr. 1instr. 2...instr. k

30

Preemption Bounding

� Limit exploration to schedules with a smallnumber c of preemptions� Preemption = Context switches forced by the

scheduler

� Number of schedules: O((n2 · k)c · n!)� Exponential in c and n, but not in k

� Based on empirical observation: Mostconcurrency bugs can be triggered with few (< 2)preemptions

31

Implementation and Results

� Implemented via binary instrumentation

� Applied to eight mid-size and large systems (upto 175K lines of code),

� Found a total of 27 bugs

� Major benefit over stress testing: Once a failure isdetected, can easily reproduce and debug it

32

Summary

� Concurrent programming is inevitable

� Writing correct concurrent programsis hard

� Techniques to detect concurrencybugs� Dynamic data race detection� Test generation and thread safety checking� Systematic exploration of interleavings

Documents

(Part 2) Testing Concurrent Programs Program Testing and ... · Thread Safety (2) “behaves correctly when accessed from multiple threads ... with no additional synchronization