Testing Concurrent Programs Program Testing and Analysissoftware-lab.org/.../Teaching/Winter_2015/PTA/lecture08.pdf · 2015-12-07 · 5 Why Bother with Concurrency? The free lunch

1

Program Testing and Analysis:

Testing Concurrent Programs

Dr. Michael Pradel

Software Lab, TU Darmstadt

2

Warm-up Quiz

var a = (0.1 + 0.2) + 0.3;var b = 0.1 + (0.2 + 0.3);console.log(a === b);

What does the following code print?

Something elsefalsetrue

2

Warm-up Quiz




2

Warm-up Quiz




Floating point numbers are representedwith finite precision(not only in JavaScript)

2

Warm-up Quiz




0.30000000000000004(due to rounding)

3

Mid-term

� Difficulty ≈What to expect for finalexam

� Exam and results will be madeavailable

� Results will be send out this week� Some students: Very good results� Other students: Have some work to do

3

Mid-term

4

Outline

1. Introduction

2. Dynamic Data Race Detection

3. Testing Thread-Safe Classes

4. Exploring Interleavings

Mostly based on these papers:

� Eraser: A Dynamic Data Race Detector for MultithreadedPrograms, Savage et al., ACM TOCS, 1997

� Fully Automatic and Precise Detection of Thread SafetyViolations, Pradel and Gross, PLDI 2012

� Finding and Reproducing Heisenbugs in ConcurrentPrograms, Musuvathi et al., USENIX 2008

5

Why Bother with Concurrency?

� The free lunch provided by Moore’s law is over� CPU clock speeds stopped to increase around 2005� Instead, multi-core processors became mainstream� Need concurrent programs to make full use of the

hardware

� Many real-world problems are inherentlyconcurrent, e.g.,� Servers must handle multiple concurrent requests� Computations done on huge data often are

”embarrasingly parallel”

5



hardware



5



hardware



6

Concurrency Styles

� Message-passing� Popular for large-scale scientific computing, e.g.,

MPI (message-passing interface)� Used in actor concurrency model, e.g., popular in

Erlang and Scala� No shared memory (ideally), all communication via

messages

� Thread-based, shared memory� Multiple concurrently executing threads� All threads access the same shared memory� Synchronize via locks and barriers

6

Concurrency Styles

� Message-passing� Popular for large-scale scientific computing, e.g.,

MPI (message-passing interface)� Used in actor concurrency model, e.g., popular in

Erlang and Scala� No shared memory (ideally), all communication via

messages

� Thread-based, shared memory� Multiple concurrently executing threads� All threads access the same shared memory� Synchronize via locks and barriers

Focus of this lecture

7

Example

int a = 0, b = 0;

boolean r = false, t = false;

a = 1;

r = true;

t = r;

b = a;

Thread 1 Thread 2

What does this program mean?

→ Behavior depends on threadinterleaving

1

9

Sequential Consistency

Assumption made here:Programs execute under sequential consistency

� Program order is preserved: Each thread’sinstructions execute in the specified order

� Shared memory behaves like a global array:Reads and writes are done immediately

� We assume sequential consistency for the rest ofthe lecture

� Many real-world platforms provide more complexsemantics (”memory models”)

10

What Can Go Wrong?

Common source of errors: Data races

� Two accesses to the same shared memorylocation

� At least one is a write

� Ordering of accesses is non-deterministic

11

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

11

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

Sharedmemorylocation

Read

Write

11

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

3 races

11

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

Quiz: What values can balance

have after executing this code?

11

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

Possible outcomes:balance may be 3, 8, and 15

But: Only 8 is correct

12

Avoiding Data Races

Use locks to ensure that accesses toshared memory do not interfere

int balance = 10;

acquire(L);

int tmp1 = balance;

balance = tmp1 + 5;

release(L);

acquire(L);

int tmp2 = balance;

balance = tmp2 - 7;

release(L);

Thread 1 Thread 2

12

Avoiding Data Races


int balance = 10;

acquire(L);

int tmp1 = balance;

balance = tmp1 + 5;

release(L);

acquire(L);

int tmp2 = balance;

balance = tmp2 - 7;

release(L);

Thread 1 Thread 2

Same lock⇒ Mutuallyexclusive critical sections

12

Avoiding Data Races


int balance = 10;Thread 1 Thread 2

synchronized (L) {

int tmp1 = balance;

balance = tmp1 + 5;

}

synchronized (L) {

int tmp2 = balance;

balance = tmp2 - 7;

}

(Java syntax)

13

Outline

1. Introduction








14

Eraser: Data Race Detection

� Basic idea: Look for ”unprotected” accesses toshared memory

� Assumption: All accesses to a shared memorylocation v should happen while holding the samelock L

→ Consistent locking discipline

� Dynamic analysis that monitors all lockacquisitions, lock releases, and accesses foshared memory locations

15

Lockset Algorithm (Simple Form)

� Let locksHeld(t) be the set of locksheld by thread t

� For each shared memory location v,initialize C(v) to the set of all locks

� On each access to v by thread t� Set C(v) := C(v) ∩ locksHeld(t)

� If C(v) = ∅, issue a warning

15

Lockset Algorithm (Simple Form)

� Let locksHeld(t) be the set of locksheld by thread t

� For each shared memory location v,initialize C(v) to the set of all locks

� On each access to v by thread t� Set C(v) := C(v) ∩ locksHeld(t)

� If C(v) = ∅, issue a warning

Lockset

Lockset refinement

2

17

Simple Lockset is Too Strict

Simple lockset algorithm produces falsepositives for� variables initialized without locks held

� read-shared data read without locks held

� read-write locking mechanisms(producer-consumer style)

18

Refining the Lockset Algorithm

Virgin

ExclusiveShared-modified

Shared

� Keep state of each shared memory location� Issue warnings only in the Shared-modified

state

wrrd/wr by1st thread

wr by 2nd thread

rd by 2ndthread

rd

wr

19

Summary: Eraser

� Dynamic analysis to detect data races

� Assumes consistent locking discipline

� Limitations� May report false positives when locks are

acquired inconsistently but correctly

� May miss data races because it does notconsider all possible interleavings

20

Outline

1. Introduction








21

Thread Safety

� Popular way to encapsulate the challenges ofconcurrent programming: Thread-safe classes

� Class ensures correct synchronization

� Clients can use instances as if they were alone

� Rest of program can treat implementation ofthread-safe class as a blackbox

22

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code” page 18

22

Thread Safety (2)


22

Thread Safety (2)


22

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”

“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”

page 18

StringBuffer API documentation, JDK 6

22

Thread Safety (2)

“behaves correctly when accessedfrom multiple threads ... with noadditional synchronization ... (inthe) calling code”

“operations ... behave as if they occurin some serial order that is consistentwith the order of the method callsmade by each of the individualthreads”

page 18

StringBuffer API documentation, JDK 6

23

Example from JDK

StringBuffer b = new StringBuffer()

b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

23

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

Quiz: What can be the content of b ifStringBuffer is thread-safe?

23

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

23

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

23

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

23

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

23

Example from JDK


b.append("a")

b.append("b")

b.append("c")

Thread 1 Thread 2

"abc" 3 "cab" 3 "acb" 3 "ac" 7 "bac" 7

24

Testing Thread-Safe Classes

� Correctness of program relies on thread safety ofspecific classes

� But: What if the class is actually not thread-safe?

� ConTeGe = Concurrent Test Generator

� Creates multi-threaded unit tests

� Detects thread safety violations by comparingconcurrent behavior against linearizations

25

Example Bug from JDK


b.append("abc")

b.insert(1, b) b.deleteCharAt(1)

Thread 1 Thread 2

25

Example Bug from JDK


b.append("abc")


Thread 1 Thread 2

IndexOutOfBoundsException

Confirmed as bug: Issue #7100996

!

26

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle

Generate aconcurrent test

26

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


26

ConTeGe

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


27

Generating Concurrent Tests


b.append("abc")


Thread 1 Thread 2

Example:

27



b.append("abc")


Thread 1 Thread 2

Sequential prefix:

Create and set upCUT instance

Example:

27



b.append("abc")


Thread 1 Thread 2

Concurrent suffixes:

Use shared CUTinstance

Example:

28

Test Generation Algorithm

1. Create prefix� Instantiate CUT

� Call methods

2. Create suffixes for prefix� Call methods on shared CUT instance

3. Prefix + two suffixes = test

Selection of methods similar tofeedback-directed test generation

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect aconstructor

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect aconstructor

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

After adding a call:Execute

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


3

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

Randomlyselect amethod

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


b.append(/* String */)

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

b.append(/* String */)

Arguments:a) Take available objectb) Call method returning

required typec) Random value

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2



29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2


3

29

Creating a Prefix


� Call methods


b.append("abc")


Thread 1 Thread 2

30

Creating Suffixes

2. Create suffixesfor prefix

� Call methods onshared CUT instance

30

Creating Suffixes




b.append("abc")

b.insert(1, b)

30

Creating Suffixes




b.append("abc")

b.insert(1, b)


30

Creating Suffixes




b.append("abc")

b.insert(1, b)


b.insert(/* int */, /* CharSequence */)

30

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(/* int */, /* CharSequence */)



30

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(-5, b)



30

Creating Suffixes




b.append("abc")



30

Creating Suffixes




b.append("abc")



!

30

Creating Suffixes




b.append("abc")

b.insert(1, b)b.insert(/* int */, /* CharSequence */)



30

Creating Suffixes




b.append("abc")

b.insert(1, b)



30

Creating Suffixes




b.append("abc")

b.insert(1, b)


30

Creating Suffixes




b.append("abc")

b.insert(1, b)


3

30

Creating Suffixes




b.append("abc")

b.insert(1, b)

30

Creating Suffixes




b.append("abc")


30

Creating Suffixes




b.append("abc")



30

Creating Suffixes




b.append("abc")



3

30

Creating Suffixes




b.append("abc")


31

Creating a Test


31

Creating a Test



b.append("abc")


31

Creating a Test



b.append("abc")


Thread 1 Thread 2

Spawn new threadfor each suffix

32

Approach

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


32

Approach

Bug

Classundertest(CUT)

Execute

Thread safetyoracle


33

Thread Safety Oracle

Does the test executionexpose a thread safetyviolation?

� Focus on exceptionsand deadlocks

� Compare concurrentexecution tolinearizations

34

Assumptions

Concurrency-only crashes are undesired

� Matches definition of thread safety

Control over all input to tests

� Sequential execution: Deterministic

35

Linearizations

� Put all calls into one thread� Preserve order of calls within a thread

35

Linearizations


21 3

35

Linearizations

213

2

13

21

3


21 3

36

The Oracle

Exception ordeadlock?

Execute concurrently

No

Yes

Yes

No Thread safetyviolation

Samefailure?

Execute linearization All linearizationschecked

3

3

36

The Oracle



No

Yes

Yes


Samefailure?



3

3

36

The Oracle



No

Yes

Yes


Samefailure?


Exception ordeadlock? 3

3

36

The Oracle



No

Yes

Yes


Samefailure?


Execute linearization

3

3

36

The Oracle



No

Yes

Yes


Samefailure?


Samefailure?

3

3

36

The Oracle



No

Yes

Yes


Samefailure?


Execute linearization

3

3

36

The Oracle



No

Yes

Yes


Samefailure?


Samefailure?

3

3

36

The Oracle



No

Yes

Yes


Samefailure?


3

3

All linearizationschecked

36

The Oracle



No

Yes

Yes


Samefailure?


Thread safetyviolation

3

3

37

Example


b.append("abc")


Thread 1 Thread 2

37

Example


b.append("abc")


Thread 1 Thread 2

!

37

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

!

37

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

StringBuffer b = ..

b.append("abc")

b.deleteCharAt(1)

b.insert(1, b) 3

!

37

Example


b.append("abc")


StringBuffer b = ..

b.append("abc")

b.insert(1, b)

b.deleteCharAt(1) 3

Thread 1 Thread 2

StringBuffer b = ..

b.append("abc")

b.deleteCharAt(1)

b.insert(1, b) 3

!Thread safety violation

38

Properties of the Oracle

Sound but incomplete *

� All reported violations are real� Cannot guarantee thread safety

Independent of bug type� Data races� Atomicity violations� Deadlocks

* with respect to incorrectness

39

Implementation & Results

� Implemented for Java classes

� Applied to popular thread-safe classesfrom JDK, Apache libraries, etc.

� Found 15 concurrency bugs, includingpreviously unknown problems in JDK

� Takes between several seconds andseveral hours (worst case: 19 hours)

40

Open Challenges

� How to generate tests that are likely to triggerbugs? (Currently: random decisions)

� Static analysis to find potential bugs; focus on thoseduring test generation

� Use feedback from test execution to steer testgeneration towards not yet explored behavior

� How to generate tests for larger pieces ofconcurrent software, e.g., entire libraries orprograms?

40

Open Challenges

� How to generate tests that are likely to triggerbugs? (Currently: random decisions)

� Static analysis to find potential bugs; focus on thoseduring test generation

� Use feedback from test execution to steer testgeneration towards not yet explored behavior

� How to generate tests for larger pieces ofconcurrent software, e.g., entire libraries orprograms?

Hint: Opportunities for master theses

41

Outline

1. Introduction








42

Scheduling Non-Determinism

� A single program executed with a single inputmay have many different interleavings

� Scheduler decides interleavingsnon-deterministically

� Some interleavings may expose bugs, othersexecute correctly (”Heisenbugs”)

� Challenge: How to explore different interleavings?How to detect buggy interleavings?

43

CHESS in a Nutshell

� A user mode scheduler that controlsall scheduling non-determinism

� Guarantees:� Every program run takes a new thread

interleaving� Can reproduce the interleaving for every run

� Systematic but non-exhaustiveexploration of the set of possibleinterleavings

44

Tree of Interleavings

� Search space of possibleinterleavings: Represent as a tree

� Node = points of scheduling decision

� Edge = decisions taken

� Each path = one possible schedule

45

Example

// bank account

int balance = 10;

// deposit money

int tmp1 = balance;

balance = tmp1 + 5;

// withdraw money

int tmp2 = balance;

balance = tmp2 - 7;

Thread 1 Thread 2

3

47

State Space Explosion

Thread 1:

instr. 1instr. 2...instr. k

n threads

k instructions

� Number ofinterleavings: O(nn·k)

� Exponential in both n

and k

� Typically: n < 10,k > 100

� Exploring allinterleavings does notscale to largeprograms (i.e., large k)

Thread 2:

instr. 1instr. 2...instr. k

48

Preemption Bounding

� Limit exploration to schedules with a smallnumber c of preemptions� Preemption = Context switches forced by the

scheduler

� Number of schedules: O((n2 · k)c · n!)� Exponential in c and n, but not in k

� Based on empirical observation: Mostconcurrency bugs can be triggered with few (< 2)interleavings

49

Implementation and Results

� Implemented via binary instrumentation

� Applied to eight mid-size and large systems (upto 175K lines of code),

� Found a total of 27 bugs

� Major benefit over stress testing: Once a failure isdetected, can easily reproduce and debug it

50

Summary

� Concurrent programming is inevitable

� Writing correct concurrent programsis hard

� Techniques to detect concurrencybugs� Dynamic data race detection� Test generation and thread safety checking� Systematic exploration of interleavings

Documents

Testing Concurrent Programs Program Testing and Analysissoftware-lab.org/.../Teaching/Winter_2015/PTA/lecture08.pdf · 2015-12-07 · 5 Why Bother with Concurrency? The free lunch