IBM Haifa Research Lab

IBM Haifa Labs

An Open Source SimulationModel of Software Testing

Shmuel Ur

Elad Yom-Tov

Paul Wernick

http://qp.research.ibm.com/concurrency_testing

2

IBM Haifa Labs

What Are the Development Tasks?

Writing Code Creating lines of code, creating bugs

Testing Creating tests, running tests

Reviews Searching the code for bugs

Debugging Looking for the bug root cause

Fixing Replacing some code that has a bug

Putting new bugs in?

We will ignore specification, design, maintenance

3

IBM Haifa Labs

Simulation Program Development

A program of certain size with certain quality need to be createdSize is measured in LOC

Quality is measured in # of bugs

A developer may spend time doing of the development tasks, The task may result in artifacts being createdWriting code create new line objects, one for each line, which may

contain bugs

Writing tests, results in tests each of which can test a subset of the line objects. Automated tests are more expensive.

Running a test takes some time (depends if automated) and may find bugs in specific line objects. Running a test a second time is not as effective

…

4

IBM Haifa Labs

Default Simulation Values

# Code lines per hour = 30

Bugs per line = 0.01

Debugging time Correction = 1 + (LOC since bug)/2000

Hours to fix bug = base 2, multiply by 2 if a month passed, multiply by 3 if two months passed

Probability discovered by inspection = 0.5 : Laitenberger and DeBaud suggest that 70% is achievable

# Lines reviewed per hour = 200

Cost of testing = 1.14 hour per one hundred lines to do unit testing Other measures exist, cost of creating tests, cost of running tests

Probability of finding bugs in test = .5 Probability of finding bug when re-running the test = 0

5

IBM Haifa Labs

Simulation Cycle

Write Test

Write code

Run Test

Check

Size

(b) Code building cycle: Test First

Done

Debug

Test /inspect

Debug

Check

Quality

(c) System test

Done

Write code

Test / Inspect

Debug

Check

Size

(a) Code building cycle : iterative

Done

6

IBM Haifa Labs

Sample Simulation

7

IBM Haifa Labs

Sample Simulation

8

IBM Haifa Labs

Overnight Regression – Chaotic behavior

9

IBM Haifa Labs

Overall Productivity when Code Creation Rate Increase

10

IBM Haifa Labs

What do Developers Really Do?

11

IBM Haifa Labs

Impact of Length of Iteration in Iterative Software Development

12

IBM Haifa Labs

Is Pair Programming a Good Idea? Depends!!

13

IBM Haifa Labs

Automating Testing May be Overrated

14

IBM Haifa Labs

Future Direction

Encapsulation

Reduce cost of debugging, review …

Code complexity

Planning tool

Quality requirement impact on methodology

Not one size fit all