Icse 2011 ds_1

1

Pragmatic Prioritization of Software Quality Assurance Efforts

Emad ShihabQueen’s University

2

Motivation

Software ^ has bugsalways

always

How to allocate quality assurance efforts for legacy and new code?

Q:

managers ^ have limited resources

3

One Answer: Defect Prediction

Many recent studies focused on defect prediction

….However adoption in practice remains low

4

An Industrial Example

How to prioritize the creation of unit tests for a large legacy system

5

Criteria for Pragmatic QA Approaches

Focused Granularity

Unit of prioritization must be focused (e.g., function or method)

Current Approach Desired Approach

File, Folder or subsystem level (Thousands of LOC)

6

Criteria for Pragmatic QA Approaches

Responsive Feedback

Continuous reflection of system changes.

Current Approach Desired Approach

Predict for next release

7

Criteria for Pragmatic QA Approaches

Consider Effort

Consider effort in recommendations (e.g., use size)

Current Approach Desired Approach

No consideration for effort

In addition, need to evaluate the generality of the approaches to understand implications of findings

8

Proposed Pragmatic SQA Approaches

Unit test creation

How to allocate quality assurance efforts for legacy code?Q:

A:

Change risk

How to allocate quality assurance efforts for new code?

Q:

A:

Complete

In Progress

9

Unit Test CreationApproach

1. Search modification record comments for keywords, bug identifiers and modified files

2. Extract source code of modified file(s) and compare to previous version to identify changed functions

3. Combine data from 1 and 2 to calculate function-level heuristics

10

Unit Test CreationHeuristics

Most Frequently Modified (MFM)Most Recently Modified (MRM)

Most Frequently Fixed (MFF)Most Recently Fixed (MRF)

Largest Fixed (LF)Largest Modified (LM)

Change Risk (CR)Size Risk (SR)

Random

Modification

Fix

Size

RiskRandom

11

Unit Test CreationHeuristic Performance

Based on a heuristic, generate list of 10 functions to write unit tests for

Use size of function to measure effort required to write unit test

12

Unit Test Creation Usefulness: Was writing the unit test useful?

Time to write unit testA

B

C

6 bug fixes

2 bug fixes

0 bug fixes

Usefulness = 2/3 = 66.67%

13

Unit Test CreationUsefulness of Heuristics

Random

Most Recently Modified (MRM)

Size Risk (SR)

Change Risk (CR)

Most Recently Fixed (MRF)

Most Frequently Modified (MFM)

Most Frequently Fixed (MFF)

Largest Modified (LM)

Largest Fixed (LF)

0 10 20 30 40 50 60 70 80 90 100

27.7

43.1

48.8

55

56.9

80

83.8

84.7

87

8.5

9.9

12.6

17.4

16

28.1

32.3

32.9

44.7

Open SourceCommercial

Revisiting the Pragmatic QA CriteriaUnit Test Creation

Unit Test CreationFunction level recommendations

Heuristic lists are continuously updated

Size of function used to consider effort required

Focused Granularity

Consider Effort

Responsive Feedback

Evaluate Generality: Studied Open Source and Commercial systems

15

Change RiskProposed Approach

1. Mine changes and extract change metrics

2. Identify bug-inducing changes

3. Predict risky changes

Revisiting the Pragmatic QA CriteriaChange Risk

Change RiskMap changes to modified functions

Flag risky changes while they are fresh in developer’s minds

Size of change used to consider effort

Focused Granularity

Consider Effort

Responsive Feedback

Evaluate Generality: Study on Open Source and Commercial systems

Thank youI am grateful for this opportunity!

Backups

Revisiting the Pragmatic SQA Criteria

Unit Test Creation Change Risk

Focused Granularity

Estimate Effort

Evaluate Generality

Timely Feedback

Pragmatic SQA Criteria

What we have What we wantFile level Unit of prioritization

must be focused (e.g., function)

Act on results in future release

Must be able to act on results of approach quickly

No effort provided Provide an estimate of the effort required for the task

Focused Granularity

Estimate Effort

Timely Feedback

Evaluate Generality

To help understand implications of findings

21

Changes RiskNew code

Identify risky changes for closer examination before they are integrated into the code base

22

Unit Test CreationImplemented code

Prioritize the creation of unit tests for implemented code

23

Revisiting the Pragmatic-aware Criteria

Focused Granularity Timely feedback Estimate Effort Evaluate Generality

24

Avoid the most bugs effectively!

Write unit tests for functions with best Return on Investment (ROI)

How can we avoid the most bugs given limited resources?

25

Defect Prediction... An Example

Uses code metrics and pre-release defects to predict files with post-release defects in the Eclipse project.

26

Motivation

Software has^ bugs and

managers have ^ limited resources

always

always

How to allocate quality assurance efforts for implemented and new code?

Q:

A: Re-active and pro-active SQA

27

Proposed ApproachExtracting Historical Data

1. Extract change data to calculate metrics (e.g., churn)2. Use SZZ to identify and predict bug-inducing changes3.

28

Answer … Defect Prediction!

Use code metrics and process metrics to predict which files are most likely to contain defects in future releases

29

Defect Prediction... The Response

“The results are interesting, but of limited use to us ...”

... Why?

30

Defect Prediction... Lessons Learned

1. Identifying files is too coarse of a granularity

2. Having to wait till the next release is too late

3. Need an estimate of effort required

4. Why should we believe this will work for us?

31

Criteria for Pragmatic SQA Approaches

Timely Feedback

Must be able to act on results of approach quickly

32

Criteria for Pragmatic SQA Approaches

Estimate Effort

Provide an estimate of the effort required for the task

33

Criteria for Pragmatic SQA Approaches

Evaluate Generality

Evaluate on multiple projects/domains to clearly understand implications of the findings