Value-Based Software Testing: An Approach to Prioritizing Tests

STC 2014

VU NGUYEN

Faculty of Information Technology University of Science, VNU-HCMC

Agenda

• Introduction

• Value-based software engineering

• Value-based software testing

2

Introduction - 1

3

This release is very important for our users, we want your team to test it within 5 days

The release includes many changes and new features. We have to test 2000 test cases, and that takes at least 10 days to complete

Well, we understand that. But we cannot release later than 5 days

As a QA engineer, what should you do?

Your client

You

Introduction - 2

• Pareto Principle or 80-20 rule

– 80% of the land in Italy owned by 20% of population

– 80% of the wealth and income controlled by 20% of population

– 80% of accidents committed 20% motorists

– 80% of Facebook posts made by 20% users

4

Introduction - 3

• In software development

– 80% of MS Office users use only 20% of its features

– 80% of the business value come from 20% of the features

– 80% of the defects come from 20% of the modules

– 80% of the defects come from 20% of the test cases

5

Introduction - 4

• Value-based solution to the test request above

6

% of

value

from

testing

Test cases

100

80

60

40

20

500 1000 1500

Agenda

• Introduction

• Value-based software engineering

• Value-based software testing

7

Value-neutral software practices

• Value-neutral software practices

– Every requirement, use case, module, test case is equally important

• Value-neutral practices are risky because

– project is constrained by time, budget, staff, quality

– software lifecycle is ever reduced

8

Value-based SE

• Value considerations are important to software project success

• Value

– An equivalent in goods, services, or money

– Relative worth, utility or importance

• Non-monetary values are important

– Fairness, customer satisfaction, trust

• Values vary by stakeholders

9

Value-based SE - 2

• VBSE: software development practices take into account value considerations

• Examples

– Value-based requirements • Defining objectives by taking into account stakeholder

value propositions

• Feature prioritization based on business value, cost, risk

– Value-based testing • Test case prioritization based on business value, cost, risk

10

Value-based SE - 3

• Some important VBSE practices

– Expectations management • Manage stakeholders’ expectations focusing on their value

propositions

– Risk management • Manage most important risks (eg., top ten risks)

– Prioritization • Prioritize requirements, use cases, test cases, etc. based

on value

11

Agenda

• Introduction

• Value-based software engineering

• Value-based software testing

12

Value-based testing practice

• Test management and execution based on value

• A value-based testing approach

1. Derive test cases from prioritized requirements

2. Determine risk exposures and risk reduction leverage related to each test case

3. Prioritize test cases using risks

13

Risk management

• Risk is a potential problem

• Risk Exposure (RE) = Loss x Probability

– Loss: the value lost if the risk occurs

– Probability: the chance of the risk to occur

• Risk Reduction Leverage (RRL)

RRL = (REbefore – REafter)/Cost

Cost: can be the effort spent for testing

REbefore, REafter: Risk Exposure before and after testing is done

14

Test case prioritization - 1

• For each test case, determine RE using

– Business value

– Defect criticality

– Defect-prone components and developers

RE = Size of Loss x Probability of Loss

– E.g., Size = 20 person-hours, Probability = 20% RE = 20 * 20% = 4

15

Size of Loss

Probability of Loss

Test case prioritization - 2

• Value-based goal: maximizing the value gained in testing

– by reducing risk associated with tests

– meaning, maximizing Risk Reduction Leverage (RRL)

• Greedy approach: prioritize test cases based on RRL (focusing on tests with high RRL first)

RRL = (REbefore – REafter)/Cost

16

Research evidence

• Qi Li et al, 2012

– Results from experiment at Galorath Inc.

17

74.07%

77.78%

83.95%

90.12% 93.83% 95.06%

100.00%

4.94% 6.17%

9.88%

16.05%

22.22%

25.93%

58.02%

35.80%

39.51%

45.68%

51.85%

58.02% 61.73%

87.65%

0.00%

10.00%

20.00%

30.00%

40.00%

50.00%

60.00%

70.00%

80.00%

90.00%

100.00%

8 10 12 14 16 18 20 22 24 26 28 30Stop Testing

Value-based

Value-neutral

Value-inverse

Tool support

• Calculations can be done easily in Excel

• Test cases are prioritized/sorted according to RRL

18

Making it simple to use

• It can be time-consuming to estimate risks for all test cases

• We can approximate the process by

– ignoring risks after testing • prioritizing based on REbefore/Effort

– prioritizing features/functions to test instead of test cases

19

Making it simple to use - 2

• Ignoring risks after testing

– Assuming we get the same risk reduction rate after testing

20

Making it simple to use - 3

• Prioritizing at feature level

– Prioritizing features instead of test cases

– Link test cases with features

– Run test cases according to prioritized features

21

Conclusions

• Value considerations important to success of software projects

• Values dependent on stakeholders

– Monetary, satisfaction, importance, cost, trust, happiness, etc.

• Value-based testing: prioritizing tests using risks is important to maximize values of testing

22

References

• Qi Li, Barry Boehm, “Value-Based Software Test Prioritization”, Annual Research Review, CSSE, USC, Mar 2012 (http://csse.usc.edu/csse/event/2012/ARR/presentations/QLi_ARR_March2012.pdf)

• Barry Boehm, “Value-Based Software Engineering”. ACM Software Engineering Notes, 2003; 28(2)

• Barry Boehm, and Victor Basili, "Software Defect Reduction Top 10 List," Computer, vol. 34, no. 1, pp. 135-137, Jan. 2001 (http://csse.usc.edu/csse/TECHRPTS/2001/usccse2001-510/usccse2001-510d.doc)

23

© 2014 HCMC Software Testing Club

THANK YOU