© G. Ruhe 2008 1

Günther Ruhe

University of Calgary &

Expert Decisions Inc.

@

Software Engineering Institute

Pittsburgh,

January 15, 2008

The Art and Science ofSoftware Release Planning

© G. Ruhe 2008 2

Agenda

Decision Support in Requirements Engineering Software Release Planning EVOLVE* Decision Support System ReleasePlanner® Case Studies Future Research

© G. Ruhe 2008 3

Laboratory for Software Engineering Decision Support

Created in July 2001 at the University of Calgary

Research team of 12 researchers (2 undergraduate, 3 graduate, 5 PhD students, and 2 profs)

Research topics: Decision support (systems) forSoftware release planningProject portfolio planningSoftware resource managementCOTS selectionVerification and validation

Research approach: Interdisciplinary Both fundamental and applied researchEmpirical validation of results

University spin-off company: Expert Decisions

© G. Ruhe 2008 4

Problemdescription

Userrequirements

Developer’srequirements

Systemdesign

Componentrequirements

Component-leveldesign

Unitcode

Executablecomponents

Executablesystem

Usablesystem

Usedsystem

is verified against

is developed into

is integrated into

is validated against

How to allocate test effort?

When to terminate testing ?

Inspection?Re-inspection?

Reuse of components?

Classes? Objects?Methods?

Which technique?Which artifacts?

Decisions, Decisions, Decisions ….

Which requirements?

© G. Ruhe 2008 5

Description of a Decision Problem

Set A = {a1, a2, …} of alternatives (these alternatives are not necessarily described explicitly);

Set G = {g1, g2, …,gn} of criteria to evaluate each alternative a A from different perspectives, and

Preference structure (partial order defined on A)

Main categories of decision problems: Selection (P): Select one alternative a* A or a subset A* A; Triage (P): Assign each alternative a A to one of the

classes C1, C2, …, Ck.

Ranking (P): Arrange all alternatives in A according to an order a1 ≥ a2 ≥ …( a ≥ b means “alternative

a is at least as good as b”). Other classification: Structured versus unstructured decisions Strategic, tactical, and operational decisions

© G. Ruhe 2008 6

Decisions in Requirements Engineering

Activity Decision Level

Identification of business goals StrategicIdentification of stakeholders Tactical Level of involvement of stakeholders Tactical Selection of elicitation techniques Tactical Requirements elicitation Operational Requirements reuse Operational Necessity check Operational Feasibility check OperationalConsistency and completeness check OperationalRequirements negotiation OperationalRequirements prioritization Operational

Requirements documentation Requirements document standard Strategic Identification of problems Operational Actions to address the identified problems Operational Acceptance of requirements Strategic

Elicitation

Requirements analysis and negotiation

Requirements validation

© G. Ruhe 2008 7

Executableprogram

Changerequirement

SoftwareEvolution

Programvariants

PlanningProject goals

Project plan

SoftwareDevelopment

Specification

Information is uncertain inconsistent incomplete fuzzy

Decision space of large size of high complexity is dynamically

changing

Multiple objectives Usability Security Reliability Maintainability Portability

Hard and softconstraints on Time Effort Quality Resources

Why are RE Decisions so Hard?

© G. Ruhe 2008 8

Expected Benefits from Decision Support

Basic assumption

The more qualified computer-based/intelligent decision support is provided based on well-defined processes and roles, the more likely it is to find an appropriate decision.

Decision support systems combine the intellectual resources of individuals and organizations with the capabilities of the computer

– to generate and evaluate (new) solution alternatives;

– to allow more transparent decisions (to be better understood by involved people);

– to analyse decisions pro-actively;

– to better react on changes in the problem parameters (re-planning);

– to make more effective (trade-off) decisions (improved quality);

– to make more efficient decisions (improved cost-benefit).

© G. Ruhe 2008 9

Agenda

Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research

© G. Ruhe 2008 10

Planning for Software Releases

Incremental (not monolithic) software development

Set of competing (to be implemented) features

Stakeholder prioritize objects in terms of urgency, business value, risk, …

Release cycle time can be predefined or can be open

Limited resources and budgets need to considered

Release: Set of features constituting the next version of the product

Release can be refined into increments (strategic versus operational planning)

Hypothesis: Comprehensive and systematic release planning results in more reliable plans, better customer satisfaction, and more efficient usage of short resource ( = maximum value)

© G. Ruhe 200811

Decision Support for Product Release Planning (1/2)

Uncertainty is pervasive and unavoidable in software engineering. Uncertain software engineering decision problems are unlikely to be explicitly

modeled and completely formalized.

Set

of F

eatu

res/

Obj

ects

Interation 1

Interation 2

Interation 3

Release k

Release k+2

Release k+1

! Stakeholder involvement

! Unclear objectives ! Effectiveness &efficiency

! Resource, risk, andtechnologicalconstraints

© G. Ruhe 2008 12

Decision Support for Product Release Planning (2/2)

Decisions in software engineering are made by humans. They are based on both explicitly formulated and implicitly known objectives and constraints.

The goal of decision support is not to replace human judgment and expertise, but to assist in making better decisions.

Based on uncertain models, any formalized computational technique (subsumed as computational intelligence) in isolation is unlikely to determine meaningful results.

The advantage of the human intelligence based approach is that it is able to better handle soft and implicit objectives and constraints.

Support to solve

− to solve the right problem and

− to solve the problem right.

© G. Ruhe 2008 13

Maturity Factors for Product Release Planning

Processes rigour Level 1 Level 5

Degree of integration into product lifecycle management Level 1 Level 5

Planning based on measurement and defined criteria Level 1 Level 5

Type and degree of stakeholder involvement Level 1 Level 5

Continuity of planning and re-planning Level 1 Level 5

© G. Ruhe 2008 14

Art and Science of Release Planning

Art: Focus on the human intuition and communication– Pro: Flexible, easy to perform, ability to handle soft and implicit

concerns– Con: Effectiveness and efficiency for mid-size to large scale problems,

not transparent and not repeatable, re-planning is not supported, stakeholder involvement limited, resource consumption hard to take into account

Science: Emphasis on formalization of the problem and application of computational algorithms to generate best solutions.– Pro: Effective and efficient, easy re-planning, transparent, repeatable,

comprehensive stakeholder involvement possible, resource consumption part of the solution procedure

– Con: Needs formal description of the problem, needs computer support

© G. Ruhe 2008 15

Analysis of Existing Approaches RP MethodsDimensions

[Greer 2004]

[Penny 2002] [Bagnall et al. 2001]

[Jung 1998] [Karlsson & Ryan 1997]

[Denne & Cleland-

Huang 2004]

[Ruhe & Ngo-The 2004]

Scope 1 release 1 release 1 release 1 release 1 release Chunks of small releases

2 releases planned ahead

Objectives Based on benefit of system changes

Based on benefit of features to customers

Based on weight of customers

Based on value of requirements

Based on customer satisfaction

Based on return on investment

Based on value, urgency, stakeholders weights and satisfaction

Stakeholders involvement

Project manager

Developers, project management

Project manager, customer

Involvement of project manager is implied

Project manager, customer, users

All major stakeholders

All major stakeholders

Prioritization mechanism

Optimization-based

No defined prioritization scheme

Optimization-based

Optimization AHP IFM Heuristics Stakeholders prioritize features; Optimization heuristics used to balance conflicts

Technological constraints

Not available

Not available precedence Not available Not available Precedence(precursor)

Coupling and precedence

Resource constraints

Cost, risk Effort Cost Cost Cost Cost, time-to-market

Effort, risk, schedule

System constraints Operational risks

Not available Not available Not available Not available Not available Not available

Character and quality of solutions

One solution plan

One solution plan

One solution plan by any chosen search algorithm

One solution plan generated

One solution plan provided

One plan spanning many release periods

Several alternative solution plans are provided. These plans are diversified and fulfill some target quality level

Tool support Not available

Time-tracking system

Not available Not available Not available Partially available

ReleasePlanner®

© G. Ruhe 2008 16

Agenda

Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research

© G. Ruhe 2008 17

Evolutionary Solution Approach EVOLVE*

Computational Intelligence

Interation 1 Release 1

Release 2Interation 2

Interation 3 Release 3

Human Intelligence

© G. Ruhe 2008 18

Phase 1 - Modeling: Definition/update of

– Decision variables– Technological and resource constraints– Objectives– Stakeholder voting

Phase 2 - Exploration: Generate alternative solutions from the application of

− Voting analysis− Specialized linear integer programming− Branch and bound algorithms− Heuristics

Phase 3 - Consolidation: Human decision maker studies solution alternatives

− Maximally diversified solution alternatives− Absolute and relative degrees of stakeholder satisfaction− Comparison between solutions− Sorting capabilities− Reporting capabilities− Definition of re-planning scenarios

The Three Phases of EVOLVE*

© G. Ruhe 2008 19

Diversification Principle

Set of qualified &maximally

diversified

alternative solutions

Value

Urg

en

cy

95% Optimality alternatives

A portfolio of qualified solutions being structurally diversified is more appropriate than a single solution that is formally optimal but unlikely to solve the “right problem”.

© G. Ruhe 2008 20

Agenda

Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research

21

© G. Ruhe 2008 22

Business objects &objectives

Resource

types

&

their ca

pacitie

sResource

consumptions

Ran

king

of

obje

cts

Stakeholder &

their importance

Key Components

© G. Ruhe 2008 23

ReleasePlanner® for Elicitation of Customer Priorities

© G. Ruhe 2008 24

Feature elicitation

Problemspecification (2)

Feature repository (1)

Resource utilizationanalysis (3)

Generation of releaseplan alternatives (7)

Evaluation of planalternatives (8)

Selection of mostattractive planning scenarios

and alternatives New

/ev

olvi

ng fe

atur

es

Sho

rtag

e of

res

ourc

es

Stakeholder voting (4)

Stakeholdervoting analysis (5)

Product Manager Stakeholders ReleasePlanner®

Resource Estimation

What-if analysis (9)

Stakeholderaccess definition

Reporting (10)

Corporate Business IT

Business data,products and

processes

(6)

© G. Ruhe 2008 25

Agenda

Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research

© G. Ruhe 2008 26

Case Studies and Lessons Learned (LL)

Corel iGrafx (2003/04): Product release planning

LL: Structured process Broader stakeholder involvement Siemens CT SE 3 (2004 - ): Service portfolio planning

LL: Explicit process and objectives Facilitator for understanding & improvement. Involvement of key account managers.

City of Calgary (2005 - ): Project portfolio planning

LL: Integration of release planning into business processes WestJet (2005/06): Project portfolio planning

LL: Risk mitigation from running what-if scenarios Solid Technology (2005): Product release planning

LL: Greater likelihood of feasible and stable plans Chartwell Technology (2005 - ): Product release planning

LL: More conscious release decisions more competitive products Siemens Audiologische Technik (2007): better understanding of

(world-wide) customer needs

© G. Ruhe 2008 27

Case Study at Trema Laboratories

Case study based on release planning process at Trema Laboratories, Inc.

Used both current ad hoc planning methodology and ReleasePlanner®– Determination of the operational feasibility of the strategic release plan– Assessment of the value of using ReleasePlanner® compared to ad

hoc planning Study used one of the projects for the delivery of the next release:

– Releases delivered every three months – Team size of 14 with varying levels of skills, experience and availability– Roles vary from product specialist, analyst/designer, developer, and/or

tester– Adopted a 2-weekly increment– Six stakeholders with varying interests and input and geographically

dispersed.

© G. Ruhe 2008 28

Comparison of Solutions

PS AD DV TSCMM-1019A CFF - Phase II 4 3 25 6CMM-1090A Report Manager Infrastructure for Forecast Reports - Phase II 2 2 13 3CMM-1118A Overall CMM Integration 5 4 16 5CMM-1076 Transaction Message Integration 2 4 14 2CMM-1092 CMM-ACM Integration 9 6 32 3CMM-1028 Target Balance 3 2 35 5CMM-1049 Transaction / Cash Record Reporting 1 2 10 2CMM-1043 Bank Records for Internal Transactions - Creation and Reconciliation 5 3 26 6CMM-1034 Handling Bank Transactions originated by the Bank 1 2 11 1CMM-1119 Menu Cleanup 1 0 8 1CMM-1047 Security and Auditing for MIHB 5 5 15 5CMM-1016 Automation Performance and Coverage 1 2 15 2CMM-1022 Parsing and Enriching 6 3 29 2CMM-1120 Flexible Parameters 1 2 20 2CMM-0638 Context Sensitive Help 0 0 6 29

TOTAL EFFORT 46 40 275 74

Size (Mandays)CMM Functionality Description

CMM 7.1.0 - R1 Deliverables

ReleasePlanner® Adhoc Release Plan

Comparing Results from ad hoc and ReleasePlanner®

© G. Ruhe 2008 29

Conclusions

PS AD DV TSCMM-1019A CFF - Phase II 4 3 25 6CMM-1090A Report Manager Infrastructure for Forecast Reports - Phase II 2 2 13 3CMM-1118A Overall CMM Integration 5 4 16 5CMM-1076 Transaction Message Integration 2 4 14 2CMM-1092 CMM-ACM Integration 9 6 32 3CMM-1028 Target Balance 3 2 35 5CMM-1049 Transaction / Cash Record Reporting 1 2 10 2CMM-1043 Bank Records for Internal Transactions - Creation and Reconciliation 5 3 26 6CMM-1034 Handling Bank Transactions originated by the Bank 1 2 11 1CMM-1119 Menu Cleanup 1 0 8 1CMM-1047 Security and Auditing for MIHB 5 5 15 5CMM-1016 Automation Performance and Coverage 1 2 15 2CMM-1022 Parsing and Enriching 6 3 29 2CMM-1120 Flexible Parameters 1 2 20 2CMM-0638 Context Sensitive Help 0 0 6 29

TOTAL EFFORT 46 40 275 74

Size (Mandays)CMM Functionality Description

CMM 7.1.0 - R1 Deliverables

ReleasePlanner®

Adhoc Release Plan

Comparing Results from ad hoc and ReleasePlanner®

One of the five alternative plans from ReleasePlanner® is similar to the plan generated using the ad hoc method

Alternative 1 has an additional requirement implemented

Using ReleasePlanner® has taken less than a day

Ad hoc approach to arrive at same conclusions took weeks of work and meetings

ReleasePlanner® allows easy re-planning in case of changed requirements, capacities or effort estimates.

Substantial time savings

Higher likelihood to understand and address customer priorities

© G. Ruhe 2008 30

Added Value Quantitative

49%

11%

31%

9%

Initial Planning

Replanning

Stakeholder Savings

Sales Revenue Increase

Breakdown of Value Added by

ReleasePlannerTM

V(T) = $187,250.00 for T = 1 year

© G. Ruhe 2008 31

Agenda

Decision Support in Requirements Engineering Software Release Planning EVOLVE* ReleasePlanner® Case Studies Future Research

© G. Ruhe 2008 32

Summary and Conclusions

Requirements engineering is a decision-centric process

Decision support for product release planning aims to move from reactive proactive mode of performance

Good release decisions are based on explicit and systematic processes aimed to balance intuition and rationality

Ongoing and future research:

– Release planning for product lines

– Non-linear release planning

– Operational resource allocation

– Release planning with flexible release dates

– Learning and explanation for release planning

– Light-weight re-planning

– Empirical evaluation