53
MA FullͲday Tutorial 11/11/2013 8:30 AM "Agile Release Planning, Metrics, and Retrospectives" Presented by: Michael Mah QSM Associates, Inc. Brought to you by: 340 Corporate Way, Suite 300, Orange Park, FL 32073 888Ͳ268Ͳ8770 ͼ 904Ͳ278Ͳ0524 ͼ [email protected] ͼ www.sqe.com

Agile Release Planning, Metrics, and Retrospectives

Embed Size (px)

DESCRIPTION

How do you compare the productivity and quality you achieve with agile practices with that of traditional waterfall projects? Join Michael Mah to learn about both agile and waterfall metrics and how these metrics behave in real projects. Learn how to use your own data to move from sketches on a whiteboard to create agile project trends on productivity, time-to-market, and defect rates. Using recent, real-world case studies, Michael offers a practical, expert view of agile measurement, showing you these metrics in action on retrospectives and release estimation and planning. In hands-on exercises, learn how to replicate these techniques to make your own comparisons for time, cost, and quality. Working in pairs, calculate productivity metrics using the templates Michael employs in his consulting practice. You can leverage these new metrics to make the case for changing to more agile practices and creating realistic project commitments in your organization. Take back new ways for communicating to key decision makers the value of implementing agile development practices.

Citation preview

Page 1: Agile Release Planning, Metrics, and Retrospectives

MA FullͲday�Tutorial�11/11/2013�8:30�AM�

�����

"Agile Release Planning, Metrics, and Retrospectives"

���

Presented by:

Michael Mah QSM Associates, Inc.

��������

Brought�to�you�by:��

��

340�Corporate�Way,�Suite�300,�Orange�Park,�FL�32073�888Ͳ268Ͳ8770�ͼ�904Ͳ278Ͳ0524�ͼ�[email protected]�ͼ�www.sqe.com

Page 2: Agile Release Planning, Metrics, and Retrospectives

Michael Mah QSM Associates, Inc.

With twenty-five years of industry experience Michael Mah teaches, writes, and consults for QSM Associates to tech companies on measuring and estimating software projects for offshore, waterfall, and agile. Michael and his QSM partners have researched thousands of projects worldwide. His work examines time-pressure dynamics of teams and their contribution to project success and failure. Michael’s clients include Boeing, Progressive, Verizon Wireless, Nationwide, JPMorgan Chase, Roche, and other Fortune 100 companies. He is the director of the Benchmarking Practice at the Cutter Consortium in the US. A private pilot, Michael lives in the mountains of western Massachusetts. He can be reached at qsma.com.

Page 3: Agile Release Planning, Metrics, and Retrospectives

1

10/22/13

Better Software Conference East Boston, MA

November, 2013

Agile Release Planning, Metrics, and Retrospectives - Workshop Slides

Michael Mah Managing Partner

QSM Associates, Inc. 75 South Church Street

Pittsfield, MA 01201 413-499-0988

Fax 413-447-7322 e-mail: [email protected]

Website: www.qsma.com Blog: www.optimalfriction.com

Copyright QSM Associates, Inc. 10/22/13 2

Michael Mah is the director of the Benchmarking Practice at the Cutter Consortium, a Boston-based IT think-tank, and served as past editor of the IT Metrics Strategies

publication. He is also managing partner at QSM Associates Inc. based in Massachusetts USA. Michael teaches, writes, and consults to technology companies on measuring, estimating and managing software projects, whether in-house, offshore, waterfall, or agile. With over 25 years of experience, Michael and his partners have derived productivity patterns for thousands of software projects collected worldwide across engineering and business applications. His current work examines time-pressure dynamics of teams, and its role in project success and failure. In addition to his background in physics and electrical engineering, he is a mediator specializing in dispute resolution for technology projects. He has a degree in electrics engineering from Tufts University. His training on dispute resolution, mediation, and participatory processes is from the Program on Negotiation at Harvard Law School and the Radcliffe Institute for Advanced Study.

He can be reached at [email protected].

Michael Mah

Page 4: Agile Release Planning, Metrics, and Retrospectives

2

Copyright QSM Associates, Inc. 10/22/13 3

We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools working software over comprehensive documentation Customer collaboration over contract negotiation responding to change over following a plan That is, while there is value in the items on the right, we value the items on the left more.

© 2001 Kent Beck, Mike Beedle, Arie van Bennekum, Alistair Cockburn, Ward Cunningham, James Grenning, Jim Highsmith, Andrew Hunt, Ron Jeffries, Jon Kern, Brian Marick, Robert C. Martin, Steve Mellor, Ken Schwaber, Jeff Sutherland, Dave Thomas, Martin Fowler

Manifesto for Agile Software Development

Copyright QSM Associates, Inc. 10/22/13 4

�Without metrics, you�re just another person with a different opinion.�

Page 5: Agile Release Planning, Metrics, and Retrospectives

3

Copyright QSM Associates, Inc. 10/22/13 5

�Frothy eloquence neither convinces nor satisfies me. I am from Missouri. You have got to show me.�

- Missouri Congressman Willard Duncan Vandiver, 1899

Copyright QSM Associates, Inc. 10/22/13 6

�Metrics� Is Not a Dirty Waterfall Word… ! Metrics (or measures) can be as light or as heavy as

you want them to be. Think of them as – �Information� ! Familiar examples of measures that inform us:

! Stock Market Indexes ! Blood Tests (i.e. cholesterol) ! Newborn �Apgar Score� ! Astronomy – Distance (i.e. Light-Years, Astronomical Units (AUs)…) ! Others…

!  In technology, we often want to know information about what works and what doesn�t work, whether we�re �Better, Faster, Cheaper� or if we�re �more productive.�

Page 6: Agile Release Planning, Metrics, and Retrospectives

4

Copyright QSM Associates, Inc. 10/22/13 7

Measures that Inform On… ! Time (Faster) - Project Duration, perhaps by phase. ! Cost (Cheaper) - Effort (including labor rates) ! Quality (Better) – Bugs/Defects, User Satisfaction ! Project Scope – Features, Requirements, Use Cases,

Stories etc.

One way of capturing this information is by drawing a picture…

Copyright QSM Associates, Inc. 10/22/13 8

An Example of a Waterfall Picture

Page 7: Agile Release Planning, Metrics, and Retrospectives

5

Copyright QSM Associates, Inc. 10/22/13 9

An Example of an Agile Picture

Copyright QSM Associates, Inc. 10/22/13 10

Entering Metrics from Whiteboard into QSM SLIM Model

Page 8: Agile Release Planning, Metrics, and Retrospectives

6

Copyright QSM Associates, Inc. 10/22/13 11

Project Interviews

Copyright QSM Associates, Inc. 10/22/13 12

Short Exercise – Observations of the Two

! What similarities do you observe? With regard to the following ! Phases? ! Time? ! Staffing? ! Project Scope/Size? ! Defects?

! What differences do you observe?

Page 9: Agile Release Planning, Metrics, and Retrospectives

7

Copyright QSM Associates, Inc. 10/22/13 13

An Example of an Agile Picture

Copyright QSM Associates, Inc. 10/22/13 14

An Example of a Waterfall Picture

Page 10: Agile Release Planning, Metrics, and Retrospectives

8

Copyright QSM Associates, Inc. 10/22/13 15

An Example of a Waterfall Picture

Copyright QSM Associates, Inc. 10/22/13 16

Exercise #1

Metrics Capture: Agile XP Release (Part 1)

Page 11: Agile Release Planning, Metrics, and Retrospectives

9

Copyright QSM Associates, Inc. 10/22/13 17

! You will learn how to: (Part 1) ! �Translate� a description of a project into a

whiteboard staffing sketch, illustrating the major phases of the work over time, and the staffing used for both the story phase and the build (iterations) phase. (Project interview)

! Extract the key information measures from the sketch. (Metrics capture)

Objectives of this Exercise

Copyright QSM Associates, Inc. 10/22/13 18

Exercise #1

Metrics Capture: Agile XP Release (Part 2)

Page 12: Agile Release Planning, Metrics, and Retrospectives

10

Copyright QSM Associates, Inc. 10/22/13 19

! You will learn how to: (Part 2) ! Open a data collection template (SLIM-Datamanager)

and use it to record this information into a file. ! Save the file.

Objectives of this Exercise

10/22/13

Learn to Measure (and Deal Explicitly with) Project Size

Page 13: Agile Release Planning, Metrics, and Retrospectives

11

Copyright QSM Associates, Inc. 10/22/13 21

Learn to Measure Size

!  Many people today size projects in terms of Person Hours

!  That�s not size !  That�s effort !  Some also size a project as being �25 People� !  That�s not size !  That�s staffing (headcount)

Copyright QSM Associates, Inc. 10/22/13 22

How Far to the Airport?

!  If you tell me 40 minutes, you haven�t answered my question

!  Distance is not measured in minutes !  Distance is measured in miles, or feet, or

kilometers, or …

Page 14: Agile Release Planning, Metrics, and Retrospectives

12

Copyright QSM Associates, Inc. 10/22/13 23

How Far to the Airport?

!  40 minutes is assuming a certain !  Vehicle !  Route !  Time of day !  Traffic pattern !  Speed…

!  This is an estimate of time

Copyright QSM Associates, Inc. 10/22/13 24

How Far to the Airport?

!  If you tell me 30 miles, you have answered my question

!  I can estimate the time for a future airport trip by considering my history for any given !  Vehicle !  Route !  Time of day !  Traffic pattern !  …

Page 15: Agile Release Planning, Metrics, and Retrospectives

13

Copyright QSM Associates, Inc. 10/22/13 25

Size Categories

New Modified

Unmodified

Adapted (adds work;

impacts Size)

Created from Scratch (adds work; impacts Size)

Purchased / Reused (adds complexity; impacts Productivity)

Copyright QSM Associates, Inc. 10/22/13 26

What Do We Mean By Effective Size?

70 UnitsNew ModifieEffectiv deS S S= + =

New 20 Units

Modified 50 Units Adapted

(adds work; impacts Size)

Created from Scratch (adds work; impacts Size)

Page 16: Agile Release Planning, Metrics, and Retrospectives

14

Copyright QSM Associates, Inc. 10/22/13 27

Usefulness of Various Size Units ! There are many different

units people use to size software

! They are all related to what must be created, but at different levels of abstraction

! Each can be useful depending on where you are in the software development lifecycle

Copyright QSM Associates, Inc. 10/22/13 28

What Do We Mean By Size?

Business Concern: Value, Price Focus: Bang for the Buck Size Measures:

Requirements Function Points Use Cases Stories/Story Points Features

Business Concern: Cost Focus: Productivity Size Measures:

Lines of Code Statements Program Actions Modules, Objects

Development

Process

Product Need

Units of Need Units of Work

Page 17: Agile Release Planning, Metrics, and Retrospectives

15

Copyright QSM Associates, Inc. 10/22/13 29

Dividing Units of Need It may be helpful to divide the Units of Need

into: ! Simple ! Medium ! Complex

Copyright QSM Associates, Inc. 10/22/13 30

Getting �Gearing Factors� Once you know what the Units of Need and Units of Work

are, you ask: ! How large is a simple one? ! How large is a medium one? ! How large is a large one? ! How many small, medium, and large ones will there be? This gives you Gearing Factors

Page 18: Agile Release Planning, Metrics, and Retrospectives

16

Copyright QSM Associates, Inc. 10/22/13 31

Gearing Factor !  It is valuable to know how many Units of Work are typically

associated with a given Unit of Need. ! This is the �Gearing Factor� - It can be calculated from

completed projects. !  It can be thought of as a �currency conversion,� informing us

about how much software (Units of Work) it took to implement a feature, a story, or a requirement. Examples: ! 200 lines of code (source instructions) in a C++ Object. ! 3 Objects to implement a simple feature. 6 Objects to

implement a complex feature. ! 10 Stories in an agile Iteration. ! Others…

!  If desired, we can tally a total Units of Need and Units of Work in a spreadsheet.

Copyright QSM Associates, Inc. 10/22/13 32

# Component NameMost LikelyHigh

Most Likely Expected

1 Simple Foundation Table 5 11 552 Average Foundation Table 15 25 3753 Complex Foundation Table 20 9 1804 Gaps Simple (PeopleSoft Customizations) 34 11 3743 Gaps Average (PeopleSoft Customizations) 66 25 16504 Gaps Complex (PeopleSoft Customizations) 345 9 31055 Business Rules 5 0 06 Data Conversion 6 2496 149769 Interface Simple 320 276 88320

10 Interface Average 620 127 7874011 Interface Complex 1520 21 3192013 PeopleSoft Upgrade Rework 0 0 014 Custom Report Simple 25 0 015 Custom Report Average 50 0 016 Custom Report Complex 100 47 4700

Estimated Total # of Implementation Units = 224,395

Code, Instructions, or Implementation Units (IUs) per Component Number of Components

in the �Cart�

Sizing Example – A Component �Shopping Cart�

Page 19: Agile Release Planning, Metrics, and Retrospectives

17

Copyright QSM Associates, Inc. 10/22/13 33

Agile Teams Explicitly Deal with, and Measure Size…

Copyright QSM Associates, Inc. 10/22/13 34

Example: On Sizing, Agile Teams Use… Stories !  A story, or a feature, is described by the product owner. You might

also describe it as a �requirement.� !  Not all stories are created equal. Some are smaller, some are

larger than others. !  Story points are a unit of measure for expressing the overall size of

a story. There is no set rule for this. It is an amalgamation of the effort, the complexity, the risk, etc. associated with a story.

!  The range of the scale can be 1-10, 1-7 (or whatever), depending on which book you reference. Agile authors haven�t seemed to set any standard; they say that what�s important is that the numbers are relative. i.e. a story with 10 story points is 2x one that is scored at a 5, and if you�re using a 10 scale, then a 5 is �average.�

Page 20: Agile Release Planning, Metrics, and Retrospectives

18

Copyright QSM Associates, Inc. 10/22/13 35

It Takes a Certain Amount of Code to Produce a Story

!  Within a given iteration (say… 2 weeks), an agile team accomplishes the work to produce a certain number of stories, and the associated story points. The number of story points accomplished in an iteration is called �velocity.�

!  Since we�re talking about creating �software� in a given iteration, these features/stories are manifested by programmers who create new code, and/or adapt (modify) existing code to produce the stories/points.

!  In an iteration (and across an entire release), we can express the total amount of code that delivers these stories/points, and understand the proportional relationship between them. i.e. �It took about 14,000 lines of code to produce 10 story points in this iteration, which translates to (on average) 1,400 lines of code per story point.

!  If you suppose that a release is aiming for a total of 200 story points, it might involve 280,000 lines of new/modified code (1,400 x 200).

Copyright QSM Associates, Inc. 10/22/13 36

Exercise #2

Creating Schedule and Effort Trendlines

Page 21: Agile Release Planning, Metrics, and Retrospectives

19

Copyright QSM Associates, Inc. 10/22/13 37

Objectives of this Exercise

! You will learn how to: ! Create an X-Y Graph for a group of projects –

smaller releases on the left, larger releases on the right – for two metrics of interest: Schedule and Effort.

! Understand how to visually construct a quick Regression Fit through the data to determine the average trend, and the high-low.

! Use this baseline as a framework to evaluate potential scenarios for a future project.

10/22/13

Learn to Measure (and Deal Explicitly with) Productivity

Page 22: Agile Release Planning, Metrics, and Retrospectives

20

Copyright QSM Associates, Inc. 10/22/13 39

Software Development Core Metrics

Duration

Effort

Discovered Defects

Produced Software

(Size)

How long?

How much?

How good?

Copyright QSM Associates, Inc. 10/22/13 40

How Would You Describe �Productivity Improvement?�

Producing a certain amount of functionality | or features, faster, with lower cost at the same or higher level of quality… or Within a given timeline, producing more functionality or features, at lower cost, at the same or higher level of quality… or … other variations on the theme

Page 23: Agile Release Planning, Metrics, and Retrospectives

21

Copyright QSM Associates, Inc. 10/22/13 41

Production Equation Conceptual Form

Deliverable Size

is over Time at some Productivity Effort

Copyright QSM Associates, Inc. 10/22/13 42

Production Equation (Rearranged) Conceptual Form

Deliverable Size

is

over

Time

Productivity

Effort

and

Page 24: Agile Release Planning, Metrics, and Retrospectives

22

Copyright QSM Associates, Inc. 10/22/13 43

Size = 272,768 SLOC

Effort = 249 Person-Months

Time = 13 Months * WFSO 5.1

PI = SIZE

TIME EFFORT = 19.4

Production Equation (In Actual Practice) Calibration Form

Copyright QSM Associates, Inc. 10/22/13 44

Productivity contributing elements ! Nobody knows how many elements effect a given

environment�s ability to produce a system ! There are at least dozens, perhaps thousands ! Nobody knows what the effect of their interaction is

Page 25: Agile Release Planning, Metrics, and Retrospectives

23

Copyright QSM Associates, Inc. 10/22/13 45

Productivity typical factors

! Tooling / Methods ! Infrastructure ! Tools ! Standards

! Technical Difficulty ! Hardware Constraints ! Algorithm Complexity ! Logic Complexity ! Management

Complexity ! Platform Stability

! Personnel Profile ! Management ! Environment ! Team Capabilities

! Integration Issues ! Amount of reused

software ! Integration complexity ! Number of interfaces ! Existing documentation

Copyright QSM Associates, Inc. 10/22/13 46

Productivity Index (PI) (industry values by application type)

0 2 4 6 8 10 12 14 16 18 20 22 24

Productivity Index (PI) w/ ±1 Standard Deviation

Avionics

Business

Command and Control

Microcode

Process Control

Real Time

Scientific

System

Telecommunications

Information

Engineering

Real Time

Page 26: Agile Release Planning, Metrics, and Retrospectives

24

Copyright QSM Associates, Inc. 10/22/13 47

What�s a PI Worth?

Size = 350 Java Classes/Objects Burdened Labor Rate = $120,000/PY

Productivity Index

Effort (PM)

Schedule (Mos)

Cost ($)

18 17 16

42

56

78

9.4

10.5

11.6

420,000

560,000

780,000

MTTD (Days)

4.1 3.5 2.8

Copyright QSM Associates, Inc. 10/22/13 48

Exercise #3

Assessing 6 Agile XP Releases

Page 27: Agile Release Planning, Metrics, and Retrospectives

25

Copyright QSM Associates, Inc. 10/22/13 49

Objectives of this Exercise

! You will learn how to: ! Look at productivity patterns for a group of

completed projects. ! Examine if productivity is rising or falling over time. ! Understand how demonstrated/accomplished

schedules and effort (high – low) relate to derived productivity values (low-high).

! Create your own trendlines for schedule, staffing, and defects

! Assess productivity targets implied by proposed deadline-scope pairings and evaluate they are reasonable (or not) when �sanity checked� against past accomplishments.

Copyright QSM Associates, Inc. 10/22/13 50

Two Case Studies:

Co-located Agile XP and Distributed SCRUM

Page 28: Agile Release Planning, Metrics, and Retrospectives

26

Copyright QSM Associates, Inc. 51

Co-Located XP Case Study — Follett Software

! Team size ! 24 Developers ! 7 Testers ! 3 Customers ! 3 Project Leaders

! Code Base ! 1,000,000 lines of code ! 7,000 automated unit test ! 10,000 automated acceptance

test

Copyright QSM Associates, Inc. 52

Why XP for Follett?

! �XP allowed us to start building based on current assumptions�

! �XP approach allowed us to change directions when needed�

! �XP iterations gave us a �pilot project� test bed�

! �Focus on building customer value gave high visibility�

Page 29: Agile Release Planning, Metrics, and Retrospectives

27

Copyright QSM Associates, Inc. 10/22/13 53

Robert Lucas, Nobel Prize (Economics): The force of concentration, or �clustering� of human creativity and talent … the powerful economic gains when smart and talented people locate in close proximity to one another. �Human capital externalities�: the productivity and innovation gains that occur when human beings cluster together.

Source: Richard Florida Flight of the Creative Class

On Co-Location of Smart People

Copyright QSM Associates, Inc. 54

People Management ! XP says �XP works in

small- to medium-sized teams�

! How we evolved or extended this rule ! Subteams ! 1 large room is mandatory

! Trade-offs ! Communication between

subteams ! 1 room noise level

(distractions) ! Lack of personal space

Page 30: Agile Release Planning, Metrics, and Retrospectives

28

Copyright QSM Associates, Inc. 10/22/13 55

Copyright QSM Associates, Inc. 10/22/13 56

Page 31: Agile Release Planning, Metrics, and Retrospectives

29

Copyright QSM Associates, Inc. 10/22/13 57

Copyright QSM Associates, Inc. 10/22/13 58

Destiny Release 6.5 – Whiteboard Sketch

Page 32: Agile Release Planning, Metrics, and Retrospectives

30

Copyright QSM Associates, Inc. 10/22/13 59

Input to SLIM-DataManager

Size

Time

Defects

Effort

Copyright QSM Associates, Inc. 10/22/13 60

SLIM Replica – Destiny 6.5 Staffing & Probability Analysis

Avg Staff (people)<Destiny Release 6.5>

1 2 3 4 5 6 7 8 9 10 11 12Apr'05

May Jun Jul Aug Sep Oct Nov Dec Jan'06

Feb Mar Apr May Jun

0

10

20

30

40

50

Avg Staff (people)

876543 21

R&DC&T

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

SOLUTION PANEL - <Destiny Release 6.5>

DurationEffortCost

Peak StaffMTTD

Start Date

C&T11.0400

3400.036.5

0.6387/2/2005

Life Cycle12.0446

3791.036.5

0.6386/1/2005

MonthsPM$ (K)peopleDays

PI=24.7 MBI=4.8 Eff SLOC=893,298

CONTROL PANEL - <Destiny Release 6.5>

PI 19.8 29.6

24.7

Peak Staff 29.2 43.8

36.5

Eff SLOC (K) 715 1072

893

Page 33: Agile Release Planning, Metrics, and Retrospectives

31

Copyright QSM Associates, Inc. 10/22/13 61

Trendline Assessment – Build Phase Staffing Main Build Peak Staff vs. Size

100 1,000

Effective SLOC (thousands)

0.1

1

10

100

1,000

Peak Staff (FTEs)

Rel 6.0

Rel 6.5

Rel 5.0

Rel 7.5 Rel 7.0 Rel 8.0Rel 6.0

Rel 6.5

Rel 5.0

Rel 7.5 Rel 7.0 Rel 8.0

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Normal Staffing

Copyright QSM Associates, Inc. 10/22/13 62

Trendline Assessment – Build Phase Schedule Main Build Phase Duration vs Size

100 1,000

Effective SLOC (thousands)

1

10

100

Time (Months)Rel 5.0

Rel 6.0

Rel 6.5

Rel 7.5

Rel 7.0

Rel 8.0Rel 5.0

Rel 6.0

Rel 6.5

Rel 7.5

Rel 7.0

Rel 8.0

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Schedules are Half Industry

Page 34: Agile Release Planning, Metrics, and Retrospectives

32

Copyright QSM Associates, Inc. 10/22/13 63

Trendline Assessment – Defects/Quality Defects During Test

100 1,000

Effective SLOC (thousands)

10

100

1,000

10,000

Errors (SysInt-Del)

Rel 5.0

Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0

Rel 8.0

Rel 5.0

Rel 6.0 Rel 6.5Rel 7.5 Rel 7.0

Rel 8.0

Business Sy stems Av ionic Sy stems Command & Control Microcode Sy stems Process Control QSM 2005 BusinessAv g. Line Sty le 1 Sigma Line Sty le

Far Fewer Defects: 50% - 66% Below Industry

Copyright QSM Associates, Inc. 10/22/13 64

Industry Average

Current Performance

Delta

Project Cost

$3.5 Million

$2.2 Million

-$1.3M

Schedule

12.6 months

7.8 months

-4.8 mos

Cumulative

Defects

2,890

1450

-50%

Staffing

35

35

n/a

* Using average project size of 500,000 lines of new and modified code

Follett vs. Industry Average

Page 35: Agile Release Planning, Metrics, and Retrospectives

33

Copyright QSM Associates, Inc. 65

Follett and XP: It has worked incredibly well… ! Destiny Library Manager:

" Award of Excellence 2004, presented by Technology and Learning magazine (December 2004).

" Awards Portfolio 2004, presented by Media and Methods magazine (May/June 2004).

" Technology & Learning Award of Excellence  2006, 2007 ! Destiny Textbook Manager

" Awards Portfolio 2005, presented by Media and Methods magazine (May/June 2005).

" Technology & Learning Award of Excellence  2007 ! Destiny Enriched Services

" Technology & Learning Award of Excellence  2007   Follett Software provides Library Automation Solutions to 52% of the K12 market. Destiny Library Manager: Single largest product market share in K12 with 19% of the total market and continues to outpace the competition in market growth.

Copyright QSM Associates, Inc. 10/22/13 66

Page 36: Agile Release Planning, Metrics, and Retrospectives

34

Copyright QSM Associates, Inc. 67

Distributed SCRUM Case Study — BMC Software ! Team size

! 90-95 Total ! 33 Developers ! 37 QA ! 20-25 Architects, PMs,

Mgrs

! 4 Locations ! US and India ! Very Large Releases ! 7 SCRUM Teams

Copyright QSM Associates, Inc. 10/22/13 68

Benchmark Interview — Highlights

Method: ! Conducted on-site interviews on both releases. ! Gathered industry standard core metrics for elapsed

time, effort, size*, and defects. ! Benchmarked the results, calculated performance

values, and compared them to the QSM database. ! Assessed schedule performance, FTE staffing levels,

effort, defects, and productivity values for the Rqmts (Story) and Main Build development phases.

* Iterations, stories, and the resultant added/changed code

Page 37: Agile Release Planning, Metrics, and Retrospectives

35

Copyright QSM Associates, Inc. 10/22/13 69

Project Interviews

Copyright QSM Associates, Inc. 10/22/13 70

Project Interviews

Page 38: Agile Release Planning, Metrics, and Retrospectives

36

Copyright QSM Associates, Inc. 10/22/13 71

Whiteboard Sketch – Performance Mgr R2.3

Copyright QSM Associates, Inc. 10/22/13 72

0

20

40

60

80

100

120

140

160

Defect Type (All)

Count of Severity*

Create Date Status Mode Status Severity* TR-Version

Product+*

Release 2.3 Defect Rate

Page 39: Agile Release Planning, Metrics, and Retrospectives

37

Copyright QSM Associates, Inc. 10/22/13 73

Input to SLIM-Data Manager

Size

Time

Defects

Effort

Size

Time

Copyright QSM Associates, Inc. 10/22/13 74

Staffing & Probability Analysis

Avg Staff (people)<Performance Manager Rel 2.3>

1 2 3 4 5 6 7 8 9Apr'06

May Jun Jul Aug Sep Oct Nov Dec Jan'07

Feb Mar

0

20

40

60

80

100

120

Avg Staff (people)

109876543 21

R&DC&TP_Mnt

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

Milestones 0 - CSR 1 - SRR 2 - HLDR 3 - LLDR 4 - CUT 5 - IC 6 - STC 7 - UAT 8 - FCR 9 - 99R 10 - 99.9R

SOLUTION PANEL - <Performance Manager Rel 2.3>

DurationEffortCost

Peak StaffMTTD

Start Date

C&T5.3488

4880.092.8

0.1047/2/2006

Life Cycle7.0556

5561.292.8

0.2326/1/2006

MonthsPM$ (K)peopleDays

PI=28.3 MBI=8.3 Eff SLOC=844,710

CONTROL PANEL - <Performance Manager Rel 2.3>

PI 22.6 33.9

28.3

Peak Staff 74.2 111.3

92.8

Eff SLOC (K) 676 1014

845

SLIM Replica — PerfMgr Rel 2.3

Page 40: Agile Release Planning, Metrics, and Retrospectives

38

Copyright QSM Associates, Inc. 10/22/13 75

Trendline Assessment ! The following graphs illustrate the staffing, schedule,

and effort, and defects for the BUILD phase (vertical axis).

! On each graph, projects of smaller, medium, and progressively larger sizes (e.g., number of stories) are shown along the horizontal axis. Release 2.4 is shown on the left at 526 stories (569k LOC), Release 2.3 on the right at 918 stories (845k LOC).

! The center line on the comparison graphs represents the QSM Industry Average, while the upper and lower dashed lines are the +/- 1 standard deviation ranges of the reference database (16th and 84th percentiles).

Copyright QSM Associates, Inc. 10/22/13 76

Agile Assessment — Schedule BUILD Phase Schedule

10 100 1,000STORIES (thousands)

1

10

100

C&T Duration (Months)

BMC Rel 2.4BMC Rel 2.3

BMC Rel 2.4BMC Rel 2.3

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Agile projects are faster as a whole. (BMC (and also Follett) are highlighted)

Page 41: Agile Release Planning, Metrics, and Retrospectives

39

Copyright QSM Associates, Inc. 10/22/13 77

Agile Assessment — Staffing BUILD Phase Staff ing

10 100 1,000STORIES (thousands)

1

10

100

1,000

C&T Peak Staff (People)

BMC Rel 2.4BMC Rel 2.3

BMC Rel 2.4BMC Rel 2.3

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Agile Projects� team sizes are fairly typical BMC elects to run with large teams.

Copyright QSM Associates, Inc. 10/22/13 78

Agile Assessment – Quality Bugs

10 100 1,000STORIES (thousands)

1

10

100

1,000

10,000

Errors (SysInt-Del)

BMC Rel 2.4BMC Rel 2.3

BMC Rel 2.4BMC Rel 2.3

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Follett and BMC bug rates are significantly lower

Page 42: Agile Release Planning, Metrics, and Retrospectives

40

Copyright QSM Associates, Inc. 10/22/13 79

Summary View — Agile Data Main Build Trends

BUILD Phase Schedule

10 100 1,000

STORIES (thousands)

1

10

100

C&T Duration (Months)

BUILD Phase Ef f ort

10 100 1,000

STORIES (thousands)

1

10

100

1,000

10,000

C&T Effort (PM)

BUILD Phase Staf f ing

10 100 1,000

STORIES (thousands)

1

10

100

1,000

C&T Peak Staff (People)

Bugs

10 100 1,000

STORIES (thousands)

1

10

100

1,000

10,000

Errors (SysInt-Del)

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Av g. Line Sty le 1 Sigma Line Sty le

Agile projects as a whole achieve faster speed

Low Defects for BMC & Follett

Copyright QSM Associates, Inc. 10/22/13 80

Productivity Index Assessment Productivity Index/Velocity

10 100 1,000STORIES (thousands)

0

5

10

15

20

25

30

35

PI

Agile Companies Company B SCRUM Company A - Agile XP QSM 2005 Business Avg. Line Style1 Sigma Line Style

Agile projects as a whole tend to exhibit higher PIs (Follett/BMC are circled)

Page 43: Agile Release Planning, Metrics, and Retrospectives

41

Copyright QSM Associates, Inc. 10/22/13 81

Productivity Index: Five Companies Using Agile Avg, Min, Max PI vs Organization

Agile #1 - Follett

Agile #2 - BMC

Company B

Company C

Company D

Organization

0 5 10 15 20 25 30 35 40Avg, Min, Max PI

All Systems Avg. Line Style

BMC and Follett lead the pack

Copyright QSM Associates, Inc. 10/22/13 82

Industry Average

Current Performance

Delta

Project Cost

$5.5 Million

$5.2 Million

-$.3M

Schedule

15 months

6.3 months

-8.7 mos

Defects During QA

713

635

-11%

Staffing

40

92

+52

BMC vs. Industry Average

Page 44: Agile Release Planning, Metrics, and Retrospectives

42

Copyright QSM Associates, Inc. 10/22/13 83

BMC �Secret Sauce�

Copyright QSM Associates, Inc. 10/22/13 84

BMC �Secret Sauce� (con�t) ! Buy-In

! VP-Level (or higher) Senior Executive Sponsorship ! Scrum Master Training ! Core Group Energized and Passionate

! Staying �Releasable� ! Nightly Builds/Test ! 2-week Iteration Demos ! Frequent, Rigorous Peer Code Review

! Dusk-to-Dawn Teamwork ! Communication Techniques for Information Flow ! Wikis, Video-conferencing, Periodic On-Site Meetings ! Co-Located Release Planning ! Scrum of Scrum Meetings (US Time)

Page 45: Agile Release Planning, Metrics, and Retrospectives

43

Copyright QSM Associates, Inc. 10/22/13 85

BMC �Secret Sauce� (con�t) ! Backlogs

! One Master Backlog AND Multiple Backlog Management ! One Setup for User Stories Across Teams ! Added �Requirements Architect� to Interface Product Mgt with R&D

! �Holding Back the Waterfall� ! Test Driven Development ! Retrospective Meetings to Not Regress into old Waterfall Habits ! Outside Source to Audit the Process

10/22/13

Tying It All Together: Release and Iteration Planning

Page 46: Agile Release Planning, Metrics, and Retrospectives

44

Copyright QSM Associates, Inc. 10/22/13 87

Collect and Use History Learn from the Past

! Observe and discover patterns ! Determine cause and effect ! Behave accordingly

Copyright QSM Associates, Inc. 10/22/13 88

Your Software Development Core Metrics History

Duration

Effort

Discovered Defects

Produced Software

(Size)

How long?

How much?

How good?

Page 47: Agile Release Planning, Metrics, and Retrospectives

45

Copyright QSM Associates, Inc. 10/22/13 89

! Given a Certain Development Efficiency/Productivity from Observed Patterns

! And Given the Deadline ! With a Team of �X� People ...

�Real World Deadline Driven Estimation�

How Much Functionality Can We Build? How Much Functionality Should We Promise?

Copyright QSM Associates, Inc. 10/22/13 90

Rifkin�s Dicta

On Software Estimation: !  Commitments have to be based on work to be performed

(scope/size); therefore, there must be agreement on this. !  Estimates have to be based on the work to be performed

(scope/size) and historical records of performance. !  Commitments must not exceed the capability to perform,

or else there is no reason to estimate.

Stan Rifkin Master Systems Inc. Carnegie Mellon SEI

Page 48: Agile Release Planning, Metrics, and Retrospectives

46

Copyright QSM Associates, Inc. 10/22/13 91

Productivity Relationship Conceptual Form

Deliverable Size

is over Time at some Productivity Effort

Copyright QSM Associates, Inc. 10/22/13 92

Productivity Relationship (Rearranged) Historical Productivity Measurement

Deliverable Size

is

over

Time

Productivity

Effort

and

Page 49: Agile Release Planning, Metrics, and Retrospectives

47

Copyright QSM Associates, Inc. 10/22/13 93

Size = 272,768 SLOC

Effort = 249 Person-Months

Time = 13 Months * WFS 5.1

PI = SIZE

TIME EFFORT = 19.4

Step 1 - Derive Productivity Index from History (preferably more than 1 project)

Copyright QSM Associates, Inc. 10/22/13 94

PI

Direct Reading from SLIM-DataManager

Page 50: Agile Release Planning, Metrics, and Retrospectives

48

Copyright QSM Associates, Inc. 10/22/13 95

Step 2 - Identify Proposed Time and Effort

Time to June Deadline = 6 Months"Budgeted Effort = 24 Person-Months"

Project Staffing Profile

2 3

5 5 5 4

0 1 2 3 4 5 6

Jan Feb Mar Apr May Jun

Copyright QSM Associates, Inc. 10/22/13 96

Step 3 - Derive Size Implied by PI

Deliverable Size

is over Time at some Productivity Effort

Page 51: Agile Release Planning, Metrics, and Retrospectives

49

Copyright QSM Associates, Inc. 10/22/13 97

Step 3 – Determine (triaged) Size

What PI?

What Size?

Copyright QSM Associates, Inc. 10/22/13 98

Step 4 - Map to Functional Size Units

! Based on resultant size, translate that down to additional size units, such as number of features, technical requirements, stories, story points, etc.

! For example, if typically, it takes 2 objects per technical requirement (from observed history), then try to promise no more than 2X objects – or X technical requirements - in a 6 month time frame.

Page 52: Agile Release Planning, Metrics, and Retrospectives

50

Copyright QSM Associates, Inc. 10/22/13 99

Exercise #4

Release and Iteration Planning

Copyright QSM Associates, Inc. 10/22/13 100

Objectives of this Exercise

!  You will learn how to: ! Look at productivity patterns for a group of completed

projects. ! Determine what historical productivity values are relevant to

help estimate a new project. ! Given a target deadline, �reverse calculating� the size/scope

that would be possible within the schedule and allocated effort. You will do this in terms of Units of Work (C++ Objects) and Units of Need (Technical Requirements).

! If the project can not deliver the desired amount of (full) functionality, you�ll understand how to start negotiating for additional time, effort, or how to make the case for reduced scope (or incremental releases), by making a data-driven argument.

Page 53: Agile Release Planning, Metrics, and Retrospectives

51

Copyright QSM Associates, Inc. 10/22/13 101

For Additional Information

Michael Mah email: [email protected] website: www.qsma.com blog: www.optimalfriction.com twitter: @michaelcmah Tel: 1 413-499-0988 Andrea Gelli Email: [email protected] Website: www.qsm-europe.com Tel: +41 79 379 9807