C prime webinar-ppt-validating agile

Presenter: Jeff Howey – Agile Coach, CSM

4100 E. Third Ave, Suite 205, Foster City, CA 94404 | 650-‐931-‐1651 | www.cprime.com The leader in training and consulGng for project management and agile development

Agile Performance Metrics

© 2012, cPrime. All Rights Reserved


!   If you have a quesGon, please type it in the “QuesGons” Chat Window. Our team will collect quesGons and include responses to many of them when we post the material online.

!  We will post a copy of the presentaGon, a recording of the webinar and follow-‐up informaGon on www.cprime.com !   We will send email with links !   Feel free to share!


!  What is the current adopGon of Agile processes in your organizaGon? !   PracGcing Agile processes on most/all projects !   PiloGng Agile on select projects !   Considering Agile for future projects !   Not considering Agile in the foreseeable future


And What Metrics Will Demonstrate Impact?


!  Don’t we value working so/ware over comprehensive documenta6on?

!  Why do I need tools and processes if our focus is on individuals and interac6ons?

!   Is management measuring my performance based on how many story points I complete when my goal is to increase customer sa6sfac6on?

!  Do my stakeholders really care about numbers as long as I am responsive to my customer’s requests?

-‐ Excerpts from the Agile Manifesto


!   The OrganizaGon as a whole needs to understand how and why Agile is making a difference in project delivery

!   The Agile Team needs insight into areas they can improve

!  Agile sGll values processes, tools and documentaGon!

!  No more “fake it, ‘Gl you make it” jusGficaGons !   You need to “move it, ‘Gl you prove it” !   And keep proving it


Data from 50,000 projects Copyright © by The Standish Group International, Inc.

Challenged •  late (100% median) •  over budget (50% median) •  lacking funcGonality •  low quality

1994 1996 1998 2000 2004 2006 2009 2010

% Success 16 27 26 28 29 35 32 37

% Challenged 53 33 46 49 53 46 44 42

% Failed 31 40 28 23 18 19 24 21

0 10 20 30 40 50 60

% 50

,000 P

rojec

ts

Standish Chaos Reports Project Success

37% Success

42% Challenge

21% Fail

2010 Project Success


!  How would you rate your team’s history of delivering projects? !   More successes than failures !   More failures than successes !   About the same as Standish reports !   I’d rather not say


7%

13%

16%

19%

45% 64%

Feature Usage by Frequency

Always

Ohen

SomeGmes

Rarely

Never

64% of product features are rarely or never used 2006 – The Standish Group

How much did this cost???


•  The average Fortune 500 sohware project costs $3.2 million and takes 1,290 (resource) months to complete. Yet only 37% of respondents answered "yes" when asked if their projects met users' needs

•  2008 – Voke, Inc.


!   “In 2002, agile projects made up less than 2% of overall projects and less than 5% of new applicaGon development projects. Today, agile projects account for almost 9% of all projects and 29% of new applicaGon development projects...

The increase in project success rates can directly @e back to projects resolved through the agile process.”

2011 Chaos Manifesto Copyright © by The Standish Group International, Inc


!   Predictability !   Delivery Date !   Delivery Content

!   Reliability !   Of EsGmates !   Of Requirements

!  Quality !   IniGal Quality !   Cost of Defect ResoluGon

!   Customer SaGsfacGon


!  Understand History !   What is your personal best/worst/average in each

measure?

!   Set a Baseline Target !   At least aim to improve 1-‐3 measures in the next

3-‐6 months without hurGng any others metrics !   Avoid serng aim to improve everything

immediately

!   Set Future Targets !   ConGnue improvement of key measures !   Target to improve some others



!  How ohen do you make major releases? !   Monthly or more frequently !   Quarterly !   Every 6-‐12 months !   Once a year or less frequently


!   Somewhat related to Reliability of EsGmates but disGnctly different !   Target dates should be met without “heroics” of the team

!   Baseline from History !   What % of projects deliver within a range +/-‐ the iniGal

date? !   By how much are projects delivered late when they are not

within a given range?

!   Set Goals! !   Somewhere between “best” and “average” history !   Definitely beter than “worst”


Average Variance of Release Date from Plan !  Mean: 35% late !  Median: 33% late

What is your reality?

Project ID Delivery Plan Date Actual Delivery Date

Delta from Plan Total Days in Project

1 3/31/2009 4/30/2009 +30 days 180

2 12/31/2009 12/31/2009 0 days 180

3 3/31/2010 7/31/2010 +210 days 480

4 3/31/2010 5/31/2010 +60 days 60

5 6/30/2011 7/15/2011 +15 days 180

6 6/30/2011 12/15/2011 +165 days 480

7 1/31/2012 4/30/2012 (est) +90 days 90*


!   First, understand WHY target dates have been missed in the past !   What can Agile address? Uncertainty or Changing

Scope in Requirements, unrealisGc dates for full scope compleGon, etc.

!   What can Agile not solve? Corporate culture, ritual-‐intensive release processes, etc.

!   Brainstorm soluGon opGons !   PrioriGze opGons to pursue !   Inspect, Adapt and maintain Transparency while implemenGng soluGon alternaGves


!  Define a Target % of Releases that make the date !   Adjust scope based on priority to deliver

funcGonality that is good enough ON the planned date

!  Define a beta-‐type release strategy where feasible !   Especially in organizaGons with long/complex

release procedures for producGon availability !   IdenGfy where can you release a beta version and

receive valuable feedback



!   Comparison of !   IniGal Requirements vs. Delivered Requirements:

What % of funcGonality was removed from scope? !   Were undelivered requirements rescheduled for later

release? !   Were features given up out of frustraGon?

!  Not always bad to deliver less than planned !   Were features not needed? !   Were they simplified -‐ but sGll “good enough” for users?

!   If it was actually a “savings” or diverted focus to higher prioriGes, it could be a benefit


!  How many implemented stories were rejected by the Product Owner at a Sprint Review?

!  What were the Drivers? !   incomplete requirements? (Product Owner has

opportuniGes to improve) !   incomplete implementaGon? (Agile Team has

opportuniGes to improve)


!  Agile Teams -‐ Set goals around the number of stories that are rejected from the Product Owner once implemented !   Is the right number for you 80%? 95%?



!   Look at Story Points first !   Aher implementaGon, perhaps during

RetrospecGves, replay Planning Poker® !   “Knowing what you know now, what would be the

esGmate for this story” !   Measure all stories (big and small) to keep fair track items

!   Compare and Average variances !   Break down by Small/Medium/Large !   Is the team beter at esGmaGng in one area vs. others?

!   Adjust the esGmaGng process as needed


!  As the team refines their EsGmaGng Process, conGnue to review the Product Backlog and re-‐esGmate upcoming User & Technical Stories !   Review the “Top 1/3” of the Product Backlog if Gme

allows for the first Release or first 5-‐7 Sprints of a new project or new team composiGon

!   Reset and baseline Story Points, conGnue to track progress toward beter esGmaGon

!   Keep this grooming process short! !   PerfecGon is expensive and unatainable !   Set limits (1 hour per Sprint during Story Review

MeeGngs, perhaps?)


!  At the Sprint Backlog (hours) level vs. the Product Backlog level (Story Points) !   IdenGfy any recurring procedures that may be

candidates for project standards and assign hours esGmates to include in future Sprints

!   Compare Actual:EsGmated hours for tasks and understand variances !   They are expected, so idenGfy paterns and don’t dwell

for more than 5 minutes on specific reasons !   For instance, “programming stored procedures are

always underesGmated” or “Joe is new to Java so takes 25% more Gme than Sally does at coding right now”

!   Adjust esGmaGng process to account for paterns



!  What % of Stories are prioriGzed for change to the iniGal implementaGon in the following 3 Sprints? Why? !   Focus criGcal atenGon on Stories that are

prioriGzed for change in the Sprint immediately following iniGal implementaGon

!   Product Owners – This really is one of your biggest measures !   Does this indicate you need to do more research or

refinement of Stories prior to Sprint Planning?



!   Just kidding – 0 defects is expensive and likely indicates shortcomings in the tesGng process!

!   IdenGfy historical number of defects idenGfied post-‐release by priority (P1, P2, etc.)

!   Set goals by Release to drasGcally reduce the number of defects idenGfied post-‐release

!   Set goals and standards to minimize or eliminate high-‐priority defects to consider a Sprint “done” !   For example, the Agile Team sets a standard that

they cannot exit a Sprint with outstanding P1’s



!   Track at an Aggregate Level and begin to IdenGfy where defects are being resolved or (ideally) prevented !   Agile Teams & Product Owners are all on the hook!



!  UlGmately this is the only metric that used to compare Agile Teams !   Story Points for Blue Team <> Story Points for Green Team !   Velocity for Blue Team <> Velocity for Green Team !   Defect Rate for Blue Team <> Defect Rate for Green Team

!   IdenGfy your customers – Internal & External !   Track what is important to Your Customer!

!   Time to Market !   Length of Visit per page !   Click-‐through Rate

!   Ideally track and monitor per Release


!  Use a 4-‐point Scale vs. 5 or 10 !   Forces a “3” or “4” RaGng and avoids “Average” !   4 keeps it a simple comparison vs. 10 !   Words like “Awful, Could Be Beter, It Works, Great”

are more powerful than “Poor, Fair, Good, Excellent”

!   Be sure to capture comments! !   Allow input on every quesGon, not just a generic

feedback text entry secGon !   AcGvely Solicit! “Tell us what could be beter” is

more powerful than “Comments”


!   Record feedback and scores for future reference !  During RetrospecGve and Story Review meeGngs, review the feedback and keep it fresh

!  Adapt funcGonality, schedule or whatever elements are important to the Customer and given as feedback

!   Incorporate “What could we do beter?” feedback into features or processes



!  Defects per developer !   Legacy performance evaluaGon criteria !   NOT valid with Agile: self-‐organizing teams should

allow individuals to learn new things – the “test-‐n-‐learn” process anGcipates defects

!   Lines of Code/Story Points per developer or team !   My 5-‐point Story is NOT your 5-‐point Story (this is

true at the individual and team level)

!  What other metrics do you think are missing? !   Compare to the Agile Manifesto and Principles to

validate their use


!  Once goals are set for any set of metrics, these steps are CRITICAL !   ConGnue to measure against goals !   Review as an Agile Team !   Share with Product Owners, Management,

Customers

!   ConGnuously Refine goals !  Do not tackle everything at once!

!   PrioriGze what is most important to the Agile Team and the Customer to address first

!   Take on new challenges when goals are met



!  How much money is 6ed up in projects that are approved but delayed or “on deck”

!   Can we prove that releasing more frequently based on prioriGzed features helps the organizaGon more wisely invest in Products & Projects?

!   This is tricky to measure and even more tricky to track, but it is CRITICAL


!  Do you have projects that are approved, but are overdue, delayed or sirng "on deck" without resources to work them? !   Yes, and maybe Agile can help us manage our

investments beter !   Yes, but I don’t think Agile would help !   No, we only allocate budget for projects when

resources are ready to go !   None of the above quite fit


!  As organizaGons implement Agile, it is criGcal to track the $ invested in projects and the speed at which the investments are realized through Product Releases

!   Supports buy-‐in and evangelizaGon of Agile benefits where Agile makes sense


!  How much money has been spent on the 64% of features that are rarely or never used? Can Agile put that to beter use in the future?

!  Do you track usage of features? !   Through automaGon in the applicaGon !   Via User Survey

!   Product Owners need this informaGon to prioriGze work. IT/Project Management needs this to understand impact on Por�olio


Create Your Own, but Here are Some Ideas


Metric DescripIon Target *

Sprint Predictability FracGon of Sprints in which all planned Stories are completed

> 90%

Deliverable RejecGon Rate

Number of Stories rejected by Product Owner at end of Sprint

< 1

Requirement RejecGon Rate

Number of Stories rejected in Sprint Planning MeeGng

< 1

EsGmaGon Reliability

Accuracy of Sprint Backlog esGmate ±20% of hours in 80% of Sprints

Tracking Reliability No. of Tasks with incorrect Status per day < 5%

Defect Impact Effort / Sprint for P0, P1 ProducGon bugs (i.e., more effort devoted to value delivery)

< 20%

Quality P0, P1 ProducGon bugs, per Release 3X reducGon

Cost of Defects Cost to correct ProducGon bugs, per Release 3X reducGon

* Example Goal: “The Team aims to achieve these aOer 3 Releases”


Metric DescripIon

Time to Market Time from request to implementaGon for highest-‐value features

Rate of Value Delivery Value delivered per quarter / release

Adaptability How well / frequently do we change course successfully, when needed?

Predictability Extent to which we deliver planned features on schedule (excepGng change requests)

Customer SaGsfacGon How well are we meeGng customer needs?

Customer InteracGon How well do we involve customers in defining needs and gerng feedback?

Internal CollaboraGon How well are we meeGng internal needs of various departments? (Support, Sales, etc.)

These are more tricky to track but very useful


!  How did we do? !   I learned quite a few things that I can use !   A good reminder of the basics and a few new ideas !   I’m not sure how I would apply most of what I

heard !   Waste of Gme

Business

C prime webinar-ppt-validating agile