Lessons Learned in Project Metrics · Lessons Learned in Project Metrics: Are Your Metrics Dumb or Smart? My Background •Project Manager and Trainer •>20 years IT experience on

Leading Answers Inc. - Lessons Learned in Metrics

1Copyright © 2010 Mike Griffiths. All Rights Reserved www.LeadingAnswers.com

February 16th, 2010

Software QA Discussion Group

Mike Griffiths, PMP

Leading Answers Inc.

Lessons Learned in Project Metrics:Are Your Metrics Dumb or Smart?

www.LeadingAnswers.com

My Background

• Project Manager and Trainer• >20 years IT experience on utilities, defense, & finance

• 8 years PMO Agile-to-Traditional Integration

• Agile Project Management• Helped create Agile method DSDM in 1994

• 16 years agile project experience (DSDM, XP, Scrum, FDD)

• Board director of Agile Alliance and APLN

• Author, trainer, and presenter Agile Conference 2001-9

• Traditional Project Management• PMP, PRINCE2 certifications

• PMBOK v3 contributor and reviewer

• Trainer for PMI SeminarsWorld 2005-2010

• Presenter PMI Global Congress 2004-2009



• 1 <Star>

• 2

• 3

• 4

• 5

Agenda

4) Measuring Up

3) “Design Factory” Metrics

2) The Hawthorne Effect

1) Measurement Troubles

6) Smart Metric examples

5) Leading vs. Lagging Metrics

Metrics are like fire…



Measurement

Not all observations are useful: • The sun rises up in the sky in the morning and then falls down again at night

• Planets revolve around the earth

• Stars come out at night

• Heavier objects fall faster than lighter objects

”There are so many possible measures in a software process that some random selection of metrics will not likely turn up something of value” – Watts Humphrey

Not all useful things are easily observable: • Spouse’s mood

• Sponsor Confidence

• Team Commitment

"Not everything that can be counted counts, and not everything that counts can be counted“ – Albert Einstein.

The Hawthorne Effect

You will influence what you measure...



Metrics Should be:

1. Simple, self-generating

2. Relevant to the end-goal

3. Ideally, leading, future-focused

Design Factory Metrics

Traditional Metrics Examined

Desirable Characteristics: The Hawthorne Effect is positive

Simple, self generating

Relevant to the end-goal

Leading, future-focused

Traditional Metrics:• Lines of Code Written – poor, does not reward

simplification, leads to code bloat

• Function Points Delivered – poor, effort to generate, not relevant to the end-goal of project

• Hours Worked – poor, leads to long hours, burn-out, defects, consumed budgets



Smarter Project Metrics

Since you will influence what you measure…

Choose Smarter Metrics:

• Features Accepted


• User Satisfaction

• Defect Cycle Times

Smarter Metrics


Not StartedIn ProgressAccepted

Scope (points) IPS Project Progress Spend

0

500

1000

1500

2000

2500

3000

1/10

/06

1/12

/06

1/2/

07

1/4/

07

1/6/

07

1/8/

07

1/10

/07

1/12

/07

1/2/

08

1/4/

08

1/6/

08

1/8/

08

1/10

/08

1/12

/08

1/2/

09

1/4/

09

1/6/

09

1/8/

09

1/10

/09

1/12

/09

Scope BuiltAccepted

C:/ConcordApril16.JPG



Smarter Metrics


0

50

100

150

200

250

300

350

400

450

Jan Feb Mar Apr May Jun Jul Aug

Time

Featu

res

Not Started

In Progress

Completed

Not StartedIn ProgressAccepted

Feature Based Reporting

Example 2: Parking lot Diagrams

Feature Set Status: Enter Order

Details

(15)

24%Percentage Complete:

Not Started

Work in Progress

Completed

Attention (i.e. behind)

Percentage Complete Bar

Mar 2010Target Completion Month:

Not Complete Complete

FB Chief Programmers Initials

Feature Set Name

Number of Features in

the Feature Set

The Feature Set called Enter Order

Details has 15 features and is

currently 24% complete. It is due to

be completed by March 2010

C:/ConcordApril16.JPG



Create New

Order

(5)

100%

Capture

Customer

Details

(9)

75%

Enter Order

Details

(15)

24%

Process

Payment

(11)

Create New

Customer

(8)

55%

Amend

Customer

Details

(6)

20%

Archive

Customer

(4)

Stock

Search

(6)

95%

Item

Details

(12)

75%

Order Processing

Customer Management

Inventory Management

Jan 2010

Feb 2010

Mar 2010

Apr 2010

Jan 2010

Feb 2010

Feb 2010Feb 2010

Mar 2010

CM DH LF RS NC

AW

KB

SW SW

Feature Based Reporting

Smarter Metrics

User Satisfaction



Smarter Metrics


Smarter Metrics

• Defect Cycle Time



• “You get what you measure”

• “You get only what you measure, nothing else”

• “You tend to loose the things that you can’t measure: insight, collaboration, creativity”

• Nucor Steel - From startup in 1970 to North America’s largest and most profitable steel company

• Incentive pay based on productivity

• Measure one level above where expected / directly controllable

Measuring Up

• Defects could be traced back to individual developers, but may be environmental

• Rolling up to an entire team and get testers involved earlier to provide more timely and valuable feedback to developers

“Instead of making sure that people are measured within their span of control. It is more effective to measure people one level above their span of control. This is the best way to encourage teamwork, collaboration, and global, rather than local optimization” – Mary Poppendieck

Measuring Up in Software



• For an accountant, a perfect view of the past might be useful

• For a project manager, a perfect or even an imperfect view of the future is far more useful

Leading vs. Lagging Metrics

Pay less attention to Lagging Metrics

• Actual values

Pay more attention to Leading Metrics

• Trends

• Impacts of projections

Trends

CR = Change

Requests

Clarif. =

Clarifications

Observations Mar 3 Mar 10 Mar 17 Mar 24 Mar 31

Defects Opened 5 25 30 20 10

Defects Closed 1 16 35 22 15

Defects Remaining 4 13 8 6 1

CR Opened 0 18 20 23 12

CR Closed 0 11 21 16 9

CR Remaining 0 7 6 13 16

Clarif. Opened 9 12 14 8 2

Clarif. Closed 6 14 11 9 5

Clarif. Remaining 3 1 4 3 0

Total Observations 7 21 18 22 17

Observation Trends

0

2

4

6

8

10

12

14

16

18

Mar 3 Mar 10 Mar 17 Mar 24 Mar 31

Defects Remaining

CR Remaining

Clarif. Remaining



Trends - Risks ReducedJan Feb Mar Apr

ID Risk Short Name Imp Prob Sev Imp Prob Sev Imp Prob Sev Imp Prob Sev

1 JDBC driver performance 3 2 6 3 0 0 3 0 0 3 0 0

2 Calling Oracle Stored Procs via web service 2 2 4 2 0 0 2 0 0 2 0 0

3 Remote app distribution to PDA's 3 2 6 3 1 3 3 0 0 3 0 0

4 Oracle Warehouse Builder stability 2 2 4 2 3 6 2 2 4 2 0 0

5 Source system availability 2 1 2 2 1 2 2 0 0 2 0 0

6 Access to user community 2 1 2 2 2 4 2 2 4 2 1 2

7 Availability of Architect 2 2 4 2 3 6 2 2 4 2 0 0

8 Server Upgrade necessary 1 2 2 1 1 1 1 0 0 1 0 0

9 Oracle Handheld Warehouse Browser Launch 3 1 3 3 1 3 3 3 9 3 1 3

10 PST changes for BC 0 0 0 0 0 0 2 2 4 2 1 2

33 25 25 7

Project Risk Profile

0

5

10

15

20

25

30

35

Jan Feb Mar Apr

PST changes for BC

Oracle Handheld Warehouse Browser Launch

Server Upgrade necessary

Availability of Architect

Access to user community

Source system availability

Oracle Warehouse Builder stability

Remote app distribution to PDA's

Calling Oracle Stored Procs via web service

JDBC driver performance

Analyst UI Designer Code & UT Sys. Test

45 30 40 35

• Identifying constraints

• Buffering

• Managing flow

Cycle Times



Cycle Times

Construction Cumulative Flow

0

20

40

60

80

100

120

140

20/02/

2004

27/02/

2004

05/03/

2004

12/03/

2004

19/03/

2004

26/03/

2004

02/04/

2004

09/04/

2004

16/04/

2004

23/04/

2004

30/04/

2004

07/05/

2004

14/05/

2004

21/05/

2004

28/05/

2004

Total Features

Analysis

UI Design

Code & UT

Sys Test

}Widening

Area

Bottleneck Activity = “UI Design” (activity below widening area)

Little’s Law

Little’s Law:

Cycle times are proportional to queue lengths.

(We can predict completion times based on queue size)



Task Boards & Kanban

Summary

1. Use metrics wisely

2. Leverage the Hawthorne Effect

3. Measure Up

4. Favour Leading Metrics

More info and slides: www.LeadingAnswers.com

Email: [email protected]

Documents

Lessons Learned in Project Metrics · Lessons Learned in Project Metrics: Are Your Metrics Dumb or Smart? My Background •Project Manager and Trainer •>20 years IT experience on