Project Performance MetricsProject Performance Metrics

Jason Uyder, MSIS, PMP, ITILFireman’s Fund Insurance Company

February 16, 2012

Definitions

Metric: A system of related measures that facilitates the

quantification of some particular characteristic

Metic: A standard of measurement

Mathematics A geometric function that describes the distances between pairs of points in a space.

DefinitionsDefinitions

Quality The standard of something as measured against other

things of a similar kind; the degree of excellence of something: "quality of life".

General excellence of standard or level: "quality beers".

Quality The delta between expectations and experience – (Unknown)

Quality Metric The quantifiable measurement of the delta between

expectations and reality

ImportanceImportance

As we all know, having good solid metrics and measures is ‘key’ for every Project:

How else would we report status?

Know when the project is complete?

Know if the project was successful?

Challenges (1)Challenges (1)

Projects often defined or characterized by IT activities: Implement new system, install software

When really the project is to improve a business activity/function: Improve Customer Service, gain market share, increase

efficiency

We often assume good project pre-work was done and if that we complete project activities we will realize the business benefit.

Challenges (2)Challenges (2) People sponsoring the project don’t understand

project management

Don’t understand the ‘core’ business and/or the ‘core’ processes well enough

As Frank said in December – ‘Today people seem to value decision making more than analysis’

You may be challenging assumptions and beliefs, forcing folks to confront reality

Often forces a culture change.

Challenges (3)Challenges (3) Often Metrics Programs are viewed as compliance

activities

Make sure you understand the driving force behind the desire for Metrics ‘Figures don’t lie but liars can figure’

Metrics are the ‘shoe horn’ into the project, the real value is ‘action’ and ‘take away items’

Have a Charter for the program, treat the program as a project, have your own CSF, Communication Plan, Remediation Plan

General Quality MetricsGeneral Quality Metrics

Product Metrics describe characteristics of a product such as it’s size, complexity, features, and performance.

Common product metrics are mean time to failure, defect density, customer problem, and customer satisfaction metrics.

Process Metrics are strictly for evaluating & improving the effectiveness of development and maintenance processes.

Product MetricsProduct Metrics Critical Success Factors

Why we are doing the project Can be challenging to measure

Lack of data, common understanding of what and how to measure

Can be financial, but should be measurable in the short term

Not usually ROI, should be linked to CBA

Mostly operations based measurements

Should provide clear evidence of success

In Service Based Economy - Transactions are often the focus

© Copyright Allianz 10

Pro

cess

Fin

anci

al

Criteria Measurement Frequency Baseline Target Results

Reduce the cycle time

End to end cycle time – (Point at which a candidate starts an application to the time rejected or licensed)

Sample @ 90% Confidence Level

Monthly

Average = 34 Days

50% Reduction in

averageCycle time

TBD

Reduce the number of steps in the process

Number of Process Steps (End to End)

Confirm process steps at

deployment and when in steady

state

57 Current Steps50%

ReductionTBD

Reduce staffing costs in the

Agent Licensing Department by $500,000 and

improve service

Departmental budget 1.5M RR

Annual 1.5M RR1M RR after project costs

TBD

Pro

cess

(S

pee

d &

A

ccu

racy

)Critical Success Factors

A limited number of CSF is crucial, too many usually means you are trying to do too muchdon’t really know what you’re doing

Critical Success FactorsCritical Success Factors Need Drivers CTQs

How long does it take to Fill out Paperwork

User Friendly Forms Minimal Redundancy

Easy Access To Current Forms

Minimal Data Entry

(# of Fields) Quick Response Complete/Legible forms

Technology to speed investigation

Minimal Handling

Fast/ Reliable/Consistent & Cost Efficient Agent Licensing Process

Consistent Decision

Process Localize Decision Making

Minimize Special Case Decisions

VOC’, Root Causes & CTQs

Collected the ‘VOC’ which determined that due to the industry compensation model that getting an Agent Licensed & Appointed quickly greatly increased the likelihood of individual Agent success.

Therefore, Speed & accuracy became this projects CSF:

How fast ‘cycle time’ and consistently ‘Variance’ can we process Agents through the Licensing process correctly.

What are the CTQs for a fast, reliable ….start with the Drivers and drill down to specific measurable CTQsIT Systems won’t address these CTQs

Critical Success Factors

Average Cycle Time WentFrom 34 Days to 16 Days

PROCESS STEPS 57 8

“HANDOFFS” 34 6

AVERAGE TIME SAVED PER APPLICANT: 2 1/4 HOURS

H.O. STAFF REDUCTION OF 6 DUE TO PROCESS IMPROVEMENT

THE ABILITY TO ACCESS INFORMATION AND WORK ANYWHERE IN THE WORLD WITH AN INTERNET CONNECTION

Critical Success FactorsCritical Success Factors

© Copyright Allianz 13

Criteria Measurement Frequency Baseline Target Results

Reduce the cycle time

End to end cycle time – (Point at which a candidate starts an

application to the time rejected or Licensed)

Sample @ 90% Confidence Level

MonthlyAverage = 34 Days

50% Reduction in averageCycle time

Pass New Average 16 Days

Reduce the number of steps in the

process

Number of Process Steps (End to End)

Confirm process steps at deployment and when in steady

state

57 Current Steps 50% ReductionPass

New Process has 8 steps

Reduce staffing costs in the Agent

Licensing Department by $500,000 and

improve service

Departmental budget 1.5M RR Annual 1.5M RR1M RR after project costs

TBD – 6FTE reduction

($450K Salary Reduction + )

Process MetricsProcess Metrics Project Quality and Process Compliance Metrics

How is the project performing as a process, often called projects status or included in project status reports – Fairly Standard Project Metrics:

Schedule Performance Index Cost Performance Index Deliverable Timeliness (Monthly) Deliverable Timeliness (Cumulative) Change Request Impact Defects per 1K Hrs Development Testing Effectiveness % Pre-Production Defects

Program/ProjectQuality

Deliver On Schedule

Deliver Consistently & Predicatively

Deliver On Budget

Are we meeting our schedule?

Are we managing scope effectively?

Are deliverables completed on time?

Are we on budget?

Are the estimatedefforts (costs) accurate?

Are we catching problems (defects) when we should?

Is the peer review processfinding or ‘heading off’ issues?

üSchedule Performance IndexüChange Request ImpactüDeliverable Timeliness

üCost Performance IndexüTesting Effectivenessü%of Pre-Production Defects IdentifiedüDefects per 1k Hrs of Code DevelopmentüPeer Review Effectiveness (Start with New Projects)

Does the code sufficiently meet our requirements?

Initial Project Quality ScorecardInitial Project Quality Scorecard

Project Quality and Process Compliance MetricsProject Quality and Process Compliance Metrics

Project/Release

Findings Action Items Outcomes

Proj A CPI is over target at 1.13, however it has come down .03 from last month. The root causes of the high CPI were overestimates of requirements and design activities and resources not applying time to correctly to activities.

None required at this time. Since refresher trainings and reminders on the importance of proper timekeeping activities, CPI has trended back down to close (.03) to targets.

Proj B Change Request Impact is currently at 13%. The majority of this score is the result of PCR#3 (Scope Realignment) 1100 Vendor hrs + 300 internal hours.

The release is creating a mitigation plan and looking at the potential impacts/dependencies to the overall E4 Release.

TBD

Ultimately the real value is in the ‘action items’ and ‘take away items’ !

How can you influence and improve the project and the project management practices in your organization ?

Mentor & Coach

UpdateUpdate

Not yet available Not applicable or not available for release

* PCR to Budget Ratio data is go-forward from 2/1/11 only

Neutral

Trending

Quality & Performance Program Expansion and Continuous Improvement

February 2009 - first quality scorecard produced for XXXX with only 11 scores and several metrics that were not included or TBD.

• Project & Program leadership meet monthly to review findings, discuss remediation plans, and most importantly gain a better understanding of how to manage and improve project performance with quantitative data.

• Scores that are out of range are discussed, root causes identified, and where warranted, process improvement initiatives are implemented.

• Remediation plans are in place to ensure process compliance, and overall performance and related scores. Lunch and Learn sessions are provided, as well as various training sessions in an effort to minimize the learning curve for the metrics program.

Feb 2010 May 2011

All Division programs are now represented with a more comprehensive suite of KPIs. Enterprise Systems have completed a calibration period and will be included going forward.

Quality & Performance Program Expansion Quality & Performance Program Expansion and Continuous Improvementand Continuous Improvement

Yes – This is really hard, but should be really simple !

You may be challenging assumptions and beliefs, forcing folks to confront reality

Often forces a culture change, or highlights the needs for one

Can be a double edge sword – make sure leaders really want to know

As companies ‘tighten their belt’ this type of analysis becomes more valuable or is perceived as more valuable

SummarySummary