Debra J. Perry Harris Corporation · Debra J. Perry Harris Corporation. How Do We Get On The Road To Maturity? assuredcommunications!" NDIA Conference - 2 12-15 November 2007 What

assuredcommunications™How Do We Get On The Road To Maturity? NDIA Conference - 112-15 November 2007

How Do We Get On The Road To Maturity?

Debra J. PerryHarris Corporation

http://www.pdfcomplete.com/cms/hppl/tabid/108/Default.aspx?r=q8b3uige22


What do we want?

From this…From this…

To this…To this…

But how?But how?



Where Do We Start?



Government Communications Systems Division: What We Do…

Space and ground satellite Space and ground satellite communications systemscommunications systems

Operations and support servicesOperations and support servicesIntelligence, surveillance, and Intelligence, surveillance, and reconnaissancereconnaissance

Aviation electronicsAviation electronics Communications and information Communications and information networksnetworks

We innovate, integrate, and manage technology.



Roadmap to Getting There 1 of 3

• ROI Need– Convince management of payoff– Convince other stakeholders management is

convinced– Convince other stakeholders process improvement

helps them




• ROI Need– Convince management of payoff– Convince other stakeholders management is convinced– Convince other stakeholders process improvement helps them

• Convince Management– Share process improvement vision– Provide external ROI

• SEI data• Other companies’ data

– Provide internal ROI• Harris examples• Other possibilities




• ROI Need– Convince management of payoff– Convince other stakeholders management is convinced– Convince other stakeholders process improvement helps them

• Convince Management– Share process improvement vision– Provide external ROI

• SEI data• Other companies’ data

– Provide internal ROI• Harris examples• Other possibilities

• Convince Other Stakeholders– Show management support– Educate and train– Show value for them



ROI for Management

• ROI – Return on Investment• Convince Management of Payoff

– External Research• Read and Research • Attend SEI Courses• Hire consultants

– Internal Research• Develop a plan• Develop ROI

– Educate management– Start or improve your process group



The Process Management Premise

• The Quality of a System is Highly Influenced by the Quality of the Process Used to Acquire, Develop and Maintain It

• This Premise Implies a Focus on the Processes as well as on Products:– This is a long-established premise in manufacturing– Belief in this premise is visible worldwide in quality

movements in manufacturing and service industries (e.g., ISO standards)

– This premise is also applicable to development

(Introduction to CMMI Version 1.2)



Ad Hoc Processes

• Processes are Not Well Defined and Improvised by Practitioners and their Management

• Process Descriptions are Not Rigorously Followed or Enforced

• Performance is Highly Dependent on Current Practitioners• Understanding of the Current Status of a Project is Limited• Immature Processes Result in Fighting Fires:

– There is no time to improve – instead practitioners are constantly reacting

– Firefighters get burned– Embers might rekindle later




Improved Processes

• Process Descriptions are Consistent with the Way Work is Actually Done

• They are Defined, Documented and Continuously Improved

• Processes are Supported Visibly by Management and Others

• They are Well Controlled – Process Fidelity is Evaluated and Enforced

• There is Constructive Use of Product and Process Measurement

• Technology is Introduced in a Disciplined Manner




Benefits for Improving Processes

• Processes Enable you to Understand What is Going on

• People Develop their Potential More Fully and are More Effective Within the Organization

• By Defining, Measuring and Controlling the Process, Improvements are More Successful and Sustained

• The Likelihood that Appropriate Technology, Techniques and Tools are Introduced Successfully Increases

• More Benefit Information is Available in the August 2006 SEI Technical Report, Performance Results of CMMI-Base Process Improvement (CMU/SEI-2006-TR-004) at:http://www.sei.cmu.edu/publications/documents/06.reports/06tr004.html




ROI At Different Levels

• Process Improvement Start Up• Reaching Next Level of Maturity• Out of Phase Defect Removal• Reach Higher Levels of Maturity• Optimization and Maintenance

RO

I Diff

icul

ty

Profiles of Level 5 CMMI Organizations“Little has been done to study the return on investment (ROI) of high maturity organizations that have reached Level 5”

Cos

ts

ROI Should be More Specific as you Move Up in Maturity



ROI for Other Stakeholders 1 of 2

• Convince Other Stakeholders Management is Convinced– Show management is investing in process

improvement– Show management is enforcing compliance

Management should fund and provide oversight for process improvement efforts and review and reward process compliance.



ROI for Other Stakeholders 2 of 2

• Convince Other Stakeholders Management is Convinced– Show management is investing in process

improvement– Show management is enforcing compliance

• Convince Other Stakeholders Process Improvement Helps Them– Relieves chaos– Reduces defects and rework– Facilitates improvements– Saves time and money

Toyota makes process improvement a way of life.



Convince Management

• Share Process Improvement Vision• Provide External ROI

– SEI data– Other companies’ data

• Provide Internal ROI– Harris examples– Other possibilities



Share The Vision

• Find a Strong Leader with Clout and Credibility• Develop a Vision – What Will Motivate?

– Fewer problem programs– Faster, cheaper, better– Need CMMI to compete

• Develop a -Plan• Determine Estimated ROI• Present the Plan to Management• Be Determined!• Get Management Commitment



Example Process Improvement Vision

• Enhance the Current Processes to:– Improve efficiency and value for all stakeholders– Reduce cost and rework– Provide predictable program execution– Increase competitive advantage



Process Improvement Steps

• Establish Process Group, if Possible• Document Current Processes• Institutionalize Processes• Determine Process Improvement Goals

– Short term goals (start with PP, PMC, MA)– Long term goals (add PAs, plan for SCAMPI)– Estimate ROI

• Prioritize Process Improvement Goals• Develop Process Improvement Plan

– Determine process improvement measurements– Begin measurement collection efforts



Performance Results of CMMI®-Based Process Improvement 2006

224.0:11.7:127.7:1Return on Investment

714%-4%55%Customer Satisfaction

3448%2%132%Quality2061%11%329%Productivity2250%2%95%Schedule2934%3%87%Cost

Number Data PointsMedianLowHighImprovements



LM Customer Satisfaction: Award Fees

Customer Satisfaction Continues to ImproveCustomer Satisfaction Continues to Improve

Award fees increased by 55% compared to an earlier SW-CMM ML2 baseline.

SW CMM L2 SW CMM L3 SW CMM L4 SW CMM L5 CMMI L5

Per C

ent

SW CMM L2: 1993-1995SW CMM L3: 1996-1997SW CMM L4: 1998-1999SW CMM L5: 2000-2001CMMI L5: 2002

PotentialAdditionalAward Fee

100%

AdditionalAward FeeAchieved =55% of thePotentialAdditionalAward Fee

Award Fees

vs. CMM Level



NG ROI Example 1 of 2

2.1

3.93.5

6.16.6

01234567

1 2 3 4 5Build

Def

ect D

ensi

ty (D

efec

ts/K

LOC

)

DP 1CAR

DP 3CAR

DP 2CAR

Quality

Improvements in:

Defect Prevention Using PSP and CAR at CMMI ML5

Integrating PSPsm and CMMI® Level 5. Gabriel Hoffman, Northrop Grumman IT . May 1, 2003



NG ROI Example 2 of 2

Resultsmet 25+ milestones in a rowearned a rating of “Exceptional” in every applicable category on a formal Contractor Performance Evaluation Survey

Hours Invested: 124 in Defect Prevention (CAR) Hours saved: 1650 hours (15 hours per defect)ROI: 13:1

Appraised at CMMI ML 5 in December 2002

Integrating PSPsm and CMMI® Level 5. Gabriel Hoffman, Northrop Grumman IT . May 1, 2003

Quality

Schedule / cycle time

Customer satisfaction

Cost of quality/ ROI

Improvements in:



External ROI – Ericsson Ltd.

• PROCESS IMPROVEMENT

$1,890,000Total Cost Savings$2,524,000Savings from Inspections$634,000Total Costs$358,000Execution of Inspections$69,000Training costs$207,000Support Expenditures



External ROI – Others page 1 of 2



External ROI – Others page 2 of 2

7536452113

Level 5

02

0/0150/23130

Delta 6/06

1Sikorsky AircraftSAIC

2RaytheonNorthrop Grumman (including Mel.)

3Lockheed MartinGeneral DynamicsComputer Sciences CorporationBoeingBAE

Level 4Federal Contractor

Industry Published SCAMPI Results



Harris ROI Example

• Program Collected Metrics• Program Analyzed Metrics• Program Made Process Improvement Changes• Process Changes Improved Metrics



New zone ofquality control

QualityImprovement

Original zone ofquality control

Chronic waste

Capability Control Chart

Qualityimprovement

New zone ofquality control

Less chronic waste

Indicates a problem

Control chart

Leve

l 4Le

vel 5

Quantitative ManagementWhat Is QM?

“Voice of the Process”• Predictable & quantitatively

understood• Statistics & quantitative

techniques to manage processes & results

• Address “Special Causes”

“Voice of the Customer”• Continual & measurable

process improvement to meet business objectives

• Incremental & innovative process improvements

• Address “Common Causes”



• GCSD Engineering Measurement Objectives “Monitor key functional processes for predictability and measurable improvement (e.g., productivity, quality)”

• Current Data- Intuitively suggests that process has improved over time- Predictable? - Measurable improvement?

Process ImprovementsL4/L5 Example – Defect Reduction

County Rejections (Count by Month)

0

5

10

15

20

25

30

Oct-03

Nov-03

Dec-03

Jan-0

4

Feb-04

Mar-04

Apr-04

May-04

Jun-0

4Ju

l-04

Aug-04

Sep-04

Oct-04

Nov-04

Dec-04

Jan-0

5

Feb-05

Mar-05

Apr-05

May-05

Jun-0

5Ju

l-05

Aug-05

Sep-05

Oct-05

Nov-05

Dec-05

Jan-0

6

Firs

t tim

e re

ject

ions

Number of Rejections

• Modified Objective: “Monitor key functional processes for predictability and measurable improvement (e.g., 10% productivity, 20% quality)”


0

5

10

15

20

25

30

Oct-03

Nov-03

Dec-03

Jan-0

4

Feb-04

Mar-04

Apr-04

May-04

Jun-0

4Ju

l-04

Aug-04

Sep-04

Oct-04

Nov-04

Dec-04

Jan-0

5

Feb-05

Mar-05

Apr-05

May-05

Jun-0

5Ju

l-05

Aug-05

Sep-05

Oct-05

Nov-05

Dec-05

Jan-0

6

Firs

t tim

e re

ject

ions

Number of Rejections

* not enough data points to classify as statistical control but can use trial limits

• Stable performance (within control limits*)

• Predictable average of 9.1 rejections per month (capability)


0

5

10

15

20

25

30

Oct-03

Nov-03

Dec-03

Jan-0

4

Feb-04

Mar-04

Apr-04

May-04

Jun-0

4Ju

l-04

Aug-04

Sep-04

Oct-04

Nov-04

Dec-04

Jan-0

5

Feb-05

Mar-05

Apr-05

May-05

Jun-0

5Ju

l-05

Aug-05

Sep-05

Oct-05

Nov-05

Dec-05

Jan-0

6

Firs

t tim

e re

ject

ions

Number of Rejections UCL Mean LCL

Changes made:• Team personnel changes• Stakeholder participation

formalized• Peer Review process

deployed more formally• Return to Green

Rejections (Count by Month)

0

5

10

15

20

25

30

Oct-03

Nov-03

Dec-03

Jan-0

4

Feb-0

4

Mar-04

Apr-04

May-0

4

Jun-0

4Ju

l-04

Aug-04

Sep-04

Oct-04

Nov-04

Dec-04

Jan-0

5

Feb-0

5

Mar-05

Apr-05

May-0

5

Jun-0

5Ju

l-05

Aug-05

Sep-0

5

Oct-05

Nov-05

Dec-05

Jan-0

6

Firs

t tim

e re

ject

ions

Number of Rejections UCL Mean LCL

Re-calculated limits due to process changes –

• Stable performance (within tighter control limits*)

• Predictable average of 2.5 rejections per month (capability) shows 73% measurable improvement

90% AF 97.8% AF 100% AF 100% AF



CPI

Average 1.02

Average 1.12

Upper Control Limit 1.51 Upper Control Limit 1.52

Lower Control Limit 0.54

Lower Control Limit 0.72

0.00

0.20

0.40

0.60

0.80

1.00

1.20

1.40

1.60

1.80

2003

10

2003

12

2004

02

2004

04

2004

06

2004

08

2004

10

2004

12

2005

02

2005

04

2005

06

2005

08

2005

10

2005

12

2006

02

2006

04

CPI

CPIAverageUpper Control LimitLower Control Limit

Process ImprovementsL4/L5 Example – CPI

90% AF 98% AF 100% AF 100% AF

• After process improvements implemented:– Average cost performance improved 10% – Lower end of predictable performance range improved 33%

BENEFITS: Higher productivity, cost stability, higher award fees

TEAM & PROCESS IMPROVEMENTS MADE

100% AF



Internal ROI Ideas

• Rework ROI– Estimate rework cost– Estimate defect reduction– Estimate cost avoidance

• UPS ROI– Estimate cost for downtime– Determine cost of UPS– Estimate savings

• Automation ROI– Estimate manual effort– Estimate automated effort– Estimate savings

• Other Suggestions?



Convince Other Stakeholders

• Show Management Support• Educate and Train

• Motivate to change• Communicate vision• Teach Tools and Techniques• Reward compliance

• Show Value for Them



Show Management Support

• “What You Measure You Will Improve.” – author unknown

• Harris Process Compliance Monitor– Monitors all required processes – Appropriate tailoring allowed– Artifacts of compliance entered into tool– Online product-centric evidence collection, objective

auditing– Online real-time project and organizational monitoring

of process compliance– Required compliance



• Direct artifacts are missing for more than 30 days from the initial scoring audit

• Direct artifacts are missing in the initial scoring audit or direct artifacts are present but inadequate (Note 3)

• One or more weaknesses noted (Note 1)

• Direct artifacts are present and appropriate (Note 2)• One or more weaknesses noted (Note 1)

• Direct artifacts are present and appropriate (Note 2)• No weaknesses noted (Note 1)

• Pending an appraisal

• Outside the scope of the project (e.g., Code and Unit Test Process is not applicable to a production-type program)

• To be appraised at a later date (i.e., the process has not yet been executed by the program and cannot be appraised)

Not ImplementedNI

Partially ImplementedPI

Largely ImplementedLI

Fully ImplementedFI

Not ScoredNS

Not ApplicableNA

Not YetNY

Process Compliance Color Scores

Note 1: A weakness ("gap") is considered if it is an impact to or risk of implementation of the process statementNote 2: An appropriate artifact is the IPM Expected Artifact or equivalent that demonstrates implementation of the process statementNote 3: An inadequate artifact does not demonstrate implementation of the process statement

AS

SE

SS

ME

NT

ST

AT

US

CO

LOR

SP

RO

CE

SS

CO

MP

LIA

NC

E

CO

LOR

S



Process Compliance

A

• Represents overall process compliance score for program

• Based on lowest color score – harsh, but in keeping with CMMI standards

B• Depicts scoring distribution over all process items

• More insight on overall project score

C• Depicts score for each process

executed or being executed by this program

• 3 columns identify types of processes• In PCM, point+click on underlined

acronym drills down to scoring details for process



• Practical Software and Systems Measurement• SEI Training Courses on Process Improvement• Six Sigma Training• Provide Technical Expertise• Train Them to:

– Follow the processes– Collect consistent measures– Analyze data– Suggest improvements

• Train for Appraisals

Educate User Community



Show Value for Stakeholders

• Process Improvement Proposal: Business Intelligence Implementation

• Invest in Tool, Save in Usage• Current Estimates for Creating Metrics and

Reports Monthly ~$1.5 million per Year• Current Estimate for Incorporating BI ~$2M• Time Saved Each Month Can be Used Elsewhere• Other benefits

– Consistent, timely, accurate data– Flexibility in reporting– Easier new analysis



• Without a Vision, There Can Be no Improvements– State, Share and Spread the Vision Of Process

Improvement• Management Acts Based on the Bottom Line

– Show and Tell how process improvement can improve their business (ROI)

• Other Stakeholders Tend to Follow Management– Convince Other Stakeholders that management

is convinced and is watching• Teams Want Improvement

– Show and Tell how process improvement will help them (ROI)

Summary



Debra J. PerrySW Engineer Division Process GroupHarris [email protected]

Harris Corporation http://www.harris.com/P.O. Box 37Melbourne, Florida 32902-0037

Contact Information



BACKUP SLIDES



References page 1 of 4

• Page 9-12 Introduction to CMMI Version 1.2 http://www.sei.cmu.edu/products/courses/p44b.html

• Page 13 Profiles of Level 5 CMMI Organizationshttp://www.stsc.hill.af.mil/crosstalk/2007/01/0701Reifer.html

• Page 15 No Satisfaction at Toyotahttp://www.fastcompany.com/magazine/111/open_no-satisfaction.html




• Page 20 - Performance Results of CMMI®-Based Process Improvement 2006 http://www.sei.cmu.edu/pub/documents/06.reports/pdf/06tr004.pdf

• Page 21 - Lockheed Martin resultshttp://www.sei.cmu.edu/activities/cmmi/results/pdfs/2003-CMMI-016.pdf

• Page 22 - Integrating PSPsm and CMMI® Level 5. Gabriel Hoffman, Northrop Grumman IT . May 1, 2003




• Page 24 - Case Study of an Improvement Program Featuring Reviews and Inspections, by: Robert MacFarland, Ericsson Ltd.; American Society for Quality,Software Quality Professional, June 2001, Vol. 3, No. 3

• Page 25 – Process Maturity Profile CMMI v1.1 SCAMPI v1.1 Class A Appraisal Results 2005 Mid-Year Update, September 2005 http://www.sei.cmu.edu/appraisal-program/profile/pdf/CMMI/2005sepCMMI.pdf

• Page 26 - SEI CMMI Published Appraisal Results http://sas.sei.cmu.edu/pars/




• Page 34-35 “Getting There: Tips and Considerations for Marching Toward CMMI Maturity Level 4 or 5”, by Gary Natwick, Debbie Perry and Sophie Boyd, SEPG Conference 26-29 March 2007



Process Improvement Plan

Innovation and Improvement

Predictable Results

Repeatable Practices

BenefitsCausalStatisticalComparativeAnalysis

Systemic Issues

Decision Making

Monitor and Control

Measurement Focus

Division and Project

Division and Project

ProjectImprovement Focus

Continuous Improvement

PredictiveProactiveProcess Approach

OptimizedStable and Controlled

Not Understood Quantitatively

Process Capability

ChangeCapabilityProcessManagement543Level



ROI for Management Example 1 of 2

(Profiles of Level 5 CMMI Organizations)



ROI for Management Example 2 of 2

(Profiles of Level 5 CMMI Organizations)



Raytheon North TX – L5

“… we want to achieve the performance excellence goals required by our business. We are focused on achieving performance excellence and recognition as the preferred supplier for new business.”



Northrop Grumman – L5



Industry Investment In Process Maturity

GCSD vs. Industry Headcount from SEI Survey (December 2004)

GCSD FY05

GCSD FY05

GCSD FY06REQGCSD

FY06REQ

0.0%

0.2%

0.4%

0.6%

0.8%

1.0%

1.2%

1.4%

1.6%

1.8%

2.0% Industry Median1.9% of Organization

Northrop-Grumman (Melbourne)1.5% of Engineering

Where does you company fit?



Process Improvement Flowchart

CollectData

AnalyzeData

DefineProcess

Improvement

ProcessImproved?

NoYes

Document Processes

DetermineGoals

DetermineMeasures



ROI Process Flowchart

Present toManagement

TrackActualCosts

EstimateChange Benefits

Change Approved?

No

Yes

DetermineScope

EstimateCurrent Costs

Calculate ROI


Documents

Debra J. Perry Harris Corporation · Debra J. Perry Harris Corporation. How Do We Get On The Road To Maturity? assuredcommunications!" NDIA Conference - 2 12-15 November 2007 What