Tutorial: Measuring for Performance-Driven Improvementitq.ch/pdf/sepg/PerformnceMeasurement_6Sigma101e.pdf · Tutorial: Measuring for Performance-Driven Improvement By Bob Stoddard,

page 1

Pittsburgh, PA 15213-3890

June 12th 2006 European SEPG

Tutorial: Measuring forPerformance-Driven Improvement

By Bob Stoddard, SEIPhil Bush, Motorola

This material is approved for public release. Distribution is limited by theSoftware Engineering Institute to attendees.

Sponsored by the U.S. Department of Defense© 2005 by Carnegie Mellon University

© 2005 by Carnegie Mellon University page 2


Outline

Important terms and concepts

What happens before DMAIC: project selection

DMAIC roadmap review

CMMI connections

Motorola Examples



Important Terms and Concepts

Performance

Process

Statistics



What is Performance?

What do we mean by performance?

How does it relate to both product and processimprovement?

Performance:

How well something is done.

An evaluation of the outcome(s) (Technical or Business).

In reference to the intended result of a process.

May be for a single execution, or for an aggregation orsummary.



Why is Measuring for Performanceimportant?

•Performance added to Malcolm Baldrige award criteriaupon recognizing the importance of business results

- Some early award recipients became business failures

•Track record of corporate failures based on the solepursuit of improvement models without a focus on results

•Approximate 20 year success track record of Six Sigmabased primarily on an intense focus of customer satisfactionvia performance

•Enduring High Maturity organizations characterized bystrong measurement of improvement of performance



What is a Process in Relation toProducts & Services?Any set of conditions or causes that work together to produce agiven result.

A system of causes which includes the people, materials,energy, equipment, and procedures necessary to produce aproduct or service.

Products &Services

Requirements& Ideas Work activities

Time

People Material Energy Equipment Procedures



Process Analysis Challenges

“Real”process behavior must be considered before makingconclusions about performance of products or services.

Ask these questions to find out about real process behavior:•What is the normal or inherent process variation?•What differentiates inherent from anomalous variation?•What is causing the anomalous variation?

Statistics provides the methods and tools needed tomeasure and analyze process behavior, draw conclusions(i.e. “statistical inferences”) and drive improvement!



Measuring for Performance-Driven Improvement

The class name depicts a pursuit of measurement that:•aligns with business and organizational goals•integrates with improvement models•plans, models, predicts, analyzes, tracks and ensures high

performance results•enables confident management and risk assessment by

coupling expert judgment with objective data•consistently maintains focus on customer satisfaction

This pursuit represents the original essence of boththe SEI CMM/CMMI models, as well as, the SixSigma business improvement model!



What is a Statistic?•A summary or characterization of a distribution (i.e., a

set of numbers).

•A characterization of a central tendency (e.g., mean,median, and mode).

•A characterization ofdispersion (e.g., variance,standard deviation,interquartile range, range).



Everything is a process or a product of a process.All processes have central tendency and variability.Data are used to understand these attributes and to drivedecisions to improve processes.

Statistical Thinking: A Paradigm

[ASQ 00], [ASA 01]

New mean after improvement(Spread due to common causevariation will re-establish itself.)

Original Mean

Special Cause Variation

Data Spread due toCommon Cause Variation



So, we need to improve two ways…

Target

USLLSL

Centerthe

Process

Reducethe

Spread

Target

USLLSL

Process Off Target

Defects

Target

USLLSL

Excessive Variation

Defects



Types of DataExamples•Defect counts by

type•Languages•Domains

Ordinal

Data set / observations placed intocategories; may have unequal intervals.

A B C

Nominal

Continuous(aka, variable)

Discrete(aka, categorized,

attribute)

Increasinginformation

content

Ratio

Interval

Examples•Satisfaction ratings

from surveys•Risk ratings•CMMI maturity levels•High, Medium, Low

complexity of features

Examples•Time•Cost•Code size•Variances•Effort•Number of Change

Requests

<A B C

<

0

A B

1 2

Data set with a > or < relationshipsamong the categories; may haveunequal intervals; integer valuescommonly used

Data set assigned to points on a scale inwhich the units are the same size; decimalvalues possible

Interval data setwhich also hasa true zero point;decimal valuespossible



Hypothesis TestingA formal way of making a comparison and deciding whether or notthe difference is “significant”based on statistical analysis.

Consists of a null and alternative hypothesis

•Null hypothesis states that the members of the comparison are“equal”; “there is no difference”(a concrete, default position)

•Alternative hypothesis states that there is a difference; issupported when the null hypothesis is rejected

Conclusion either rejects or fails to reject the null hypothesis.

Understanding the null and alternative hypotheses isthe key to understanding the statistical tests anddecisions taught during the remainder of this class!



Hypotheses TopicsPopulation mean equals some predefined value•Average productivity is 100 SLoC per month

Differences between two means or between two variances•Average productivity at ML 3 is greater than it was at ML 2•Variation in project productivity was greater at ML 2 than it is

at ML 3

Sample proportion equals some predefined value•90% of projects complete within 5% of their budget

Differences between proportions•A greater proportion of projects complete within 5% of their

budget at ML 3 than at ML 2

Two variables have a statistically significant relationship toeach other•Project effort is positively related to estimated SLOC



Formally Stating a HypothesisAverage productivity equals 100 SLoC•Null: Average productivity is equal to 100 SLoC•Alternative: Average productivity is not equal to 100 SLoC

A refinement•Null: Average productivity is equal to 100 SLoC•Alternative: Average productivity is less than 100 SLoC

Generally, the alternative hypothesis is the “difference”(e.g. improvement or performance problem) that weseek to learn about.

The null holds the conservative position that apparentdifferences can be explained by chance alone. Thephrase “is equal to”will always appear in the null.



Steps in Hypothesis Testing1. Understand the question to be answered using sample data.•Is average performance greater than 100?•Is variance higher in one set of projects vs. another?

2. Formulate two opposing hypotheses.•null and alternative hypotheses

2. Select a test statistic (automatically performed by Minitab).•common test statistics are t, F, and chi-square

3. Identify the alpha error to use in the hypothesis test.•The alpha error represents the probability of wrongfully rejecting the

null hypothesis. In other words, it is the probability of wrongfullyconcluding that the alternative hypothesis is true.

4. Use the resulting p-value to conclude which hypothesis to believep-value <= alpha error, conclude alternative hypothesis is truep-value > alpha error, conclude null hypothesis



Interpreting the p Value

The p value is the probability that thealternative hypothesis will occur bymere chance alone.

Thus, p<5% represents an alternativehypothesis that we should accept.



Slogan to Remember p Interpretation

“When the p is low,the null must go…

When the p is high,the null must fly”



Outline

Important terms and concepts



CMMI connections

Motorola Examples



Project Selection—1

Voice of theCustomer

Voice of theProcess

Voice of theBusiness

ImprovementProjects



Little y’s

Vital X’s

DMAIC Projects

Big Y

Goal Statement Improve the accuracy of the customer insight process

50% by the end of the year.

Predictable ResultsPredictable Results

Unit VolumeUnit VolumePricePrice ManufacturingCost

ManufacturingCost

DevelopmentCost

DevelopmentCost

Business CaseEffectiveness

Business CaseEffectiveness

Product LaunchTimeliness

Product LaunchTimeliness

•Customer Insight Process

•Market Size Forecasting

•Commercial DOE Testing

•Resource Management

•Work Allocation

•Specs / RequirementsManagement

•Software Processes

<10% <10%>75% <5%

Y’s are costbuckets orsuccess

outcomes;x’s arespecific

processes




Sponsor—Selects the Little y’s and Vital x projects; has influenceand often authority over the process and connecting organizations. Ismost often a key executive who commissions the project. Isresponsible for removing organization roadblocks and insuring all theright team members are assigned.

Champion—Owns the process being changed. Responsible andaccountable for final results. Sometimes the champion and sponsorare the same person.

Change Agent—Responsible for driving the change and leadingthe improvement team. Is proficient in the DMAIC improvementmethodology and has domain expertise and credibility within theorganization or project.




Risk AssessmentWhat’s the probability of success and what can wedo to increase it?Success probability Medium#2 Executive commitment –partial#6 Right team members available-partial

Project TypeWhat type of DSS project is this?DMADV: Change level required –breakthroughwith expectation of new processes.

Team StructureSponsor COO Team:Champion Ops VPMBB Consultant

Project PlanWhat are the milestones for applying themethodology?Milestones within the major phases of:Define ImproveMeasure ControlAnalyze

Project ScopeBound the process and focus on something that canbe achieved in 3-6 months

Goal StatementIdentifies a specific process that you are going toimprove and level of performance

Opportunity/ROI StatementWhat’s the $ value of this project?If you can’t identify a bottom line impact, why wouldmanagement invest resources in it?

Business CaseWhat’s your most important issue?What’s broken? What are we losing?What’s the linkage to big Y?

Project Selection—4Team Charter Example



OutlineImportant terms and concepts



CMMI connections

Motorola Examples



DMAIC Roadmap

Define ControlAnalyze ImproveMeasure

Defineprojectscope

Establishformalproject

Identifyneededdata

Obtaindata set

Evaluatedata quality

Summarize& baselinedata

Exploredata

Characterizeprocess &problem

Identifypossiblesolutions

Implement(pilot asneeded)

Definecontrolmethod

Implement

Updateimprovementproject scope& scale

Document

Selectsolution

Evaluate

Phase Exit Review



Define Guidance Questions

Defineprojectscope


Define

Identifyneededdata

Obtaindata set



Exploredata




Definecontrolmethod

Implement


Selectsolution

Evaluate

ControlAnalyze ImproveMeasure

Document

•What is the current problem to be solved?•What are the goals, improvement targets, & successcriteria?•What is the business case, potential savings, or benefitthat will be realized when the problem is solved?•Who are the stakeholders? The customers?•What are the relevant processes and who owns them?

M A I CD



Defining the Project ScopeWho are the stakeholders? The customers?What are the relevant processes and who owns them?

Linking Customer and Process Voices

We begin with what we already ‘know’about our customers &our processes, but make sure we’ve got it right.

•Do we understand our customer needs?

•Do we understand key processes,and how and why they came to be as they are?

•Are the connections between our processes, customerneeds, and our business evident and consistentlyunderstood?

M A I CD



Define Guidance Questions

Defineprojectscope


Define

Identifyneededdata

Obtaindata set



Exploredata




Definecontrolmethod

Implement


Selectsolution

Evaluate

ControlAnalyze ImproveMeasure

Document

•Have stakeholders agreed to the project charter orcontract?•What is the project plan, including the resourceplan and progress tracking?•How will the project progress be communicated?

M A I CD



Identifyneededdata

Measure Guidance Questions

Obtaindata set

Evaluatedataquality

Summarized& baselinedata

Exploredata




Definecontrolmethod

Implement


Selectsolution

Defineprojectscope


Measure ControlAnalyze ImproveDefine

Document

Evaluate

•What are the process outputs andperformance measures?

•What are the process inputs?

•What info is needed to understandrelationships between inputs andoutputs? Among inputs?

•What information is needed to monitorthe progress of this improvementproject?

M A I CD



Flowdown (Y-to-x) TreeDescription•Y to x is a hypothesis tool

- depicts hypothesized causal relationships betweencustomer-critical performance measures and processfactors

- represents portions of a transfer function over time

Procedure•Draw a vertical tree diagram to depict the causal relationship.

- Use information from process mapping, naturalsegmentation as inputs.

•Identify x's as uncontrollable vs. controllable and measurable.

•Select y’s and x's for initial data collection and evaluation.

M A I CD



Identifying Needed Data

We need to find out what contributes to performance:

•What are the process outputs (y’s) that drive performance?•What are key process inputs (x’s) that drive outputs and

overall performance?

Techniques to address these questions•segmentation / stratification•input and output analysis•Y to x trees

What are the process outputs and performance measures?What are the inputs? What are the relationships among outputsand inputs?

Using these techniquesyields a list of relevant,hypothesized, processfactors to measure andevaluate.

M A I CD



Segmentation vs. Stratification

Segmentation—•grouping the data according to one or more of the data

attributes (e.g. segment by domain, language, product,release)

Stratification—•grouping the data according to specific values within a given

attribute (e.g. stratify by size of project, complexity of release,experience level of developers, cycle time)

M A I CD

Y



Process MappingProcess map—a representation of major activities/tasks,subprocesses, process boundaries, key process inputs, andoutputs.

INPUTS(Sources ofVariation)

OUTPUTS(Measures ofPerformance)

•Perform a service•Produce a Product•Complete a Task

PROCESS STEP

A blending ofinputs to achieve

the desiredoutputs

•People•Material•Equipment•Policies•Procedures•Methods•Environment•Information

M A I CD



Alternative Process Map—Swim LanesD

esig

ners

Pro

gram

mer

sT

este

rsQ

ualit

yS

yste

mE

ngr

M A I CD



Rigorously Identifying Measures

•Decide on the project improvement goals (which were rooted inthe earlier problem statements).

•Use the Goal-Question-Indicator-Measure (GQIM) process toidentify and operationalize project measures.

•Instrument appropriate processes and products to provide theneeded data for the measures.

M A I CD



Roll-up ForHigher Management

Task 1Task 2Task 3

Task n

Tasks to Accomplishgoal

••

Task 1Task 2Task 3

Task n


••••

SuccessCriteriaGoal

Strategy toaccomplishthe goal

Progress Indicators

Success Indicators

Analysis Indicators

80

204060

100

Tasks

Tes

tC

ases

Co

mpl

ete

Functions

1 2 3 4 1 2 3 4

%

Reporting Periods

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual80

204060

10080

204060

10080

204060

100

ForProject Manager

Measuring Goal Achievement

How well are plans proceeding?

Have the goals beenachieved? What is theimpact of the tactics?

What are resultsof specific tasks?

M A I CD



GQIM Example

Establish the leading peachorchard in Georgia byselling more peaches thanmy competitors.

Goal Success Criteria

Strategy

Tasks

Preference

Progress Indicators

Success Indicators

AnalysisIndicators

Greater MarketShare than mycompetitors

•Discover what motivates peach buying•Boost productivity by planting more of the

most desirable variety of peach trees•Etc.

Market Share

1 2 3 4 1 2 3 4

%

Reporting Periods1 2 3 4 1 2 3 4

•Establish baseline•Survey which peach type most popular•Determine peach planting plan•Plant trees•Better marketing•etc.

Task 1Task 2Task 3Task 4

Status

Peach Type

%

M A I CD



Measures to Construct IndicatorsSuccessCriteriaGoal

Strategy toaccomplishthe goal

Progress Indicators

Success Indicators

Task 1Task 2Task 3

Task n


••••

Task 1Task 2Task 3

Task n


••••

Analysis Indicators

80

204060

100

Tasks

80

204060

100

Tasks

Tes

tC

ases

Co

mpl

ete

Functions

Tes

tC

ases

Co

mpl

ete

Functions

1 2 3 4 1 2 3 4

%

Reporting Periods1 2 3 4 1 2 3 4

%

Reporting Periods

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual

Reporting Periods

Planned

Actual80

204060

10080

204060

10080

204060

100

ForProject Manager

Roll-up ForHigher Management

Measures Measures

MeasuresMeasures

MeasuresMeasures

M A I CD



Identifyneededdata


Obtaindata set

Evaluatedataquality


Exploredata




Definecontrolmethod

Implement


Selectsolution

Defineprojectscope



Document

Evaluate

M A I CD

•Is the needed measurementinfrastructure in place?•Are the data being collected andstored?



Indicator TemplateIndicator Template

Goal ID:ObjectiveQuestion

InputsAlgorithmAssumptions

Components of aGood Goal Statements

Step 1: Goals Step 4:Operationalize Goals

OperationalizeGoal Statement

Step 9: Identify theactions needed toimplement your

measuresPlanning

TasksData Elements

Task 1

Task 2

Task 3

Task n

1 2 3 4 5

50

Y

YY

N

N

Y

Y

YY

Step 10: Prepare a plan

Verification andaction plans

Step 2:Clarifying Questions

To refine the goal

Clear articulation of thecriteria you will use todecide if the goal hasbeen met.

Step 5: Success Criteria

Step 6: Success Indicators

Postulate Success Indicators

M A I CD

Step 3:Decomposing

GoalsSubgoals byperspective

Step 7:Analysis and Progress

IndicatorsStrategies & Activities

Step 8: Identify the dataelements

DataElements

SizeDefects

Avail Source+0-0+- -

QACM?

Etc.••



Measures Address Multiple Indicators

a b c d eX

X

X

XX X

X X

X

X

SizeEffort

Defects

IndicatorsDataElements

X

Success

Analysis

Progress

SuccessIndicators

AnalysisIndicators

ProgressIndicators

Data Elements (Measures)

Tes

tC

ases

Co

mpl

ete

Functions

Planned

Actual80

204060

10080

204060

10080

204060

100

M A I CD



Identifyneededdata


Obtaindata set

Evaluatedataquality


Exploredata




Definecontrolmethod

Implement


Selectsolution

Defineprojectscope



Document

Evaluate

•Does the measurement system yieldaccurate, precise, and reproducibledata?•Are urgently needed improvementsrevealed?•Has the risk of proceeding in theabsence of 100% valid data beenarticulated?

M A I CD



Evaluating Data QualityDoes the measurement system yield accurate, precise,and reproducible data?

To find out, we need to do a measurement systemevaluation (MSE), including understanding the datasource and the reliability of the process that created it.

Frequently encountered problems include the following:•wrong data•missing data

Sometimes, a simple “eyeball”test reveals problems

Look at the frequency of each value:–Are any values out of bounds?–Does the frequency of each value make sense?–Are some used more or less frequently than expected?

M A I CD



Formal MSE Provides Answers…

•How big is the measurement error?

•What are the sources of measurement error?

•Is the measurement system stable over time?

•Is the measurement system capable?

•How can the measurement system be improved?

M A I CD



Precision

Precision is made up of two sources of variation:repeatability and reproducibility.

Precision2 MS = +Reproducibility

2 rpdRepeatability

2 rpt

M A I CD

“TotalPrecisionVariation”

“VariationBetweenPeople”

“Variationwithinsame

person”



Repeatability

Repeatability is the inherent variability of the measurementsystem.

The variation that results when repeated measurements aremade under identical conditions:•same inspector, analyst•same set up and measurement procedure•same software or document or dataset•same environmental conditions•during a short interval of time

M A I CD



Reproducibility

Reproducibility is the variation that results when differentconditions are used to make the measurement:•different software inspectors or analysts•different set up procedures, checklists at different sites•different software modules or documents•different environmental conditions;

Measured during a longer period of time.

M A I CD



MSE Metrics-Precision%Gauge Repeatability & Reproducibility (%GR&R):The fraction of total variation consumed by measurementsystem variation.

M A I CD

%100% xGRRTotal

MS

* Automotive Industry Action Group (AIAG) MSA Reference Manual, 3rd edition

*



How Much Variation is Tolerable?M A I CD

* Reference Automotive Industry Action Group (AIAG) MSA Reference Manual, 3rd edition

Unacceptable>30%

Unacceptable for “critical”measurements

(You should improve themeasurement process.)

between 10% & 30%

Acceptable<10%

If the %GRR is… Then measurement error is…



Accuracy (Bias)

Accurate Not accurate

Accuracy—The closeness of (average) reading to the correct valueor accepted reference standard.

M A I CD



Precision vs. Accuracy

Accuratebut not precise

Precisebut not accurate

Both accurateand precise

M A I CD



MSE for Continuous Data

M A I CD



When should each formal statistical approach be used?

Attribute data is on Nominal scale Fleiss’Kappa statistic

e.g. Types of Inspection Defects,Types of Test Defects,ODC Types, Priorities assignedto defects, Most categoricalinputs to project forecasting tools,Most human decisions amongalternatives

Attribute data is on Ordinal scale Kendall’s coefficients(each item has at least 3 levels)

e.g. Number of major inspection defects found,Number of test defects found,Estimated size of code to nearest 10 KSLOC,Estimated size of needed staff,Complexity and other measures used toevaluate architecture, design & code

MSE Calculations for Attribute Data—1

M A I CD




M A I CD

Null Hypothesis: Consistency by chance; no association

Alternative Hypothesis: Significant consistency & associationThus, a p value < 0.05 indicates significant and believable consistencyor association.

agreement only by chanceWhen Result = 0

too much measurement errorWhen Result < 0.7

marginal measurement errorWhen 0.70 < Result < 0.9

very low measurement errorWhen Result > 0.9

perfect agreementWhen Result = 1.0

Interpreting results of Kappa’s or Kendall’s coefficients

Interpreting the accompanying p value



MSE Calculations for Attribute Data—3How do you interpret these Kappavalues and p values for thisnominal measurement system?

M A I CD




How do youinterpret theseKendallcoefficientsand p values?

Response is an ordinal rating. Thus,appraisers get credit for comingclose to the correct answer!

M A I CD



Identifyneededdata


Obtaindata set

Evaluatedataquality


Exploredata




Definecontrolmethod

Implement


Selectsolution

Defineprojectscope



Document

Evaluate

•What does the data look like upon initialassessment? Is it what we expected?

•What is the overall performance of theprocess?

•Do we have measures for all significantfactors, as best we know them?

•Are there data to be added to theprocess map?

•Are any urgently needed improvementsrevealed?

•What assumptions have been madeabout the process and data?

M A I CD



Summarizing & Baselining the DataWhat is baselining?

Establishing a snapshot of performance (distribution of theprocess behavior) and/or the characteristics of a process.

Why should we baseline performance?It provides a basis by which to measure improvement.

How is it done?•Describe the organization’s performance using

– the 7 basic tools– a map of the process of interest, including scope

(process boundaries) and timeframe•Compare to best-in-class

– benchmarking•Gather data

– sample appropriately

M A I CD



Measure Phase Summary AdviceMeasures should

•relate closely to the issue under study

•have high information content

•pass precision, accuracy and stability tests

•permit easy and economical collection of data

•permit consistently collected, well-defined data

•show measurable variation as a set

•have diagnostic value

[Wheeler 92]

M A I CD



Analyze Guidance Questions

Define ControlAnalyze Improve

DefineprojectScope


Identifyneededdata

Obtain dataset


Summarize,baselinedata

Exploredata

Characterizeprocess andproblem



Definecontrolmethod

Implement

Updateimprovementproject scopeand scale

DocumentSelectsolution

Evaluate

Measure

•What does the data looklike?

•What is driving thevariation?

•What is the newbaseline?

•What are associated risksand assumptionsassociated with reviseddata set and baseline?

M A I CD



Exploring the Data

Probing questions during data exploration:

•What should the data look like? And, does it?- first principles, heuristics or relationships- mental model of process (refer to that black box)- what do we expect, in terms of cause & effect

•Are there yet-unexplained patterns or variation? If so,- conduct more Y to x analysis- plot, plot, plot using the basic tools

•Are there hypothesized x’s that can be removed from the list?

Objective - To completely identify the Y’s, little y’s, and x’s

What do the data look like?What is driving the variation?

M A I CD



Graphical Methods Summary

Regression Predicted LinePredict relationships in Data

Normal plotCheck Normality of Data

Pareto chartPrioritize 2+ X’s to focus on

Multi-vari chartSee Variation of Y w/2+ X’s

Box Plot chartSee Variation of Y with 1 X

Time series run chartSee Time Relationships

Scatter plotSee Relationships in Data

Purpose Graphical Method

M A I CD



Analyze Guidance Questions

Define ControlAnalyze Improve

DefineprojectScope


Identifyneededdata

Obtain dataset



Exploredata




Definecontrolmethod

Implement


DocumentSelectsolution

Evaluate

•Are there any hypothesesthat need to be tested?

•What causal factors aredriving or limiting thecapability of this process?

•What process mapupdates are needed?

•Are there any immediateissues to address? Anyurgent and obviousneeds for problemcontainment?

Measure

M A I CD



Characterizing the Process and theProblem—1What causal factors are driving or limiting the capability of thisprocess?

•Which x’s are or are not significant?•Can plausible changes in x’s deliver targeted/desired changes

in y’s and Y’s?•Do we need to find more x’s?•Do we need to refine goals?

M A I CD



Characterizing the Process and theProblem—2Are there any hypotheses that need to be tested?

To answer our questions, are there any hypotheses that need tobe tested?•How do we test? (Using tests for significant difference,

correlations, and experiments –more on this in Improve.)

What is the stability and capability of the process?•What are assignable causes for “special cause”variation?•What are root causes for “common cause”variation?

M A I CD



Controlled (Predictable) Variation

t1

t2

t3

t4

Distribution

Process behavior measured at times t1, t2, t3, and t4

All measurements

•have same central tendency and dispersion

•fall within the same limits

M A I CD



Uncontrolled (Unpredictable) Variation

t1

t2

t3

Distribution

Not all measurements•have same central tendency and dispersion•fall within the same limits

t4

Process behavior measured at times t1, t2, t3, and t4

M A I CD



Why Control Charts?

A valid inference about the behavior of the process in thefuture requires

•an understanding of previous process behavior thatsupports a judgment regarding future behavior

•a history of stable process behavior is required to supportsuch a judgment

M A I CD

Control charts are the primary mechanism toprovide an operational definition of process stability.



Control Chart Basics

Lower Control Limit (LCL)

Upper Control Limit (UCL)

SpecificationLimits

Control Limits From Process Performance Measurements(Voice of the process)

Specification Limits Set by customer, engineer, etc.(Voice of the customer)

Event Time or Sequence

Mean or Center LineCL + 3

CL - 3

M A I CD



Control Chart Example

0 5 10 15 20 25 308

12

16

20

24

28

32

LCL = 8.49

CL = 20.04

UCL = 31.6

Week of System Test

0 5 10 15 20 25 3002468101214

CL = 4.35

UCL = 14.2

MovingRange

Number ofUnresolved

ProblemReports

M A I CD



Which Control Chart Should be Used?

np chart

Continuous dataAttributes data

1< n <10n = 1

n > 10

Poisson

BinomialOther

Control Chart

We will focuson these two

M A I CD

EqualSample

Size UnequalSample

Sizec chart

u chart p chart

EqualSample

Size

UnequalSample

SizeXmR

chart

X-bar, S

chart

X-bar, R

chart

**

*Explained onsuccessive

slides




np chart


1< n <10n = 1

n > 10

Poisson

BinomialOther

Control Chart


M A I CD

EqualSample

Size UnequalSample

Sizec chart

u chart p chart

EqualSample

Size

UnequalSample

SizeXmR

chart

X-bar, S

chart

X-bar, R

chart

*Poisson data generally is in the

form of a count of items during aninterval of time, e.g. how many

fish pass a spot in the river eachminute




np chart


1< n <10n = 1

n > 10

Poisson

BinomialOther

Control Chart


M A I CD

EqualSample

Size UnequalSample

Sizec chart

u chart p chart

EqualSample

Size

UnequalSample

SizeXmR

chart

X-bar, S

chart

X-bar, R

chart

*Binomial data generally is in theform of a constant probability of

each item occurring, e.g.probability of a “1”showing up on

each roll of a die



Root Cause Tips

Play detective. Suspect everything!

Evaluate paired comparisons

•characteristics of a “good data point”vs.

•characteristics of a “bad data point”

Stratify the data into “good”and “bad”subsets and evaluate

For sporadic problems, troubleshoot in real time becausememories fade quickly

M A I CD

Now that we have stable data from controlled processes, the next step isto research the root causes of variation



Hypothesis Testing for Root Cause

Comparing a process or product to “specification”•Is the process on aim?•Is the variability satisfactory?

Comparing two processes, products or populations•Are the means (or medians) the same?•Is the variation the same?

The following slides summarize:•When to use which of the most common Hypothesis Tests•Which analytical method to use to quantify the relationship

between x’s (causes) and the Y’s (outcomes)

M A I CD



Types of Hypothesis TestsM A I CD

ANOM(Analysis of

Means)

Chi-Squaretest

Van derWaerdenNormal

scores test

IndependentKrukal-Wallis1 wayANOVA

Friedman2 way ANOVA

Paired

NormalBartletttest

Levenetest

Not Normal

ANOVA(1 & 2 wayANOVA;BalancedANOVA; GLM)MANOVA(General &Balanced)

2 Proportionstest

Fisher Exacttest (1 wayANOVA);

Chi-Squaretest

= MediansSiegel-Tukeytest

Mosestest

= Medians

IndependentMannWhitneyU test

Wilcoxonmatched

Paired

NormalF test

Levenetest

Not Normal

Independent2 samplet test

pairedt test

Paired

1 Proportionstest

>2 cellsChi-Square

BinomialSign Test

=2 cells

Kolmogorov-Smirnov

Goodness ofFit test

1 sampleWilcoxonSigned

Ranks test

1 sampleChi-Square

test

1 sample ttest

Data Type

# Samples

(Data Groups)

Interval or Ratio(Parametric Tests)

Ordinal(Non-Parametric Tests)

Mean Variance Median Variance/Fit

Nominal Proportion

Similarity Similarity

1 Sample

2 Samples

3+ Samples



Scenario Decision Path—Hypotheses

1. Identify two samples of data or two differentpopulations of data to compare with a Hypothesis

2. Test each sample or population for Normality andequal variances

3. If not Normal, either transform data to be Normal, orconduct Nonparametric Hypothesis tests (useHypothesis Decision Matrix), else

4. Conduct parametric hypothesis tests (Use HypothesisDecision Matrix)

5. Use p Value Summary chart to conclude result of test

M A I CD



Quantifying Relationships

LogisticRegression

Correlation& Regression

Chi-Square& Logit

ANOVA& MANOVA

Y

Continuous Discrete

X

Co

nti

nu

ou

sD

iscr

ete

M A I CD



Scenario Decision Path - Relationships

1. Identify a sample or population of data to analyze forcorrelation or predictive relationships.

2. Use quantifying relationships decision matrix todetermine which method to use.

3. Use p value Summary chart to conclude result.

M A I CD




Y (Continuous)

X(D

iscr

ete)

MANCOVAANCOVA

2-Way MANOVA2-Way ANOVA

1-Way MANOVA1-Way ANOVA

Hotelling’s T2,Discriminant Analysist-Test

1 Variable >= 2 Variables

1 Variable,2 levels

1 Variable,>= 2 levels

>= 2Variables

Mixture ofDiscrete &

Continuous

M A I CD




LogisticRegression

Correlation& Regression

Chi-Square& Logit

ANOVA& MANOVA

Y

Continuous Discrete

X

Co

nti

nu

ou

sD

iscr

ete

M A I CD




Y (Continuous)

X(C

on

tin

uo

us)

1 Variable >= 2 Variables

1 Variable

>= 2Variables

CanonicalAnalysis

(CANONA)

MultipleRegression

BivariateRegression

M A I CD X



ANOVA—1

The purpose of ANOVA is to test for significant differencesbetween group means in relation to variation in the data.•Example: Are the defects by release decreasing?

Procedure•variability associated with the group means is computed•variability within each group is computed•ratio of two variabilities are compared with the critical

values for the F-distribution to determine significance

[from the electronic statistics textbook]

M A I CD



ANOVA—2One-way ANOVA: Total Defects Discovered vs. Release

Source DF SS MS F PRelease 3 12.31 4.10 1.25 0.292Error 372 1223.94 3.29Total 375 1236.24

S = 1.814 R-Sq = 1.00% R-Sq(adj) = 0.20%

Individual 95% CIs For Mean Based onPooled StDev

Level N Mean StDev ---+---------+---------+---------+------Rel 1 116 0.974 2.011 (---------*--------)Rel 2 84 0.821 1.750 (----------*-----------)Rel 3 71 1.085 1.895 (-----------*-----------)Rel 4 105 0.600 1.560 (---------*---------)

---+---------+---------+---------+------0.35 0.70 1.05 1.40

Pooled StDev = 1.814

M A I CD

What is thenull

hypothesishere?



Scenario Decision Path—ANOVA1. Identify a Y and 1+ x’s to conduct ANOVA.

2. Conduct the ANOVA.

3. Use p value summary chart to conclude whether the Y isdifferent by levels of the x factor.

4. Review adjusted R squared value to see how much (%) ofthe Y behavior is explained by the ANOVA model.

5. Review the confidence intervals of the various levels of thex factor to see which are common (overlap) vs. which aredifferent (do not overlap).

6. Review the residuals chart to ensure residuals look okay.

7. If residuals do not look okay, investigate nonlinearity ordiscard model or only use model in the area that residualsare okay.

M A I CD



Regression—1The purpose of regression is to perform the basic task of ANOVAby determining whether there is significant prediction of dependent(y) variable(s) using knowledge of independent (x) variable(s).

•Example: Can the defects by release (y) be predicted usingknowledge of one or more independent variables (x)s?

•Some types of regression (all y’s & x’s continuous unlessnoted as discrete):

M A I CD

1 discrete “y”& 1+ “x”sLogistic

nonlinear version of the above typesNonlinear

multiple “y”s & 1+ “x”Multivariate

1 “y”& multiple “x”sMultiple linear

1 “y”& 1 “x”Simple linear



Regression—2

What is the nullhypothesis here?

M A I CD



Scenario Decision Path—Regression

1. Identify a Y and 1+ x’s to conduct a regression analysis.

2. Conduct the regression analysis.

3. Use p Value Summary chart to conclude which factorsadd value to the model.

4. Review Adjusted R squared value to see how much (%)of the Y behavior is explained by the model.

5. Review the 4-in-One Residuals chart to ensureresiduals look okay.

6. If Residuals do not look okay, investigate nonlinearity ordiscard model or only use model in the area thatresiduals are okay.

M A I CD



Chi-Square—1The purpose of Chi-Square analysis is to test for significantdifferences within attribute or categorical data.•performed on Contingency tabular data of 2 variables•each cell is the frequency of the two variables occurring jointly

at specific levels or settings•analyzes expected vs. actual frequencies

Types of Chi-Square analysis:•Likelihood Ratio Chi-Square analysis for nominal data•Mantel-Haenszel Chi-Square analysis for ordinal data•Chi-Square Goodness of Fit analysis to determine if one

distribution matches another (Kolgomorov-Smirnov is anexample method for interval data)

•Pearson’s Chi-Square analysis most commonly used todetermine association in a contingency table

M A I CD



Chi-Square—2

Males

Females

What is the nullhypothesis here?

M A I CD



Remember: Interpreting the p Value

M A I CD

The p value is the probability that thealternative hypothesis will occur bymere chance alone.

Thus, p<5% represents an alternativehypothesis that we should accept.



Remember: p Interpretation

M A I CD

“When the p is low,the null must go…

When the p is high,the null must fly”



p value SummaryNull

Go with nullGo withalternative

X factor adds value tomodel; model has 1+significant x’s

x factor does not addvalue; model has nosignificant x’s

LogisticRegression


Two discrete variablesare associated

Two discretevariables are notassociated

Chi-Square


X factor adds value tomodel

x factor does not addvalue to model

Regression


Difference of Y existsbetween 1+ levels of x

No difference of Yacross levels of x

ANOVA


Data does not followNormal Distribution

Data follows NormalDistribution

Tests forNormality


Two items aredifferent; associationexists

No difference exists;no associations

HypothesisTests

Alternative P < 0.05Method P > 0.05

M A I CD



Control

Evaluate


ExploreData

Improve Guidance Questions

Define Analyze ImproveMeasure

DefineprojectScope


Identifyneededdata

Obtain dataset





Definecontrolmethod

Implement


Selectsolution

•What type of improvement is needed?

•What are solution alternatives to addressurgent issues and root causes of identifiedproblems?

•What are the process factors to beadjusted?

•What is the viability of each potentialsolution?

•What is the projected impact or effect ofeach viable solution?

M A I CD



Control

Evaluate


ExploreData



DefineprojectScope


Identifyneededdata

Obtain dataset





Definecontrolmethod

Implement


Selectsolution

•What are the relative impacts andbenefits?

•What are relevant technical andlogistical factors?

•What are potential risks, issues, andunintended consequences?

M A I CD



Control

Evaluate


ExploreData



DefineprojectScope


Identifyneededdata

Obtain dataset





Definecontrolmethod

Implement


Selectsolution

•What is the action plan with roles,responsibilities, timeline and estimatedbenefit?

•Is piloting needed prior to widespreadimplementation?

M A I CD



Implement (Pilot as Needed)Is piloting needed prior to widespread implementation?

Why conduct a pilot study?

Piloting reduces the risk of rolling out a flawed process,procedure, or other solution component to a broad multi-project environments.

A pilot is to test the solution component within a bounded andcontrolled environment before the component is sanctionedfor broader use.

During a pilot study, the usability of the solution component isevaluated in a near real-world project setting.

Experience demonstrates that such a test always exposesimprovement opportunities that can be exploited to hone andrefine the solution component before broader dissemination.

M A I CD



Control

Evaluate


ExploreData



DefineprojectScope


Identifyneededdata

Obtain dataset





Definecontrolmethod

Implement


Selectsolution

•Did the solution yield the desired impact?

•Has the goal been achieved?

•If piloted, are adjustments needed to thesolution prior to widespread rollout? Isadditional piloting needed?

•How will baselines, dashboards, andother analyses change?

M A I CD



Typical Approaches to Pilot StudyEvaluation

Typical approaches that fail

X represents the introduction of a change

O represents a measurable observation

Before the change Change introduced After the Change

Approach #1 X

Approach #2 X O

Approach #3 XO

M A I CD



Control Guidance Questions


Defineprojectscope


Identifyneededdata

Obtain dataset



ExploreData




Definecontrolmethod

Implement


Document

Selectsolution

Evaluate

•Should data be compared to arange? If so, which range?

•Does procedural adherence need tobe monitored?

M A I CD



Defining Control Methods

•identification and definition (“opdefs”) of all data to be captured−answers what, where, when, how, who

•definition of data processing and reporting process−answers who, how, when

•definition of targets and control or specification limits for eachdata item (Y's and x's)

−defines actions to be taken when limits are exceeded or specialcauses are identified

•definition of a periodic audit or review process−answers who, how, when

•definition of an MSE process–answers who, how, when

•recorded data retention policy and process–answers who, how, when

Elements of a control plan include the following:

M A I CD



Control Mechanisms

•Management control–“dashboards”to monitor business results–focus on Y's–track outputs, backward looking

•Operational control–Monitor significant process variables

o Controllable and noise–Focus on x's–Track controlled inputs, forward looking

•Procedural adherence and audit

M A I CD



Risk Progress

0

5

10

15

Jan Feb Mar Apr

MedHigh

Low

Med

High

Requirements

Warning Indicators

Key Personnel

Supplier QA

Low

Med

High

Software Volatility

Project Dashboard

Size Progress

0

1000

2000

3000

4000

5000

6000

6-Oct

25-Nov

14-Jan4-M

ar23-Apr12-Jun1-Aug

20-Sep

Date

SL

OC Estimated

Size

Progress Indicators

Earned Value Efficiency Delivered Size Defects

M A I CD





Defineprojectscope


Identifyneededdata

Obtain dataset



ExploreData




Definecontrolmethod

Implement


Document

Selectsolution

Evaluate

•What updates are needed in the measurementinfrastructure?

•What process documentation needs to be updated?•What new processes or procedures need to be

established?•Who is the process or measurement owner who will

be taking responsibility for maintaining the controlscheme?

M A I CD





Defineprojectscope


Identifyneededdata

Obtain dataset



ExploreData




Definecontrolmethod

Implement


Document

Selectsolution

Evaluate

•Have we documented improvement projectsfor verification, sustainment, andorganizational learning?

•What are the realized benefits?•Is the project documented or archived in the

organization asset library?•Have documentation and responsibility been

transferred to process or measurementowner?

M A I CD






CMMI connections

Motorola Examples



CMMI Connections

CMMI can be used as a strategic framework forimproving product development processes.

DMAIC can be used as a tactical approach to improvingindividual processes.



CMMI

Engineering Support

ProcessManagement

ProjectManagement

•Organizational ProcessFocus (OPF)•Organizational ProcessDefinition (OPD)•Organizational Training (OT)•Organizational ProcessPerformance (OPP)•Organizational Innovationand Deployment (OID)

•Project Planning (PP)•Project Monitoring andControl (PMC)•Supplier Agreement Mgmt. (SAM)•Integrated Project Mgmt. (IPM)•Risk Management (RSKM)•Quantitative Project Mgmt. (QPM)

•Requirements Management (REQM)•Requirements Development (RD)•Technical Solution (TS)•Product Integration (PI)•Verification (VER)•Validation (VAL)

•Configuration Mgmt. (CM)•Process and ProductQuality Assurance (PPQA)•Measurement & Analysis (MA)•Decision Analysis andResolution (DAR)•Causal Analysis and

Resolution (CAR)

CMMI-SE/SW—Continuous



Capability Evolution of Measurementvia Generic Practices

Identify and correct the root causes of defects and other problems inthe process

5.2 Correct commoncause of problems

Ensure continuous improvement of the process in fulfilling therelevant business objectives of the organization

5.1 Ensure continuousprocess improvement

Stabilize the performance of one or more subprocesses to determinethe ability of the process to achieve the established quantitativequality and process performance objectives

4.2 Stabilize sub-processperformance

Establish and maintain quantitative objectives for the process aboutquality and process performance based on customer needs andbusiness objectives

4.1 Establish qualityobjectives

Collect work products, measures, measurement results, andimprovement information derived from planning and performing theprocess to support the future use and improvement of theorganization’s processes and process assets

3.2 Collect improvementinformation

Monitor and control the process against the plan for performing theprocess and take appropriate corrective action

2.8 Monitor and controlthe process

FocusGeneric Practice



Staged View of Measurement Evolution

Level 2—Project management

plan vs. actuals, major milestones

Level 3—Product quality

defect data for product and work products

Level 4—Process capability and control

process performance distributions

Level 5—Change management

field experiments

parameterized process models

systems dynamics models






CMMI connections

Motorola Examples



Motorola Application Examples

Primary Case Study:•Improving Customer Response Cycle Times

Additional Case Studies:•Specific instances of DMAIC toolbox usage

page 113

Pittsburgh, PA 15213-3890


DSS in the Markets

Improving Customer ResponseCycle Time

VP and General Manager - Michael KrutzChampion –Don Benkeser

Black Belt –Phil Mercy



Customer Satisfaction

•Driven by ProductsQualityFeaturesTimeliness to Market

•Driven by After Sales ServiceQualityMaintenance Issue ResolutionResponsiveness



After Sales Service

•How Much Do You Have To Do?

•How Much Can You Do?

•How Fast Can You Do It?

•What Quality Can You Achieve?

•How Much Are You Willing To Spend?



Dashboard –SR ProblemResolution

• BHAG goal identified

• Data Collection Plan

• Operational Definition

• Process Capability

• Distribution Analysis

4/26/044/26/04Measure

• Source of Variation Study

Comp DateTarget DatePhase/Activity

• Quick Wins Identified

• VOB/VOC to CTQ’s

• “AS-IS”Process Map

• SIPOC

• Team Charter

• Schematic (Yyx Alignment)

3/26/043/26/04Define

• Lessons Learned and Feedback

• Standardization/Adoption Plan

TBDVerify• Digitization Plan

• Cost Benefit Plan

• Alternative Solutions Identified

• Change Plan

11/3/0411/3/04Design

• “SHOULD BE”Process Map

CompDateTarget DatePhase/Activity

• EDA

• Queuing Analysis (routing &TIS)

• Pareto Analysis

5/30/045/30/04Analyze

ACTIVITY COMPLETED

ACTIVITY STARTED

PHASE COMPLETED

PHASE STARTED

PHASE STARTED BUT MISS COMP DATE

PHASE COMPLETED BUT MISS COMP

DATE

D M A D V



Total Backlog

500

600

700

800

900

1000

1100

1200

1300

07/02

/2003

21/02

/2003

07/03

/2003

21/03

/2003

04/04

/2003

18/04

/2003

02/05

/2003

16/05

/2003

30/05

/2003

13/06

/2003

27/06

/2003

11/07

/2003

25/07

/2003

08/08

/2003

22/08

/2003

05/09

/2003

19/09

/2003

03/10

/2003

17/10

/2003

31/10

/2003

14/11

/2003

28/11

/2003

12/12

/2003

26/12

/2003

#SR

s

Goal

Actual

GoalAchieved!581 SRs

31-Dec-03

Service Request Backlog Reduction



Grade of Service

Cyc

leT

ime

(Day

s)

50% 60% 70% 80% 90% 100%0

10

20

30

40

50

60Company A

Company B

Company C

Company DCompany E

AcceptablePerformance

2004Plan

Motorola

Motorola

Competitive Situation onCustomer Support



How Do We Get There?

•Model the Existing ProcessOrganisationProduct FeaturesStaffing Levels

•Use Model to Determine OptionsProcess ChangesOrganisational ChangesProduct Changes



PC SR Cycle time –Measure

Full analysis of CT by Priority, Contract, Time in State (Clarify) etcBottom line …..Mean of 108 days total cycle time for all Product Change SR’s.

3.12.72.31.91.51.1

95% Confidence Interval for Mu

2.102.052.00

95% Confidence Interval for Median

Variable: Log(Total)

2.04551

0.38846

2.00865

Maximum3rd QuartileMedian1st QuartileMinimum

NKurtosisSkew nessVarianceStDevMean

P-Value:A-Squared:

2.11329

0.42789

2.06434

3.012412.333982.077751.763881.00581

824-4.1E-01-3.6E-01

0.1658260.407222.03650

0.0003.681

95% Conf idence Interval for Median

95% Confidence Interval for Sigma

95% Confidence Interval for Mu

Anderson-Darling Normality Test

Descriptive Statistics

P-Value (approx): < 0.0100R: 0.9920W-test for Normality

N: 824StDev: 0.407218Average: 2.03650

321

.999

.99

.95

.80

.50

.20

.05

.01

.001

Pro

babi

lity

Log(Total)

Normal Probability Plot



PC SR Cycle time - QueuingAnalysis ‘As Is’

Queuing analysis used to model our PC SR processBottom line ….Modelled mean total time through network is 113 days

New / CNRC /Local Office /

Customer

Dev CS/MOL

Dev Eng

Confirmed todefect /

fix available /resolved

Bounced

New

Exit

1AveragesTime: 2.9 daysUtil: 98.33%Process: 76.8Queued: 53.2

AveragesTime: 2.9 daysUtil: 98.33%Process: 76.8Queued: 53.2

AveragesTime: 16.9 daysUtil: 98.5%Process: 108Queued: 58.7








100%

100%81%

19% 20%

46%

34%

7%

42%

51%



0

1

2

3

4

5

6

7

Fix delay

Wait cust close

Data collect

Ping Pong

FEknow

Mult issues

Poor Alys

Conf FA

Clarify

P3back

SR

Qua

ntity

OtherDev TrafficBounceFix Cycle

PC SR Cycle time - EDA

44%

30%

22%

4%

EDA of 23 long CT examples confirms the same key variables• Fix roll out cycle time• Bounce rate• Dev eng traffic




Robustness AnalysisSRs currently arrive at the rate of 4.35 per dayAs-Is process can operate up to a max 4.4 /dayCurrent arrival rate puts the process in the unstable part of curveEven if stable, still only 50 days mean CT (which is uncompetitive)

As-Is robustness

0

50

100

150

200

0 1 2 3 4 5 6SR arrival rate (per day)

Cyc

letim

e(d

ays)

0

500

1000

1500

2000

Qua

ntity

inpr

oces

s

u cycle timeIn Network




Key variables


Customer

Dev CS/MOL

Dev Eng



Bounced

New

Exit

100%

100%

100%

81%

19%20%

46%

34%

7%

42%

51%

• Dev eng traffic and cycle time

• Bounce Rate• Fix roll out cycle time



How Do We Best Utilize theModel?

•Brainstorm for improvements using theareas identified by the model

•Model these as standaloneimprovements to see impact

•Look for combinations that best fit ourneeds/budgets/timelines

•Model as full “what-if”scenarios•Decide on way forward




Customer

Dev CS/MOL

Dev Eng



Bounced

New

Exit

100%

100%

100%

81%

19%5%

91%

4%

2%

93%

5%

New / Filter /Data Collection

/ Diagnostics 100%

New Customer Support Model

30 (was 0) 12 (was 24)

23 (was 32) 86 (was 108)

21 (was 50) 1 (was 1)

Process Changes- New Logging- New Data Collection

People Changes- Field Tech Reps- Shared MOL Team

Product Changes- Tool Kit for Analysis- New Diagnostics

10x improvement on bounces

(delays)

Total time through network

of 30.4 days (was 113)

Fix available time of 15 days

(was 24)



Summary –DSS in the Markets

•Provides a structured analysis methodology•Provides a method for making objective

assessments•Allows for rapid scenario analysis•Simplifies complicated organisational issues•Provides a common framework for

understanding•Provides a roadmap for tracking improvements•Provides a framework for ongoing process

improvement activities



Additional Examples of DMAICToolbox Usage

•DMAIC improvements can be dynamic and range fromformal projects (as just seen) to real-time application asbusiness issues arise

•Tailoring of DMAIC phases and tool usage variesaccording to situational needs



Deciding on Features & Priorities

Three primary techniques have been used to identifywhich features to include along with customer priorities:

Analytic Hierarchy Process –used to conduct pair-wise comparisons of features and determine overallsubjective prioritization of a list of features

Design of Experiments orthogonal matrices –usedwhen features are not totally independent and whenscenarios of grouping of features is the focus

Monte Carlo simulation and optimization –usedwhen uncertainty modeling in a spreadsheet helpful toassess optimum mix of features when a variety ofuncertainties exist subject to various constraints



Deciding on Adopting New Tools

The adoption of several tools as significant productivityenhancers:

Adoption of Code Generation Tools using hypothesistesting to compare “Before”and “After”scenarios;Enabled statistical approach to state improvements andsize of improvements

Adoption of DOORS Req’ts Management Toolsetusing Hypothesis testing and regression analysis toascertain the degree of learning curve by Req’ts Analysts



Deciding Action related to NTF’s

During analysis of product returns, a significantpercentage were classified as No Trouble Found (NTF)

Percentage NTF’s deemed too high for the business andwarranty exposure

ANOVA analysis was used to analyze the NTF returnsagainst various discrete attributes to further understandwhat degree were software, and then to understand theroot causes of software issues.

Benefits of NTF analysis enabled actions which cut theNTF rate by half.



Determining Warranty Exposure

Warranty exposure, and related management reserves tobe set aside, can significantly affect financial performance.

A need arose to predict latent software faults at time ofshipment and to predict resulting warranty exposureduring the warranty period due to software.

Regression analysis on arrival rates during testingenabled predictions of latent software faults. Latentsoftware faults statistically correlated with faults found inthe field during the warranty period. Weibull analysisyielded accurate predictions of warranty expense inproduct returns due to software!



Improving Defect PhaseContainmentIndustry data has long shown the relationship betweendegree of defect escapement during product developmentand costs to resolve.

Organizations and projects needed empirical data toprioritize and justify process improvement and predictimpacts of improvements.

Using historical information on defects (phase found vsphase originated), confidence intervals from LogisticRegression were developed within the matrix to supportmonte carlo simulation of proposed improvements.Results were confidence levels of impacts of specificproposed changes.



Improving Customer Satisfaction

Traditional customer surveys did not seem to provide theinformation needed to improve customer satisfaction.

Improved surveys were implemented with categoricaldata analysis methods including mean/medianopinion scores.

Results enabled the audience to distinguish real trendsand changes in customer satisfaction from the normalnoise expected in survey data.

Action plans became solidly based on true signals fromthe customer base.



Improving Dropped Calls

Field testing is routinely used to help assess the incidenceand cause of dropped cellular calls.

Design of Experiments orthogonal matrices were usedto identify scenarios for field test staff to exercise duringdrive testing.

Results enabled software developers to pinpoint causes ofdropped calls and to resolve the high incidence of droppedcalls. Field testing also became much more efficient intesting new products to market!



SummaryThe DMAIC Roadmap guides us to

•performance-driven improvement

•using data and statistics to inform our reasoningand decisions

•understanding how work is really performed andteasing out the causal relationships that we caninfluence

•selecting and testing our ideas for improvement inan objective manner



Questions?

Documents

Tutorial: Measuring for Performance-Driven Improvementitq.ch/pdf/sepg/PerformnceMeasurement_6Sigma101e.pdf · Tutorial: Measuring for Performance-Driven Improvement By Bob Stoddard,