NG BB 47 Basic Design of Experiments

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

National GuardBlack Belt Training

Module 47

Basic Design of Experiments (DOE)

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

2

CPI Roadmap – Improve

Note: Activities and tools vary by project. Lists provided here are not necessarily all-inclusive.

TOOLS

•Brainstorming

•Replenishment Pull/Kanban

•Stocking Strategy

•Process Flow Improvement

•Process Balancing

•Standard Work

•Quick Change Over

•Design of Experiments (DOE)

•Solution Selection Matrix

• ‘To-Be’ Process Mapping

•Poka-Yoke

•6S Visual Mgt

•RIE

ACTIVITIES• Develop Potential Solutions

• Develop Evaluation Criteria

• Select Best Solutions

• Develop Future State Process Map(s)

• Develop Pilot Plan

• Pilot Solution

• Develop Full Scale Action/

Implementation Plan

• Complete Improve Gate

1.Validate the

Problem

4. Determine Root

Cause

3. Set Improvement

Targets

5. Develop Counter-

Measures

6. See Counter-MeasuresThrough

2. IdentifyPerformance

Gaps

7. Confirm Results

& Process

8. StandardizeSuccessfulProcesses

Define Measure Analyze ControlImprove

8-STEP PROCESS

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

33Basic Design of Experiments

Learning Objectives

Learn benefits of DOE methodology

Discuss differences between DOE and trial and error (one-factor-at-a-time) approaches to experimentation

Learn basic DOE terminology

Distinguish between the concepts of full and fractional factorial designs

Use Minitab to run and analyze a DOE

Use results of DOE to drive statisticallysignificant improvements

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

4

Helicopter SimulationPhase One

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

5

Exercise: Helicopter Simulation

Customers at CHI (Cellulose Helicopters Inc.) have been complaining about the limited flight time of CHI helicopters

Management wants to increase flight time to improve customer satisfaction

You are put in charge of this improvement project

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

6

Exercise: Constraints

Project Mission: Find the combination of factors that maximize flight time

Project Constraints:

Budget for testing = $1.5 M

Cost to build one prototype = $100,000

Cost per flight test = $10,000

Prototype once tested can not be altered

See allowable flight test factors and parameters on the next page

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

7

Exercise: Test Factors and Parameters

Paper Type Regular Card stock

Paper Clip No Yes

Taped Body No 3 in of tape

Taped Wing Joint No Yes

Body Width 1.42 in 2.00 in

Body Length 3.00 in 4.75 in

Wing Length 3.00 in 4.75 in

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

8

Exercise: Roles & Responsibilities

Lead Engineer – Leads the team and makes final decision on which prototypes to build and test

Test Engineer – Leads the team in conducting the test and has final say on how test are conducted

Assembly Engineer – Leads the team in building prototypes and has final say on building issues

Finance Engineer – Leads the team in tracking expenses and keeping the team on budget

Recorder – Leads the team in recording data from the trials

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

9

Exercise: Phase One Deliverables

Prepare a Phase One Report showing:

Recommendation for optimal design

Predicted flight time at optimal setting

How much money was spent

Description of experimental strategy used

Description of analysis techniques used

Recommendations for future tests

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

10

Introduction to DOE

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


1. Significant Event 2. Somebody Sees It

3. Research How can we learn more efficiently?

How Do We Learn?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


How Do We Learn? (Cont.)

Products and processes are continually providing data that could lead to their improvement - so what has been missing? There are several possibilities:

We are not collecting and analyzing the data provided

We are not proactive in data collection

We are unable to translate the data into information

A significant event has not occurred

“In order to learn, two things must occur simultaneously: something must happen (informative event) and someone must see it happen (perceptive observer).” – George Box

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


How Do We Improve?

By creating significant events and observing them, we can obtain knowledge faster

That is basically what occurs in a designed experiment

Let‟s look at an example of these two things occurring (significant event and perceptive observer) simultaneously

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Champagne Example

Wine – The fermented juice of fresh grapes used as a beverage. Wine has been in existence since the beginning of recorded history

Champagne – A clear, sparkling liquid made by way of the second fermentation of wine. First discovered by a French monk in the late 1600s

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


SPC tools and techniques improve observation, but we must wait for an event to happen in order to observe it

Need Improved Observation

Need to make sure that naturally occurring informative events are brought to the attention of the perceptive observer!

Improved observation increases the probability of observing naturally occurring informative events so appropriate action can be taken.

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Passive Observation Is Not Enough

We need to induce occurrence of informative events.

An experiment is set-up so that an informative event will occur!

By manipulating inputs to see how the output changes, we can understand and model Y (a dependent variable) as a function of X (an independent variable).

Designed Experimentation – The manipulation of controllable factors (independent variables) at different levels

to see their effect on some response (dependent variable)

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Experimental Process:

A controlled blending

of inputs which

generates corresponding

measurable outputs.

People

Material

Equipment

Policies

Procedures

Methods

Environment

Responses related toperforming a service

Responses related toproducing a product

Responses related tocompleting a task

Inputs (Factors) Outputs (Responses)

From Understanding Industrial Designed Experiments, Schmidt & Launsby

What Is Experimental Design?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Example of a Recruiting Process

Process:

Recruiting

Job Description

Marketing

Candidate Pool

Hire Quickly

Inputs(Factors)

Outputs (Responses)

Economic Environment

Type of Job

Hire Best Candidate

Location of Job

Job Application Process

EEO Requirements Hire at Competitive Pay

DOE was originally used for manufacturing quality applications - it has now expanded to many other areas where performance characteristics are of interest

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Methods of Experimentation

Experimentation has been used for a long time. Some experiments have been good, some not so good

Our early experiments can be grouped into the following general categories:

1. Trial and Error

2. One-Factor-at-a-Time (OFAT)

3. Full Factorial

4. Fractional Factorial

5. Others

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Trial and Error - Increase Gas Mileage

Problem: Gas mileage for car is 20 mpg. Would like to get > 30 mpg.

Factors:

Change brand of gas

Change octane rating

Drive slower

Tune-up car

Wash and wax car

New tires

Change tire pressure

Remove hood ornament and external radio antenna

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Problem: Gas mileage for vehicle is 20 mpg. Would like to get > 30 mpg

How many more runs would you need to figure out the best configuration of variables?

How can you explain the above results?

If there were more variables, how long would it take to get a good solution?

What if there‟s a specific combination of two or more variables that leads to the best mileage (the optimum)?

MPG Results

One-Factor-at-a-Time (OFAT)- Gas Mileage

Speed Octane Tire Pressure Miles per Gallon

55 85 30 25

65 85 30 23

55 91 30 27

55 85 35 27

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Results

How would we find this optimum with OFAT testing?

How would we know that we‟d found it?

Miles per Gallon as a Function of Speed and Tire Pressure

26 28 30 32 34 36 38

75

70

65

60

55

50

45

40

35

30

18 23 26 26 20

17

26

26

18

Tire Pressure (lbs.)

Sp

ee

d (

mp

h)

35Optimum MPG

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Is there a better way?

One-Factor-at-a-Time (OFAT)

While OFAT is simple, it is inefficient in determining optimal results:

Unnecessary experiments may be run

Time to find causal factors is significant

Don‟t know the effects of changing one factor while other factors are also changing (no model)

Inability to detect or learn about how factors work together to drive the response

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Cake Example - Interactions

An Interaction occurs when the effect of one factor, X1, on the response, Y, depends on the setting (level) of another factor, X2:

Y = f(x)

For example, when baking a cake, the temperature that you set the oven at is dependent on the time that the cake will be in the oven.

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO



Where would you set Time to get a good cake?

How would you experiment on this process to learn about this interaction?

Tim

e

Temp = 500 degreesTime = 20 minutes




Temp100 500

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO



Duncan Hines used designed experiments in the 50‟s on their cake mixes.

Their goal was a robust design for the most consistent product.

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Why Use DOE?

The structured methodology provides a directed approach to avoid time wasted with “hunt and peck” - don‟t need 30 years of experience to design the tests

The designed experiment gives a mathematical model relating the variables and responses - no more experiments where you can‟t draw conclusions

The model is easily optimized, so you know when you‟re done

The statistical significance of the results is known, so there is much greater confidence in the results

Can determine how multiple input variables interact to affect results

“Often we have used a trial and error approach to testing, or just changed one variable at a time. Why is a statistically

designed experiment better?”

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Full Factorial DOE

Full Factorial examines every possible combination of factors at the levels tested. The full factorial design is an experimental strategy that allows us to answer most questions completely.

Full factorial enables us to:

Determine the Main Effects that the factors being manipulated have on the response variable(s)

Determine the effects of factor interactions on the response variables

Estimate levels at which to set factors for best results

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


What can we do when resources are limited?

Minimum number of tests for a full factorial experiment: Xk

X = # of levels, k = # of factors

Adding another level significantly increases the number of tests!

Level 2 3 4

2 4 8 16

3 9 27 81

Factors

# of Tests

Full Factorial

Full Factorial Advantages

Information about all effects

Information about all interactions

Quantify Y=f(x)

Limitations

Amount of resources needed

Amount of time needed

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Full Factorial Notation

2 level designs are the most common because they provide a lot of information, but require the fewest tests.

The general notation for a full factorial design of 2 levels is:

2 is the number of levels for each factor (Range = High and Low)

k is the number of factors to be investigated

This is the minimum number of test runs required for a full factorial

2k = # Runs

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


OFAT Runs

Problem: Gas Mileage is 20 mpg

What conclusion do you make now?

How many runs?

How many runs at each level?

Speed Octane Tire Pressure Miles per Gallon

55 85 30 25

65 85 30 23

55 91 30 27

65 91 30 23

55 85 35 27

65 85 35 24

55 91 35 32

65 91 35 25

MPG = f(Speed, Octane, Tire Pressure)

Full Factorial Experiment

Do we think 32 is best?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Fractional Factorial

Looks at only a fraction of all the possible combinations contained in a full factorial.

If many factors are being investigated, information can be obtained with smaller investment.

Resources necessary to complete a fractional factorial are manageable.

Limitations - give up some interactions

Benefits

Economy

Speed

Fewer runs

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Fractional Factorial Notation

The general notation to designate a fractional factorial design is:

2 is the number of levels for each factor

k is the number of factors to be investigated

2-p is the size of the fraction (p = 1 1/2 fraction, p = 2 1/4 fraction, etc.)

2k-p is the number of runs

R is the resolution

pkR2 = # Runs

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Fractional Factorial Notation – Resolution

When we go to a fractional factorial design, we are not able to estimate all of the interactions

The amount that we are able to estimate is indicated by the resolution of an experiment

The higher the resolution, the more interactions we can measure Example: The designation below means fifteen factors will be

investigated in 16 runs. This design is a resolution III:

11152

III Note: A deeper discussion of design resolution is beyond the scope of the lesson. The content, above, is intended to only provide a brief explanation of the design resolution term.

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Speed

(A)

Octane

(B)

Tire Pressure

(C)

Mileage

(Y)

55 85 35 27

65 85 30 23

55 91 30 27

65 91 35 25

Gas Mileage Example

Problem: Gas mileage for vehicle is 20 mpg

Compare with previous full factorial:

How many runs?

How much information?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


A1 A2

Speed Octane Tire Pressure

55 65 85 91 30 35

23.8

24.8

25.8

26.8

27.8

Mil

ea

ge

Main Effects Plot (data means) for Mileage

In the gas mileage example, Speed,Octane, and Tire

Pressure all look to have an effect on average mileage.

DOE Will Help Us Identify Factors

Factors which shift the average

Longer line = greater effect

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


B2

B1


55 65 85 91 30 35

1.0

1.5

2.0

2.5

3.0

Sta

nd

ard

De

v

Main Effects Plot for Standard Deviation

Only Tire Pressureis affecting standard

deviation


Factors which affect variation

Flat line = no effect

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


C1C2


55 65 85 91 30 35

1.0

1.5

2.0

2.5

3.0

Sta

nd

ard

De

v

Main Effects Plot for Standard Dev


55 65 85 91 30 35

23.8

24.8

25.8

26.8

27.8

Mil

ea

ge


Only Tire Pressure affects both the average mileage and also the variability


Factors which shift the average and affect variation

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


D1 = D2

Driver Radio

-1 1 -1 1

24

25

26

27

28

Mil

ea

ge


Driver Radio

-1 1 -1 1

1.0

1.5

2.0

2.5

3.0

Sta

nd

ard

De

v

Main Effects Plot for Standard Dev

An expanded study investigated the effect of driver and radio on mileage. These factors show no effect. This is also valuable information, because these factors can be set at their most economical (least cost) or

most convenient levels.


Factors which have no effect

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Benefits of DOE

Determine input settings which optimize results and minimize costs

Quick screening for significant effects

Obtain a mathematical model relating inputs and results

Reduction in the number of tests required

Verification of the statistical significance of results

Identification of low-impact areas allows for increased flexibility/tolerances

Standardized methodology provides a directed approach

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


When Could I Use Design of Experiments?

Identification of critical factors to improve performance

Identification of unimportant factors to reduce costs

Reduction in cycle time

Reduction of scrap/rework

Scientific method for setting tolerances

Whenever you see repetitive testing

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


DOE Review

What does DOE offer us that trial and error experimentation and OFAT do not?

What are the differences between full and fractional factorial DOE‟s?

What is the minimum number of runs required for a 2-level, 3-factor full factorial experiment?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Minitab: Airline DOE Example

A contract airline is interested in reducing overall late take-off time in order to improve Soldier satisfaction

Previous Black Belt work has identified 4 key process input variables (KPIVs) that affect late time:

Dollars spent on training

Number of jets

Number of employees

% Overbooked

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Using the Airline DOE Data.mpj file and Minitab, the instructor will walk the class through the following activities:

A DOE to identify which factors affect “minutes late” in

terms of both the mean and standard deviation

Use the DOE results to determine new process settings

Hypothesis test to prove statistical significance of change.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Current settings for these factors are as follows:

Dollars spent on training 200

Number of jets 52

Number of employees 850

% Overbooked 15

The target is zero minutes late, with a specification of +/- 10 minutes.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Our goal in this experiment is to reduce late take-off times - we will measure late time in minutes

Here are the factors and their levels that we are going to investigate:

Factors Levels

Dollars spent on training 100 300

Number of Jets 50 55

Number of Employees 800 900

% Overbooked 0 25


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


First, we need to set up the test matrix

Select Stat>DOE>Factorial>Create Factorial Design


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


We left the default at: 2-level factorial design. That means we will test each factor at 2 different levels

We also selected 4 factors, since there are 4 variables that we want to test in this experiment. Select Display Available Designs to display possible experiments we can run…


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


For 4 factors, we can either do an 8 run half-fraction or a 16 run full factorial. We will go with the 16 run full factorial experiment.

Click OK to return to the previous dialog box.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Click the Designs button and highlight the 16-run Full Factorial design.

Leave the other settings at their defaults, click on OK.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Next, click the Factors button

Enter the names of the Factors – Change -1 and 1 to actual levels per chart below


FactorsLevelsDollars spent on training 100

300Number of Jets 50 55Number of Employees 800

900% Overbooked 0 25

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Click the Options button and uncheck Randomize runs.

We do want to randomize our tests when we actually run an experiment. However, for this in-class demo, it will be easiest if everyone‟s screen is the same.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Here are the tests that we need to run. Example, row 1 indicates that we first need to collect data at the low level for all four factors.(Tip: First check that you have the same test matrix. If you don‟t, it‟s likely that you did not uncheck “Randomize Runs.”)


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Next, we collect the data.

To allow us to measure variation, we need to run 3 repetitions at each set of settings.

Copy the data from the DOE data worksheet and paste into the design as shown below.

Min Late 1 = C9

Min Late 2 = C10

Min Late 3 = C11


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Run #1, with settings of $100 spent on training, 50 jets, 800 employees, and 0% overbooked, was 46.35 minutes late on the first repetition, 61.92 minutes late on the second repetition, and 75.18 minutes late on the third repetition.

$100.00 50 Jets 800 Employees 0% Overbooked

Note: This row of coded variables were all at their Low (or –1) settings


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Since we ran our DOE with 3 repetitions, and we want to analyze the variation in our DOE results, we need to prepare the worksheet by having Minitab calculate means and standard deviations.

First, we need to name some blank columns. Name a blank column StdDev Min Late, a second blank column Count Min Late, and a third blank column Mean Min Late.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Now, we will have Minitab calculate the means and standard deviations.

Select Stat>DOE>Factorial>Pre-Process Responses for Analyze Variability


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Click in the box Store standard deviations in and select the column you named StdDev Min Late

Click in the box Store number of repeats in and select the column you named Count Min Late

Click in the box Store Means (optional) in and select the column you named Mean Min Late

Click OK

Click on Compute for repeat responses across rows, then click in the cell under Repeat responses across rows of: and select Min late 1, Min late 2, and Min late 3 from the columns pane.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Minitab calculates means and standard deviations for each combination of factors. (Remember: there were 24, or 16, combinations.)

Minitab also determines the counts. (Remember: there were 3 data points at each combination, since we ran 3 repetitions at each setting of the DOE.)

Looking at this data Practically, there appears to be some significance to the factors, but nothing definitive…yet.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Before we view the statistics, we always start with the graphs.

Select Stat>DOE>Factorial>Factorial Plots.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Select Main Effects Plot.

Choose Setup and click on OK to go to next dialog box


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Select StdDev Min Late and Mean Min Late for Responses,and move all four factors from Available to the Selected box to have them included in the analysis. Click on OK.

>


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


1-1

60

45

30

15

0

1-1

1-1

60

45

30

15

0

1-1

Training Dollars

Me

an

Jets

Employees %Overbooked

Main Effects Plot for Mean Min LateData Means

The Main Effects Plot shows that the number of Employees is the only driver for Mean Min Late

Looking at this data Graphically, it appears that Employees might be a significant factor influencing the Mean of Time Late


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


1-1

8

6

4

2

1-1

1-1

8

6

4

2

1-1

Training Dollars

Me

an

Jets

Employees % Overbooked

Main Effects Plot for StdDev Min LateData Means

The Main Effects Plot shows that the number of Jets AND Employees are driving the StdDev Min Late

Looking at this data Graphically, it appears that Jets AND Employees might be a significant factor influencing the StdDev of Time Late


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Now we will run the analysis.

Select Stat>DOE>Factorial>Analyze Factorial Design


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Select Mean Min Late and StdDev Min Late as the Response.

Choose Terms to get to next dialog box


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Include Terms in the model up through second order (2). This will include the main effects and two-way interactions. Click on OK to go back to previous dialog box.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Click on Graphs and select the Pareto Chart. Click OK in both dialog boxes.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


This chart confirms what we saw earlier in the Main Effects Plot – the number of Employees has a significant impact on Mean Min Late. We also see that the interaction term BD* is significant.

CD

AC

AB

A

B

BC

AD

D

BD

C

9080706050403020100

Term

Standardized Effect

2.57

A Training Dollars

B Jets

C Employ ees

D %O v erbooked

Factor Name

Pareto Chart of the Standardized Effects(response is Mean Min Late, Alpha = 0.05)

Pareto Chart forMean Min Late

* - BD is the interaction between the factors Jetsand %Overbooked.

This is the „Critical F-statistic‟ used to determine significance.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


A

D

CD

BD

AB

BC

AC

C

AD

B

43210

Te

rm

Standardized Effect

2.571

A Training Dollars

B Jets

C Employ ees

D % O v erbooked

Factor Name

Pareto Chart of the Standardized Effects(response is StdDev Min Late, Alpha = 0.05)

This chart confirms only part of what we saw earlier in the Main Effects Plot – the number of Jets has a significant impact on StdDev Min Latebut Employees does not.

Pareto Chart forStdDev Min Late


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Factorial Fit: Mean Min Late versus Training Dollars, Jets, ...

Estimated Effects and Coefficients for Mean Min Late (coded units)

Term Effect Coef SE Coef T P

Constant 30.42 0.3191 95.34 0.000

Training Dollars -0.71 -0.35 0.3191 -1.11 0.319

Jets -0.77 -0.39 0.3191 -1.21 0.279

Employees -53.89 -26.94 0.3191 -84.44 0.000

%Overbooked -1.28 -0.64 0.3191 -2.01 0.101

Training Dollars*Jets 0.68 0.34 0.3191 1.06 0.337

Training Dollars*Employees 0.12 0.06 0.3191 0.18 0.861

Training Dollars*%Overbooked 1.13 0.56 0.3191 1.77 0.138

Jets*Employees -0.81 -0.41 0.3191 -1.27 0.259

Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020

Employees*%Overbooked -0.10 -0.05 0.3191 -0.16 0.881

S = 1.27632 PRESS = 83.4049

R-Sq = 99.93% R-Sq(pred) = 99.28% R-Sq(adj) = 99.79%

This data shows Analytically that Employees and the Jets*% Overbooked interaction are statistically significant.

In the Session window, we see that Employees and the interaction Jets*%Overbooked are the only statistically significant factors for Mean Min Late. All other main effects and 2-way interactions have a p-value > 0.05.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Factorial Fit: StdDev Min Late versus Training Dollars, Jets, ...

Estimated Effects and Coefficients for StdDev Min Late (coded units)


Constant 4.575 0.7241 6.32 0.001

Training Dollars -0.105 -0.052 0.7241 -0.07 0.945

Jets -5.930 -2.965 0.7241 -4.09 0.009

Employees -2.477 -1.239 0.7241 -1.71 0.148

% Overbooked 0.146 0.073 0.7241 0.10 0.923

Training Dollars*Jets -0.987 -0.493 0.7241 -0.68 0.526

Training Dollars*Employees 2.032 1.016 0.7241 1.40 0.219

Training Dollars*% Overbooked 2.786 1.393 0.7241 1.92 0.112

Jets*Employees 1.655 0.828 0.7241 1.14 0.305

Jets*% Overbooked -0.935 -0.468 0.7241 -0.65 0.547

Employees*% Overbooked 0.932 0.466 0.7241 0.64 0.548

S = 2.89641 PRESS = 429.527


In the Session window, we see that Jets is the only statistically significant factor for Stdev Min Late. The negative sign for Effectindicates that standard deviation decreases as Jets increases. All other main effects and 2-way interactions have a p-value > 0.05.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Summarizing what we have found in the initial analysis: Employees and the interaction between Jets and %Overbooked

had a significant impact on Mean Min Late.

As seen from the Effects, increasing Jets or %Overbookeddecreases Mean Min Late.

In addition, when the product of Jets*%Overbooked is positive, Mean Min Late will increase. If the product is negative, Mean Min Late will decrease.

Jets had a significant impact on Stdev Min Late. As seen from its Effect, increasing Jets decreases Stdev.


Employees -53.89 -26.94 0.3191 -84.44 0.000

Jets -0.77 -0.39 0.3191 -1.21 0.279

%Overbooked -1.28 -0.64 0.3191 -2.01 0.101

Jets*%Overbooked 2.14 1.07 0.3191 3.36 0.020


Jets -5.930 -2.965 0.7241 -4.09 0.009


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


The next step in the DOE analysis is to eliminate the insignificant terms. This is called “reducing the model.”

Every study is different; in this particular case, let‟s take the following approach:

Reduce the StdDev model to identify the needed setting for „Jets‟ since it:

is the only significant factor influencing StdDev

plays only a small role in driving the Mean (Coef = -0.39)

will determine where to set the factor % Overbooked in the interaction term.


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO




UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


1. Click on Terms

2. Remove all Selected Terms: except B:Jets

3. Select OK and OK

1

2

3


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


The mathematical model is taken from the „Coef‟ column of the Session Window: StdDev Min Late* = 4.575 – (2.965 x Jets)

Factorial Fit: StdDev Min Late versus Jets Estimated Effects and Coefficients for StdDev Min Late (coded units)

Term Effect Coef SE Coef T PConstant 4.575 0.7792 5.87 0.000Jets -5.930 -2.965 0.7792 -3.81 0.002

S = 3.11695 PRESS = 177.653R-Sq = 50.84% R-Sq(pred) = 35.79% R-Sq(adj) = 47.33%

Conclusion: To reduce StdDev Min Late, we should set the factor „Jets‟ to the +1 level (55).


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Now, let‟s reduce the model for the response, Mean.



UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


1. Click on Terms

2. Remove all Selected Terms: except B:Jets, C: Employees, D: % Overbooked and the interaction BD.

3. Select OK and OK

1

2

3


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


The mathematical model is taken from the „Coef‟ column of the Session Window:

Mean Min Late = 30.42 – (0.39 x Jets) – (26.94 x Employees)–(0.64 x %Overbooked) + (1.07 x Jets x %Overbooked)

Factorial Fit: Mean Min Late versus Jets, Employees, %Overbooked

Estimated Effects and Coefficients for Mean Min Late (coded units)


Constant 30.42 0.3354 90.71 0.000

Jets -0.77 -0.39 0.3354 -1.15 0.273

Employees -53.89 -26.94 0.3354 -80.34 0.000

%Overbooked -1.28 -0.64 0.3354 -1.91 0.082

Jets*%Overbooked 2.14 1.07 0.3354 3.19 0.009

S = 1.34148 PRESS = 41.8811



UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Of the four factors that were investigated, two (Employees and Jets), plus the Jets*%Overbooked interaction, were significant.

Jets – to reduce variation, we need to increase Jets to 55.

Employees - to reduce the average late time from 30 minutes, we need to increase Employees from 850 to 900.

Training Budget - had no effect, and can be reduced to $100k as a budget savings.

% Overbooked - had marginal effect on time late and on variation – should be reduced to 0% to improve customer satisfaction.

What would you do if this were your organization?


UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Is the Change Significant?

1. Conduct a hypothesis test.

2. Open the Capability Data worksheet within the Airline DOE Data.mpj file.

3. Since we have the baseline sample and the improved sample, select Stat>Basic Statistics>2-Sample t…

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Is the Change Significant? (continued)

4. Select „Samples in different columns‟

5. Select „Baseline Data‟ for First: and „New Data‟ for Second:

6. Select „Graphs…‟

4

5

6

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO



7. Select Boxplots of data

8. Click on OK

9. Interpret boxplot. Does there appear to be a graphical difference?

7

89

New DataBaseline Data

50

40

30

20

10

0

Da

ta

Boxplot of Baseline Data, New Data

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO



10.Review Minitabs Session Window output.

11.Can we state, with 95% confidence, that there is a statistical difference between our Baseline Data and the New data? (i.e. Does the Confidence Interval contain „0‟ or is the P-value less than 0.05?)

10

11

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

86

Helicopter SimulationPhase Two

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

87

Exercise: Helicopter Simulation

Customers at CHI (Cellulose Helicopters Inc.) have been complaining about the limited flight time of CHI helicopters

Management wants to increase flight time to improve customer satisfaction

You are put in charge of this improvement project

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

88

Exercise: Constraints

Project Mission: Find the combination of factors that maximize flight time

Project Constraints:

Budget for testing = $1.5 M

Cost to build one prototype = $100,000

Cost per flight test = $10,000

Prototype once tested can not be altered

See allowable flight test factors and parameters on the next page

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

89

Exercise: Test Factors and Parameters

Paper Type Regular Card stock

Paper Clip No Yes

Taped Body No 3 in of tape

Taped Wing Joint No Yes

Body Width 1.42 in 2.00 in

Body Length 3.00 in 4.75 in

Wing Length 3.00 in 4.75 in

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

90

Exercise: Roles & Responsibilities

Lead Engineer – Leads the team and makes final decision on which prototypes to build and test

Test Engineer – Leads the team in conducting the test and has final say on how test are conducted

Assembly Engineer – Leads the team in building prototypes and has final say on building issues

Finance Engineer – Leads the team in tracking expenses and keeping the team on budget

Recorder – Leads the team in recording data from the trials

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

91

Exercise: Phase Two Deliverables

Prepare a Phase Two Report showing:

Recommendation for optimal design

Predicted flight time at optimal setting

How much money was spent

Description of experimental strategy used

Description of analysis techniques used

Recommendations for future tests

Comparison of Phase One and Phase Two approaches

Which Team Has The Best Design?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


Takeaways

Types of experiments – Trial and Error, OFAT, DOE

Introductory DOE terminology

Benefits of full factorial vs. fractional designs

How to use Minitab to design, run, and analyze a DOE

Use DOE results to drive statistical improvements

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO

93

What other comments or questions

do you have?

UNCLASSIFIED / FOUO

UNCLASSIFIED / FOUO


References

Schmidt & Launsby, Understanding Industrial Designed Experiments

Education

NG BB 47 Basic Design of Experiments