Why Do a Designed Experiment

Jim Breneman©2011 ASQ & Presentation Jim Breneman

Presented live on Sep 08th, 2011

http://reliabilitycalendar.org/The_Reliability_Calendar/Webinars_‐_English/Webinars_‐_English.html

ASQ Reliability Division English Webinar SeriesOne of the monthly webinars 

on topics of interest to reliability engineers.

To view recorded webinar (available to ASQ Reliability Division members only) visit asq.org/reliability

To sign up for the free and available to anyone live webinars visit reliabilitycalendar.org and select English Webinars to find links to register for upcoming events

http://reliabilitycalendar.org/The_Reliability_Calendar/Webinars_‐_English/Webinars_‐_English.html

1

Why do a DesignedExperiment?

- What is it?- Why do I need it?- How do I do it?- Can it solve my problem?

Jim Breneman9/8/2011

What is Design of Experiments?

• A designed experiment is a test or series of tests in which purposeful changes are made to the input variables of a process or system so that we may observe and identify the reasons for changes in the output response…

Doug Montgomery

29/8/2011

Why do I need a Designed experiment?General Model of a Process or System

Output

yInputs Transformation Vehicle

...w1 w2 wp

Controllable factors

...z1 z2 zqUncontrollable factors

x1 x2… xpWe often want toMinimizeMaximize

orReduce variabilityin the Output(y)

NOISE

NOISE

Process

39/8/2011

Where does a Designed Experiment fit in?

Let’s look at Deming’s PDCA process

Plan

Check Do

Act

DefineControl

Improve Measure & Analyze

and then the DMAIC process

Major DOE use49/8/2011

Step 1: What’s the objective of Your Experiment?

• Comparative objective: – Primary goal is make a conclusion about one a-priori important factor.– i.e. is this factor “significant” (and possibly what level maximizes or minimizes the

response)• Process Improvement objective (Sequential experimentation):

– Step 1: Screening…the primary purpose of the 1st experiment is to select or screen outthe few important main effects from the many less important ones.

– Step 2: Followed up with experiment(s) to define the important 2-factor interactions, (and any 3-factor interactions that may be important based on experience).

• Response Surface objective: – The experiment is designed to allow us to estimate interaction (and even quadratic

effects), and to optimize the response or responses.– Each factor is usually at 3 levels.

59/8/2011

Step 2 What level of Evidence will I accept?

I. Controlled Trials with complete randomization.II. Empirical Evidence.

a) Controlled Trials without complete randomization.b) Case directed studies. Carefully observing cases as they occur. c) Multiple Time Sequence Studies (Looking back through data files for

patterns and drastic changes) “Scientific Studies have shown….”

III. Delphi (Agreement between a group of knowledgeable “experts”)“8 out of 10 doctors recommend …..”

IV. Personal Antidote (“In my experience”)

V. Personal bullying (“I think we should do it this way”)

DOE

69/8/2011

Step 3: The DOE Roadmap

D

M

A

I

C

Brainstorm

Sequential experimentation

DesignExperiment

Run Experiment& Collect Data

Analyze data/Interpret results

Choose variables& levels

Run confirmationtest

Incorporate into designor process

79/8/2011

The Strategy of Experimentation

Region of Interest

Region of Operability

1. Screening experiments to find the mountain range

2. Factorial/Fractional factorialexperiments to get close to the peak.

3. Response Surface Modelingto “climb” it.

89/8/2011

Review of BasicsLanguage

• Factor: An independent variable. This is what we control and change in an experiment. A factor is often generically referred to as xi.

Examples: Reaction Temperature, Bake Time, Fuel flow, Stress

• Factor Setting or Level: A particular value for a factor.For example, the factor Bake Temperature might have a setting of 275° F. Bake

Temperature is the factor, 275° F is one of the levels.

• Experimental Run: A particular combination of factor settings.

For example, one run in an experimental design (for say a composite piece) might call for a Bake Temperature of 275° F, a Bake Time of 30 minutes, and a Bake pressure of 5Atm.

99/8/2011

Review of BasicsLanguage

• Experimental Design: The complete set of runs that we plan to do. It is sometimes called the Design Matrix. Experimental design in general is often referred to as DOE or DOX (Design of Experiments).

• Response: A dependent variable. The level of the response is measured rather than controlled like a factor. It is referred to as a response because we think its level will change in response to changes in the factor settings.

One of the goals of experimental design is to relate changes in the factor levels to measured changes in the response values.

Examples of Typical Response Variables: Tensile Strength, Elongation, Thrust

• Factor Effect: The way in which changes in the level of a factor translate to changes in the response level.

109/8/2011

Review of BasicsLanguage

• Interaction: When the effect of a factor depends on the level of another factor, the two factors are said to interact:

Life(hrs)

Stress

Temp 1

Temp 2

Life=f(Stress, Temp)

119/8/2011

In General1. Several factors (or main effects) will be significant

2. Some two-factor interactions will be significant

3. Very few (if any) three-factor and higher order interactions will be significant

Concentrate on main effects and 2-factor interactions in your experiments.However, if a three-factor interaction is perceived to exist, then include it in theexperiment!

General Observations of DOEs

129/8/2011

Why Do Statistically Designed Experiments?

• Statistically designed experiments can detect and describe factor-factor interactions. Experiments that vary only one factor at a time and trial-and-error

experiments cannot.

• Statistically designed experiments offer more precise estimates of factor effects for the same number of runs compared to a one-factor-at-a-time (ladder) study.

This is because DOE’s use “hidden replication” and the power of averaging to see through noise.

Let me illustrate this with an engineering example.139/8/2011

DOE vs One-Factor-at-a-Time(OFAT)

An engineer performed an experiment on a new pieceof equipment . As a function of three factors:• Cooling Temp(°F)• Air Temperature (°F)• Metal temperature (°F)The objective was to maximize the response (y variable);

in this case, part life.The engineer performed the experiment as a one-factor-at-a-time for three factors in 15 runs

Cooling Temp Deg F

Air Temp Deg F

Metal Temp Deg F

600 2500 1900700 2500 1900800 2500 1900900 2500 1900

1000 2500 1900800 2300 1900800 2400 1900800 2500 1900800 2600 1900800 2700 1900800 2500 1700800 2500 1800800 2500 1900800 2500 2000800 2500 2100

149/8/2011

DOE vs One-Factor-at-a-Time

Illustrating this, we see that we can estimate that effect, both linear & quadratic; however, we cannot estimate interactions.

Cooling Temp(°F)

Air Temp (°F)

MetalTemp (°F)

600

700

800

1000

900

2500

2300

2600

2400

2700

159/8/2011

DOE beats OFAT – Round 1The Box Behnken designed experiment shown here and in the accompanying figure could have been performed instead. Both the One-Factor-at-a-Time and the designed experiments have 15 runs( if three center points are used in the Box Behnken design to make the design rotatable and to provide an estimate of natural variability).And, the Box-Behnken estimates interactions and their importance!

Coo

ling

Tem

p(°F

)

169/8/2011

Cooling Temp(°F)

Air Temp (°F)

MetalTemp (°F)

600

700

800

1000

900

2500

2300

2600

2400

2700

Coo

ling

Tem

p(°F

)

OFAT vs DOE – Round 2DOE’s “Hidden replication” beats OFAT

1 point

4 points

1 point

189/8/2011

DOE Review Planning Carefully

• DOE provides a useful framework for applied experimentation. However, there’s no magic involved and one of its advantages is that it forces some rigorous thinking before an experiment is started

199/8/2011

DOE Review Planning Carefully

1. What do we want to have accomplished when the experiment is finished?

2. What responses are in my job objectives?

3. How well can we measure these responses?

4. What factors are likely to cause these responses to vary?

5. How many factors can I reasonably investigate in a single experiment?

6. Over what range should they be varied?

209/8/2011

ReviewPlanning Carefully

How will I manage the noise.1. Consider making the noise factor into an experimental factor

for study.2. Hold the noise factor as constant as possible during the

experiment.3. Randomize the experiment.4. Measure the noise factor levels for future analysis

(covariates).5. Ignore it.

219/8/2011

ReviewNoise Management

• Noise makes the effects of controllable factors more difficult to see.

• If it happens that noise variables interact with controllable variables, the conclusions we draw from an experiment will only be valid at the noise variable settings experienced during the experiment.

• As a result, the experiment may not repeat at a later time.

i.e there may be an indication that noise variables are present with larger effects than the controllable variables!

229/8/2011

Types of Designed Experiments

– One-way ANOVA • Randomized Complete Block Design (RCBD)• Latin Square (& Graeco-Latin Square) Designs• Balanced Incomplete Block Designs

– Screening Designs– Factorial, Fractional Factorial Designs– Response Surface Designs– Custom Designs– Nested Designs– Split-Plot Designs– Mixture experiments

239/8/2011

Can DOE solve my problem?Model Assumptions

The Model: Y=b1X1 + b2X2 + b12X1X2 + Noise

1. Each factor can be varied independently of the others.2. All points in the experimental region are feasible:

There’s a difference between “infeasible” experimental runs and those that are merely expected to give “bad” results.

3. Factors are continuous (so a center point makes some sense)4. The experimental noise is consistent across the design space.5. All factors can be run across the other factors.6. Runs can be done in a random order.7. There is no curvature in the model.8. There are no CONSTRAINTS on Resources.

Responses:1. Performance2. $3. Safety

249/8/2011

A DOE Path ForwardAssumptions If No, then What

1Each factor can be varied independently of the others. Mixture or D-Optimal

2All points in the experimental region are feasible: D-Optimal

3Factors are continuous (so a center point makes some sense) Categorical Factorials

4The experimental noise is consistent across the design space. Blocking

5 All factors can be run across other factors Nested Designs

6 Runs can be done in a random order. Split Plots

7 There is no curvature in the model.RSM (Central Composite Models)

8 There are no CONSTRAINTS on Resources. Fractional Factorials

259/8/2011

0. Examples Of Experimental DesignsFull Factorial Experiment

D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2

G1

G2

G1

G2

G1

G2

G1

G2

E1

=

F1

F2

F1

F2

A1 A2

B1 B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

1/2 FFE

D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2

G1

G2

G1

G2

G1

G2

G1

G2

A1 A2

B1 B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

E1

F1

F2

E2

F1

F2

1/4 FFE

D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2

G1

G2

G1

G2

G1

G2

G1

G2

A1 A2

B1 B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

E1

F1

F2

E2

F1

F2

1/8 FFE

D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2 D1 D2

G1

G2

G1

G2

G1

G2

G1

G2

A1 A2

B1 B2 B1 B2

C1 C2 C1 C2 C1 C2 C1 C2

E1

F1

F2

E2

F1

F2

269/8/2011

But, you could start by doing 8 experiments with these 7 variables to find which are MOST important

Taguchi L8 Orthogonal Array or Plackett-Burman8-run screening design

VariablesTrial A B C D E F GNo. 1 2 3 4 5 6 71 1 1 1 1 1 1 12 1 1 1 2 2 2 23 1 2 2 1 1 2 24 1 2 2 2 2 1 15 2 1 2 1 2 1 26 2 1 2 2 1 2 17 2 2 1 1 2 2 18 2 2 1 2 1 1 2

279/8/2011

Before we proceed, one more “truism:” After completing an experiment and analyzing the significant factors, remember to recognize

the difference between practical and statistical significance.

Practical Importance

Statistical Significance

Yes No

Yes

No

Move Forward

Stop

Stop (document Lessons

learned)

Take moredata?

Why not thinkabout this

at the beginning?

289/8/2011

Summary• What is it?

A designed experiment is a test or series of tests in which purposeful changes are made to the input variables of a process or system so that we may observe and identify the reasons for changes in the output response…

• Why do I need it?Efficiency with hidden replication and interaction detection.

• How do I do it?Multi-factor sequential experimentation

• Can it solve my problem?

Yes, very often!299/8/2011

RAMS Course Info• 8-hour “Introduction to DOE” course,

[Jan. 26 1-5 pm + Jan. 27 8-Noon] following RAMS conference

• Registration available soon at: www.rams.orgJan. 23-26 in Reno, NV

• Course Topics:*Background: Historical Perspective *Fractional Factorial Designs*Completely Randomized Designs *Response Surface Designs*Factorial Designs *In-Class Exercises

309/8/2011

ThaaaatsAll

Folks!!

Questions??319/8/2011