MotivationAssessment and Scope
ResultsConclusions
Evaluating and Constructing Designs forRobustness to Unusable Observations
Byran Smucker1, Willis Jensen2, Zichen Wu1, and Bo Wang1
1Miami University, Oxford, OH2W.L. Gore and Associates, Flagstaff, AZ
March 2, 2017IFPAC Annual Meeting, North Bethesda, MD
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Satire from The Onion
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Optimality vs. Robustness
A Paradox?
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Box and Draper (1975)
Addressed 14 ways a response surface design can be good.
Generate information in region of interest
Ensure fitted values are close as possible to true values
Detect lack of fit
Allow for transformations
Allow for blocks
Allows for building up of sequential experiments
Provide internal estimate of variability
Robust to wild observations and non-normality
Uses minimum number of runs
Allows for graphical assessment
Simple to calculate
Robust to the factors settings (the x’s)
Do not require a lot of levels in the x’s
Provide check of constant variance assumption
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
15th Way a Design Can Be Good
Robustness to Missing Observations
Note: We distinguish between robustness to outliersand robustness to missing observations.
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
The Problem
How Many Have Ever Had an Experiment WithMissing Observations?
Prevalence of Missing Observations
Siddiqui (2011) suggested 1-10% of observations are wildCo-author’s experience suggests that values of 0-20% arepossible
Assume Observations are Missing at Random
Does not always hold: could be that factor ranges chosenpoorlyAssume the missing values are not a result of the factor levels
Why Not Just Redo the Missing Runs?
Sometimes this is possibleOther times extremely costly or impossible to do
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Generic Example of Modern Industrial Process
These characteristics lead to difficulty in redoing missing runs
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Characteristics of Modern Industrial Processes
Increasing process complexity (more steps and more variables)
Increasing equipment scales (more challenging to useequipment for experiments)
Increasing supply chain complexity (raw materials comingfrom multiple suppliers at multiple places in the process)
Increasing physical distances covered by process (differentsteps in different facilities and geographies)
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Discussion Question
What other characteristics make it more
difficult to redo missing runs?
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Regression
Standard regression model: y = Xβ + �.
Variance-covariance matrix of the least squares estimators is
Cov(β̂) = σ2(XTX)−1
The XTX matrix is called the information matrix.
We consider two types of models:
f T (x)β = β0 +k∑
i=1
βixi
f T (x)β = β0 +k∑
i=1
βixi +k−1∑i=1
k∑j=i+1
βijxixj +k∑
i=1
βiix2i .
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Classical Designs
Fractional factorial designs and Central Composite designs
http://www.itl.nist.gov/div898/handbook/pri/section3/pri3361.htm
Smucker et al. Robustness of Designs to Missing Observations
http://www.itl.nist.gov/div898/handbook/pri/section3/pri3361.htm
MotivationAssessment and Scope
ResultsConclusions
Optimal Designs
Choose design to have desirable statisticalproperties.
1. D-optimal design relates to variance-covariance matrix of leastsquares estimators:
Assume f → Expand to X→ Choose ξn to maximize |XTX|
2. I-optimal designs minimize the average prediction variance ofthe design across design space.
3. Designs constructed to be robust to missing observations.
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Measuring Design Quality for First-Order Models
Use the determinant of the information matrix,∣∣XTX∣∣.
What if m observations are missing?
DF (i ,m) =
∣∣∣XTn−m,iXn−m,i ∣∣∣|(X*)Tn (X*)n|
1/p , i = 1, 2, . . . ,(nm
)
This metric gives a sense, in an absolute way, of how muchinformation is being lost when m runs are missing.
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Measuring Design Quality for Response Surface Models
I-optimal designs because they seek to minimize the averageprediction variance across the design space:
I =
∫Rf T (x)(XTX)−1f (x) dx ,
If m observations are missing?
IF (i ,m) =I *n
In−m,i, i = 1, 2, . . . ,
(n
m
),
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
How to Assess Impact of Missing Runs
Examine how much information classical and optimal designs losewhen runs go missing.
For instance, if 1 run is missing from an n-run design, we willcompute the D-efficiency for each possible (n − 1)-run design andlook at its distribution.
We’ll look at first- and second-order response surface models,various design sizes, and a small number of missing runs(m = 0, 1, 2, 3).
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
First-order Models: k = 5
Figure: D-efficiencies for possible main effects designs, for k = 5 andn = {8, 10, 12, 16}, according to the number of missing runs.
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Second-order Models: k = 3
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Conclusions: Screening Designs
1 Missing observations have relatively little impact on first-orderdesigns unless design is small.
2 Fractional and D-optimal designs are robust to missingobservations.
3 If you are worried about losing a run or two, add a few runs atthe outset.
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Conclusions: Response Surface Designs
1 No evidence that optimal designs are less robust to missingobservations than classical designs
2 Optimal designs are more efficient, and the efficiency holds upwhen a few observations are lost.
3 Missing-robust optimal designs are better when originalnumber of runs is small
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Bottom Line
1. Optimal designs and classical designs have similarrobustness properties, in terms of missing runs.
2. For severely resource-constrained experiments for which youare concerned about missing observations, either (1) add a fewextra runs; or (2) consider using a missing-robust design.
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Some Unanswered Questions
What is the impact on spherical regions (eg. 5 level CCD vs.optimal)?
What is the impact on other types of designs (blocking,split-plots, mixtures, etc)
Are there better metrics to assess impact (ability to detectsignificant effects, width of intervals, bias and variance inpredictions)?
How could we obtain a robustness criteria based onI-optimality?
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Satirical headline: Professor celebrates landmark publication thatwill be carefully read by two people (h/t Maria Weese)
ReferenceSmucker, B.J., Jensen, W., Wu, Z., and Wang, B. (2017+).Robustness of Classical and Optimal Designs to MissingObservations. Computational Statistics & Data Analysis. In press.http://www.users.miamioh.edu/smuckebj/
Smucker et al. Robustness of Designs to Missing Observations
http://www.users.miamioh.edu/smuckebj/
MotivationAssessment and Scope
ResultsConclusions
Quick Plug
community.amstat.org/isvc; contact: Byran Smucker([email protected])
Smucker et al. Robustness of Designs to Missing Observations
community.amstat.org/isvc
MotivationAssessment and Scope
ResultsConclusions
Questions?
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and Scope
ResultsConclusions
Extra Slide. Second-order Models: k = 5
Smucker et al. Robustness of Designs to Missing Observations
MotivationAssessment and ScopeResultsConclusions