Prediction variance

Sampling plansGiven a domain, we can reduce the prediction error by good choice of the sampling pointsThe choice of sampling locations is called design of experiments or DOEThe simplest DOE is full factorial design where we sample each variable (factor) at a fixed number of values (levels)Example, with four factors and three levels each we will sample 81 pointsFull factorial design is not practical except for low dimensions

Prediction variance for full factorial designRecall that standard error (square root of prediction variance is We start with simple design domain: BoxCheapest full factorial design: two levels (not good for quadratic polynomials)For a linear polynomial standard error is then

Maximum error at vertices

Why do we get this result?

Designs for linear RSTraditionally use only two levelsOrthogonal design when XTX is diagonalFull factorial design is orthogonal, not so easy to produce other orthogonal designs with less points.Stability: Small variation of prediction variance in domain is also desirable property

Example 4.2.2Compare an orthogonal array based on equilateral triangle to right triangle at vertices (both are saturated) Linear polynomial y=b1+b2x1+b3x2

For right triangle we obtained

ComparisonPrediction variances for equilateral triangle

The maximum variance at (1,1) is three times larger than the lowest one.For right triangleMaximum variance is nine times the lowestA fairer comparison is when we restrict triangle to lie inside boxThe prediction variance is doubled. Maximum error and stability are still better, but variance in coefficients is not as good.

Quadratic RSNeed at least (n+1)(n+2)/2 pointsNeed at least three points in every directionSimplest DOE is three-level, full factorial designImpractical for n>5Also unreasonable ratio between number of points and number of coefficientsFor example, for n=8 we get 6561 samples for 45 coefficients.My rule of thumb is that you want twice as many points as coefficients

Central Composite DesignIncludes 2n vertices, 2n face points plus nc repetitions of central pointDistance of face point variesCan choose so toachieve spherical designachieve rotatibility (prediction variance is a spherical function)Stay in box (face centered) FCCCD Still impractical for n>8

Spherical CCD

From Myers and Montgomerys Response Surface Methodology. Figure 7.4 in 1995 edition (Fig. 7.5 on next slide)

Spherical CCD for n=3

Repeated observations at originUnlike linear designs, prediction variance high at originRepetition at origin decreases variance there and improves stabilityWhat other rational for choosing the origin for repetition?Repetition also gives an independent measure of magnitude of noiseCan be used also for lack-of-fit tests

Without repetition (9 points)

Contours of prediction variance for spherical CCD design.From Myers and Montgomerys Response Surface Methodology. Figure 7.10 in 1995 edition (Fig. 7.11 on next slide)

Center repeated 5 times (13 points).

d=ccdesign(2,'center', 'uniform')d = -1.0000 -1.0000 -1.0000 1.0000 1.0000 -1.0000 1.0000 1.0000 -1.4142 0 1.4142 0 0 -1.4142 0 1.4142 0 0 0 0 0 0 0 0 0 0

Variance optimal designsFull factorial and CCD are not flexible in number of pointsStandard error in coefficients

A key to most optimal DOE methods is moment matrix

A good design of experiments will maximize the terms in this matrix, especially the diagonal elementsD-optimal designs maximize determinant of moment matrixInversely proportional to square of volume of confidence region on coefficients

ExampleGiven the model y=b1x1+b2x2, and the two data points (0,0) and (1,0), find the optimum third data point (p,q) in the unit square.We have

So that the third point is (p,1), for any value of pFinding D-optimal design in higher dimensions is a difficult optimization problem often solved heuristically

Matlab example>> ny=6;nbeta=6;>> [dce,x]=cordexch(2,ny,'quadratic');>> dce' 1 1 -1 -1 0 1 -1 1 1 -1 -1 0scatter(dce(:,1),dce(:,2),200,'filled')>> det(x'*x)/ny^nbetaans = 0.0055With 12 points:>> ny=12;>> [dce,x]=cordexch(2,ny,'quadratic');>> dce' -1 1 -1 0 1 0 1 -1 1 0 -1 1 1 -1 -1 -1 1 1 -1 -1 0 0 0 1scatter(dce(:,1),dce(:,2),200,'filled')>> det(x'*x)/ny^nbetaans =0.0102

Other criteriaA-optimal minimizes trace of inverse of moment matrix, minimizes the sum of the variances of the coefficientsG-optimality minimizes the maximum of the prediction variance.

ExampleFor the previous example, find the A-optimal design

Minimum at (0,1)