ELSEVIER Computational Statistics & Data Analysis 26 (1998) 425-436

COMPUTATIONAL STATISTICS

& DATA ANALYSIS

A quantitative method for identifying active contrasts in unreplicated factorial designs based on the

half-normal plot

John Lawson a'*, Scott Gr imshaw a, Jason Burt b

"Department of Statistics, Brigham Young University, Provo, UT 84604, USA b Micron Technology Inc., Boise, ID 83707, USA

Received 1 January 1996; received in revised form 1 July 1997

Abstract

Normal or half-normal plots are often used to judge the significance of effect contrasts in unreplicated factorial or fractional factorial designs. Many quantitative procedures have also been proposed in recent literature to reduce the subjectivity of the graphical methods. In this paper we describe a new method for judging significance of effects that is both quantitative and graphical. The method consists of fitting a simple least-squares line and prediction limits to the half-normal probability plot. The rational for the statistic comes from recent papers by Lenth and Loh. This new method is a blend of Lenth's and Loh's method. It has the computational simplicity of Lenth's method with the graphical interpretation and increased power of Loh's method. We describe the calculations of this new statistic which is a supplement to the half-normal plot. We present a table of critical values for the statistic and show a simulation study to describe the power properties of the test. Finally, we present an example of the new technique using a data set from Box and Meyer.

The new statistic developed in this paper is a powerful quantitative tool for reducing the subjective- ness of the half-normal plot in accessing the significance of effects in unreplicated designs. It can be computed simply with commands available in standard statistical programs such as SAS, MINITAB or S-PLUS, and can be used graphically or numerically for more accuracy. © 1998 Elsevier Science B.V. All rights reserved.

Keywords: Normal plot; Half-normal plot; Unreplicated design; Least-squares fit; Prediction limits

* Corresponding author.

0167-9473/98/$19.00 © 1998 Elsevier Science B.V. All rights reserved P I I S0 1 6 7 - 9 4 7 3 ( 9 7 ) 0 0 0 4 0 - 6

426 J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425-436

1. Introduction

Analysis of unreplicated factorial and fractional factorial designs has been a much researched subject. The concept of effect sparsity is useful in this type of analysis. This concept, or paradigm, defined by Box and Meyer (1986), states that of the saturated set of contrasts that may be defined in these designs, usually only a few contrasts will be actually significant or "active" (i.e. nonzero). (To deal with the confusion that arises in the literature in distinguishing the difference between an effect and a contrast, Lenth (1989) suggests calling a contrast the difference between the means at the "high" and "low" levels, and calling an effect the high mean minus the overall mean, as in a typical ANOVA model.)

Graphical methods such as normal and half-normal probability plots are often used in determining which contrasts are significant in an orthogonal or indepen- dent set. Although judgement of significant contrasts based solely on plots is a subjective procedure, it is often used, especially when there is no replication and thus no measure of error available from the replicates. Testing of treatment and interaction contrasts in 2 k and 2 k-p designs are examples of when these graphical applications are useful.

In order to eliminate the subjectivity of the graphical procedures for detecting significant contrasts, many procedures have been proposed to quantify what can be seen visually.

In this article, we propose a new quantitative method which enhances the gra- phical half-normal plot. The rational for our procedure is based on methods proposed by Lenth (1989) and Loh (1992). It can be easily computed using standard statistical software, and can be represented graphically by adding confidence bands to the half-normal plot. We report on a simulation study conducted to compare the power of this new method with that of Lenth (1989).

2. Some quantitative methods suggested in the literature

In analyzing data from unreplicated experiments, a problem arises because there is no measure of error from replication in the experiment and hence, no measure of error standard deviation, a. The lack of any error measure creates a problem in analysis because, as Box and Meyer (1986) describe, it is difficult to determine which main and interaction contrasts are active or statistically significant. Daniel (1959, 1976) developed the use of normal and half-normal probability plots to help to identify significant or active contrasts from unreplicated fractional factorial experiments. Many researchers tend to shy away from the subjectiveness of judging significance based on plots, and in an attempt to placate this objectivity, many numerical analysis methods have been developed.

There are many numerical methods for analyzing unreplicated factorial experi- ments. Balakrishnan and Hamada (1994) compared a large number of the methods for orthogonal two-level designs using an extensive simulation study. Of the direct

J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425 436 427

methods they compared, most perfomed similarly with respect to power. They recommended two methods based on ease of calculation. One of these two methods was developed by Lenth (1989). Another two-stage method developed by Loh, (1992) had a familiar graphical representation based on the normal probabil- ity plot. Although this method was not included in the extensive simulation study conducted by Balakrishnan and Hamada (1994), Loh (1992) indicated that it was more powerful than Lenth's method.

The simple method for determining ac t i ve contrasts that was introduced by Lenth (1989) can be described as follows. Lenth uses a saturated orthogonal set of contrasts of interest, c ~, c2, . . . , Cm. Letting

So = 1.5 x median [c~], (1) J

Lenth then defines the "pseudo-standard error" (PSE) as

PSE = 1.5 x median [cjl. (2) [%1 < 2.5so

The PSE then may be used to calculate a margin of error in the normal way. For example,

ME = to.975:a x PSE, (3)

w h e r e t0.975; a is the 0.975th quantile o fa t distribution with d degrees of freedom. As suggested by Lenth, d = m/3 is commonly used. Using this ME, an approximate 95% confidence interval may be constructed around each ci. To account for the possibility of an inactive contrast exceeding the ME and a false conclusion being reached, Lenth then suggests calculating a simultaneous margin of error or "SME" defined as

SME = t~;a x PSE, (4)

where

7 = (1 + 0.951/")/2. (5)

A contrast is active if it exceeds the SME value and may be active if it exceeds the ME value. Balakrishnan and Hamada 's (1994) simulation studies showed that Lenth's procedure performed quite well.

Loh (1992) introduced a graphical method of identifying ac t i ve contrasts on a normal probability plot that has a corresponding numerical procedure. The basis of Loh's method is the fact that the coded factor levels (i.e., + or - ) are arbitrarily assigned by the data analyst for each factor. For example, in a 23 design there are 8 different ways to assign coded factor levels to one treatment combination i.e., (xl, x2, x3) = ( - 1, - 1, - 1) or ( - 1, - 1, 1), etc. The normal plot of effects will be different depending on how the coded levels are assigned to the factor levels. The first step in Loh's method involves finding the set of effect contrasts from each possible assignment of factor levels and then choosing the set that has the median contrasts closest to zero. Once the set of effect contrasts is chosen, the individual

428 J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425-436

contrasts are classified as either an outlier or an inlier. The outliers are the effect contrasts whose value exceeds the upper hinge (or 75th percentile plus 1.5 times the interquartile range) or is less than the lower hinge. The set of inlier effect contrasts are the complement of the outlier set.

Next Loh proposed to calculate the coefficients for two least-squares regression lines on the normal probability plot. One line is fit to all the effect contrasts and the other line is fit to the inlier set of contrasts only. The ratio of the slopes of the two lines is used as a preliminary test of significance. If this ratio exceeds a critical value, tabulated by Loh, a final significance test is applied to each contrast. This is done by placing Sheff6 prediction bands around the regression line that was fit to the inlier set of contrasts on the normal probability plot. Active effect contrasts are then identified as those that fall above the prediction bands. This determination can be done numerically or grapically.

Loh (1992) showed examples that implied that his procedure had more accurate significance levels and greater power in general than Lenth's procedure. The great advantage of Loh's method is that it adds objectivity and a numerical procedure to an already familiar and popular graphical method of identifying active contrasts. The disadvantage is that it is computationally impractical to look at 2 k sets of effect contrasts for any design involving k factors. In this paper we propose a new method based on the half normal plot that retains the appealing advantage of Loh's procedure while retaining the computational simplicity of Lenth's method.

3. A new quantitative method based on the half-normal plot

The purpose of this article is to study a numerical method of analysis for unreplicated fractional factorial designs that is a combination of Lenth's and Loh's methods, and based on the half-normal probability plot, rather than the full- normal plot. Trying all factor-level assignments, and then choosing the set that has the median contrast closest to zero, as in Loh's proposal, may become unwieldy, especially when there are many factors to be studied. Therefore, we propose using half-normal probability plots, since with these plots, all factor-level assignments result in the same values for absolute contrasts. We use the regression method described by Loh on the half-normal plot, rather than the normal plot. We develop a new table of critical values of R, for this procedure, and conduct a simulation study to compare the power of this new procedure to Lenth's procedure.

3.1. Description o f the new quantitative method based on half-normal plots

The new method can be described in four steps as follows. First, the set of out- liers and inliers of the absolute effects are determined by the preliminary method of Lenth, i.e., the outliers in the saturated set of absolute contrasts ci's are identified as those that have an absolute value greater than 2.5 x So, where So is given in

J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425 436 429

Eq. (1). Once the outliers are found, the "inliers" are determined (i.e., their comp- lement set).

Second, the least-squares line wi thout intercept will be fit to the model Ici(i) = f l lzi , where zi are the half-normal-order statistics approximated by

Zi = ~ - l i p " { - 1] 2 ' (6)

where p is an estimate of the percentile of the order statistic of the absolute contrasts [c[~/), such as p = (i - ½)/n or p = (i - 3)/(n + 5) and q>- 1( ) is the inverse normal C D F transform, and/~1 is the est imated slope.

Third, another least-squares line is fitted to the set of ordered pairs (c~, zi), where c~ are the inliers and the slope,/~2, is calculated. The ratio of the two regression slopes is called

P1 = - -

R, /~2" (7)

R, is the statistic used for the preliminary test of significance. No contrasts are deemed significant if R, does not exceed a tabled critical value for the given significance level 7. We determined significance limits through simulation for R, at ~ = 0.10, 0.05, 0.025, and 0.01, and n = 7, 8, 11, 15, 16, 17, 26, 31, 32, 35, 63, and 127.

If R, exceeds the significance level, the fourth step of the procedure is the const ruct ion of Scheff6 predict ion bands a round the least-squares regression fit to the inlier set. These bands may then be used to identify the active contrasts graphically; those that fall outside the prediction bands on a half-normal probabil- ity plot. Most s tandard statistical packages have options to compute these Scheff6 prediction limits in their regression routines, and it is easy to au tomate the whole procedure with a macro.

A numerical equivalent of this graphical procedure is accomplished by letting D be the largest absolute contrasts that is within the Scheff6 prediction bands. C* is then defined to be the max imum of C (given in Eq. (6) above) and D, i.e., C* =max(C, D). A contrast c/ is then identified as active if [ciJ > C*.

3.2. Tab le o f s i g n i f i c a n c e l imi ts

A table of critical values was developed for R,. The table is indexed by values of n that are c o m m o n numbers of contrasts in fractional factorial and or thogonal array designs, including n = 7, 8, 11, 15, 16, 17, 26, 31, 32, 35, 63, and 127. An accuracy of 0.01 was desired for the critical values in the table.

The following approach was used to generate the table of critical values. To begin a number of replicates were necessary to obtain the desired accuracy for the critical values was detrermined. A replicate consisted of generating 1000 r andom data sets. The outliers and inliers were determined as explained earlier, and two no intercept regression lines were fitted and the modified R, was calculated for each data set.

430 J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425-436

Table 1 Estimated percentage points of the null distribution of modified R,. Simulation sample size is 25000

n 90% 95% 97.5% 99%

7 1.265"* 1.534"* 1.889"** 2.506*** 8 1.196"* 1.385"* 1.606"* 2.026**

1l 1.161" 1.291 1.449" 1.740" 15 1.122 1.201 1.297" 1.447" 16 1.110 1.186 1.274 1.421" 17 1.106 1.178 1.260 1.377" 26 1.072 1.115 1.165 1.232 31 1.063 1.099 1.140 1.197 32 1.060 1.093 1.130 1.185 35 1.059 1.091 1.127 1.178 63 1.037 1.056 1.074 1.096

127 1.023 1.034 1.043 1.058

Note: Values accurate to within 0.01 unless otherwise indicated be- low. * Accurate to within 0.05. ** Accurate to within 0.10. *** See Table 2 for accuracy.

Table 2 95% confidence intervals for estimated percentage points of the null distribution of modified R 7

97.5% 99%

Lower limit Upper limit Lower limit Upper limit 1.661 2.116 2.140 2.872

T h e empi r i ca l 90th, 95th, 97.5th, a n d 99th percent i les of the d i s t r ibu t ion values were d e t e r m i n e d f r o m this set of 1000 R , values. Init ial ly, this was r epea t ed 20 t imes, a n d the s t a n d a r d dev ia t ion of each empi r ica l percent i le va lue was ca lcu la ted f r o m the 20 replicates. Solv ing for N in the fo l lowing formula :

0.01 = ( sZ /U) l /z , (8)

yielded the n u m b e r of repl icates required. In m o s t of the cases cons idered , 25 000 repl icates or less were sufficient. F o r some

values of n a n d cer ta in percent i les , the a ccu racy of 0.01 cou ld no t be ach ieved w i t h o u t huge n u m b e r s of s imula t ions . L o w e r accurac ies were accep ted for these cases a n d are ind ica ted by foo tno te s in the tab le of results. The s imula t ions were run using a SAS p r o g r a m tha t was wr i t ten to execute the new m e t h o d . Tab le s 1 a n d 2 con t a i n the o u t p u t of these s imula t ions , wi th the accu racy of each table va lue also listed.

J. Lawson et al . / Computational Statistics & Data Analysis 26 (1998) 425 436

Table 3 Contrasts and intermediate statistics for calculating R,

431

Label Con t r a s t Absolute Rank p = (r - 0.5)/15 z-score 95% Pred. cont ras t r 4 -1 ( (p + 1)/2) limit

A 0.06 0.06** 12 0.767 1.192 0.0607 B 0.25 0.25* 14 0.900 1.645 0.0794 AB - 0.01 0.01 3 0.167 0.210 0.0216 C 0.50 0.50* 15 0.967 2.128 0.0996 AC 0.0 00.0 1 0.033 0.042 0.0152 BC - 0.02 0.02 6 0.367 0.477 0.0321 ABC 0.0 0.0 2 0.100 0.126 0.0152 D 0.14 0.14" 13 0.833 1.383 0.0686 A D 0.03 0.03 10 0.633 0.903 0.0490 BD - 0.01 0.01 4 0.233 0.297 0.0250 ABD 0.02 0.02 7 0.433 0.573 0.0358 C D 0.04 0.04 11 0.700 1.036 0.0544 A C D 0.02 0.02 8 0.500 0.675 0.0399 BCD 0.01 0.01 5 0.300 0.385 0.0284 A B C D 0.02 0.02 9 0.567 0.784 0.0442

*These contras ts be long to the outl ier set. ** This is the value of D the largest absolute cont ras t within the Scheff6 limits.

4. Example of the new quantitative method

Consider the data set that is labeled Example 1 in Box and Meyer (1986). The contrasts computed from Yates algorithm on this data are shown in the second column of Table 3 below. These data are an example of an unreplicated 2 4 design with 16 runs. It involves the following four factors, each with two levels: Load (A), Flow (B), Speed (C), and Mud (D).

The median absolute contrast is 0.02, and 2.5 x So = 0.075. The absolute values of the contrasts are shown in the third column of Table 3, and those larger than 2.5 x So = 0.075 are labeled outliers and indicated with asterisks in Table 3.

The fourth column in Table 3 shows the ranks of the absolute contrasts, the fifth column shows the empirical percentiles of the absolute contrasts, and the sixth column shows their estimated half-normal-order statistics. Using all the data in columns 3 and 6, a straight line (with no intercept) was fit by least squares resulting in the estimated slope/~1 = 0.130. The least-squares fit was repeated deleting the outliers, resulting in the second estimated slope/~z = 0.038. Next R, = ~i/f12 was calculated to be 3.41. Since this is greater than the 95% critical value of 1.201 (n = 15), we may conclude with 95% confidence that there are significant contrasts in this example.

In order to determine which contrasts were significant, the critical limit C* was calculated. First the upper 95% prediction limits were determined for the least- squares line with slope fi2. These are shown in the last column in Table 3. The

432 ,L Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425-436

o.~ -~

m 0.4~

"~ 0.3

" 4 0.2

0.1

0.0

© B

o D

I I I

0 1 2

Z

L

.=._N - - - L95 - - Precl

Fig. 1. H a l f - n o r m a l p r o b a b i l i t y p lo t of E x a m p l e 1 f rom Box a n d M e y e r (1986) wi th Scheff6 p r e d i c t i o n b a n d s

largest absolute contrast that is not greater than its corresponding prediction limit is D -- 0.06 (indicated by double asterisks in Table 3). Finally, C* = max(D, C) = max(0.06, 0.12) -- 0.12. Thus, contrasts B, C, and D are deemed significant because their absolute values are greater than C*.

A half-normal probability plot of the contrasts was constructed along with the 95% Scheff6 prediction bands surrounding the regression line of the inlier set. This plot is the graphical representation of the new method and is shown in Fig. 1. The inference that contrasts B, C and D were significant is the same conclusion reached by Box and Meyer (1986) and also by Loh (1992) using different methods with the same data. A SAS program was developed that will perform the new quantitative method based on the half-normal plot, and create the half-normal probability plot of absolute contrasts with prediction bands.

5. Power study

A power study was conducted in which simulations were run to evaluate the new quantitative method and compare it to Lenth's method for the power of detecting "active'contrasts. Because of the large number of possible designs, this study was restricted to a simple 18 run design (n = 17), i.e., an L18 orthogonal array. This design was an orthogonal 18 run, 2 x 37 design that included two degrees of freedom of the AB interaction.

We chose Lenth's method as the benchmark for the power study for three reasons. (1) It was one of the two recommended by Balakrishnan and Hamada (1994), and had similar power characteristics two most of the direct methods compared in that study. (2) Although there are more powerful methods than Lenth's, they have other disadvantages such as reqiring more computational intensity or having non-monotonic power functions (see Balakrishnan and

J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425-436 433

Hamada, 1994). (3) Loh's method which is a partial motivation for the method described in this paper does not generalize directly to three-level factors studied in this simulation, and therefore could not be compared.

Two variables were studied in this simulation experiment. These variables were the size of the active contrast(s) and the number of active contrasts.

5.1. Variables of simulation experiment

The first variable to be considered in the simulation study was the active contrast size. We define the active contrast to be the regression coefficient for the linear or quadratic contrasts of three-level factors in the L18 design. Size of the contrast is measured as the number of standard deviations of experimental error. Therefore, the further a contrast is from its mean of zero, the more active it may be considered to be. Seven different sizes of active contrasts were compared in this study, which were multiples of the standard deviation of the raw data. Values of 0.5, 0.75, 1.0, 1.25, 1.5, 2.0, and 3.0 were used in the study.

The second variable considered in the simulation experiment was the number of contrasts that were set to be active, i.e. nonzero. Four cases were included: one where one contrast was nonzero, a second where two contrasts were nonzero, a third where four contrasts were nonzero, and the fourth where six contrasts were nonzero.

5.2. Methods of simulation

In order to study the power of the new quantitative method and of Lenth's method, random, standard normal data was generated. For each variable combina- tion, contrasts were selected and then set to be active by adding the number of standard deviations designated in the variable combination. The new method and Lenth's method were then separately used to determine which contrasts were significant. The empirical power was then calculated to be the percentage of trials in which all of the contrasts that were set to be active were correctly identified as active. This simulation process was repeated 10 000 times for each variable combi- nation using SAS programs. The power values were calculated for each variable combination.

5.3. Simulation results

These results are shown graphically in Figs. 2, 3, 4, and 5 which show the empir- ical power curves for the new method and Lenth's method for 1, 2, 4, and 6 active contrasts, respectively. In each case, the new method based on the half- normal plot is uniformly more powerful than Lenth's method when sigma is less than 3.0. This result is similar to what Loh (1992) found with his regression method based on the full-normal plot. He also implied that his method is more powerful than Lenth's. Therefore, both Loh's method based on normal probability plots, or the new

434 J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425 436

0

1.00 - . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . ~ .

o , , . . . . . . . . . . . . . .

o . , o . . . .

0 . 2 , - . . . . . . . . .

0.00 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I $ I I I m

0.5 1.0 1.5 2 .0 2 .5 3 .0

Sigma

Fig. 2. Power of New Method vs. Lenth's Method, 1 active contrast

~ - N e w

L e n t h

O Q.

1 . 0 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . - -

f

0 . 0 0 . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . I I I I I I

0.5 1.0 1.5 2.0 2.5 3.0

Sigma

u - New ] L e n t h

Fig. 3. Power of New Method vs. Lenth's Method, 2 active contrasts

method based on half-normal probability plots is more powerful than Lenth's method. However, the new method is much easier to compute than Loh's method.

Other observations can be made from the figures: (1) It be seen that the power of both methods monotonically increase as a function of the size of the active contrasts in ~r units. (2) It can be seen that the power of Lenth's method monotoni- cally decreases as the number of active contrasts increases from 1 to 6. This is not true for the new method. By comparing Figs 2 and 5 with Figs 3 an 4, it can be seen that the new method is actually more powerful in detecting 2-4 active contrasts (approximately 12-24% of the total number) than it is for detecting fewer or greater numbers of active contrasts.

J. Lawson et al. /Computational Statistics & Data Analysis 26 (1998) 425-436 435

0 13.

1.00

0.75 --

0.50 --

0.25 --

0 . 0 0 - -

_ ~ _ J " I f

. . . . . . . . . . . . . . . . . . . . . . . . . . . . .

I I I I I I

0.5 1.0 1.5 2.0 2.5 3.0

Sigma

_ _ _ New

L e n t h

Fig. 4. Power of New Method vs. Lenth's Method, 4 active contrasts

O)

0 Q.

1.00 --

0.75 --

0.50 --

0.25 --

0 . 0 0 - -

. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

. . . . . . . . . . . . .

. . . . . . . . . . . . . . . . . . . . . .

/ / _Zii!i'_iiiiiiiiiiiiiiiiiiiiii_. I I I I 1 I

0.5 1.0 1.5 2.0 2.5 3.0

Sigma

- - - New [

L e n t h

Fig. 5. Power of New Method vs. Lenth's Method, 6 active contrasts

6. Summary

This study presents a quantitative method of detecting significant contrasts that is a blend of methods proposed by Lenth and Loh. It retains or actually improves on the computational simplicity of Lenth's method and the graphical interpretation of Loh's method. This method is preferred over Loh's original method because it does not require examination of all possible factor level assignments, and is easy to compute using standard statistical programs. The new method is also preferred over Lenth's method because it does not require looking up percentiles of the Student's t-distribution with fractional degrees of freedom, it retains the familiar graphical interpretation of Loh's method, and is uniformly more powerful than

436 J. Lawson et al. / Computational Statistics & Data Analysis 26 (1998) 425 436

Lenth's. The new method consists of fitting two zero intercept regression models to the half-normal plot and computing the Scheff6 prediction limits. The results can be displayed graphically as in Fig. 1, or simply calculated numerically.

Acknowledgements

The authors would like to thank the referees who made valuable suggestions to improve this paper, and also Chad Crawford for help in running the SAS simula- tion programs.

References

Balakrishnan, N., Hamada, M., 1994. Analyzing unreplicated factorial experiments: A review with some new proposals. IIQP Research Report, University of Waterloo, Waterloo, Ontario, Canada, RR-94-12.

Box, G.E.P., Meyer, R.D., 1986. An analysis for unreplicated fractional factorials. Technometrics 28, 11-18.

Daniel, C., 1959. Use of half-normal plots in interpreting factorial two-level experiments. Technomet- rics 1,311-341.

Daniel, C., 1976. Applications of Statistics to Industrial Experimentation. Wiley, New York. Lenth, R.V., 1989. Quick and easy analysis of unreplicated factorials. Technometrics 31(4), 469-473. Loh, W.L., 1992. Identification of active contrasts in unreplicated factorial experiments. Comput.

Statist. Data Anal. (1992) 14, 135-148.