Statistical tests for replicated experiments Normal probability plots are a less formal diagnostic tool for detecting effects Normal probability plots

Statistical tests for Statistical tests for replicated experimentsreplicated experiments

Normal probability plots are a Normal probability plots are a lessless formal diagnostic tool for detecting formal diagnostic tool for detecting effectseffects

F-tests and t-tests provide a F-tests and t-tests provide a statistical teststatistical test of factor effects of factor effects


Statistical tests Statistical tests areare possible for possible for unreplicated designs (unreplicated unreplicated designs (unreplicated pilot studies are essential tools in pilot studies are essential tools in sample size calculations)sample size calculations)

We will first focus on statistical tests We will first focus on statistical tests for replicated designsfor replicated designs

Statistical tests for Statistical tests for replicated experiments--replicated experiments--

ExampleExample Response--Pulse rate of subjectResponse--Pulse rate of subject FactorsFactors

– Treatment (Energy Drink, Placebo) Treatment (Energy Drink, Placebo) – Setting (Moderate, Difficult)Setting (Moderate, Difficult)– Machine (Stair climber, Recumbent bike)Machine (Stair climber, Recumbent bike)


ExampleExampleT S M Responses Mean Variance

-1 -1 -1 109, 112, 116, 112 112.25 8.25

1 -1 -1 120, 123, 125, 120 122 6

-1 1 -1 130, 118, 118, 118 121 36

1 1 -1 125, 123, 130, 124 125.5 9.67

-1 -1 1 108, 110, 113, 111 110.5 4.33

1 -1 1 121, 123, 123, 118 121.25 5.58

-1 1 1 113, 117, 122, 119 117.75 14.21

1 1 1 127, 126, 125, 125 125.75 0.917


ExampleExampleT S M TS TM SM TSM

8.25 6.00 -1.375 -2.00 1.125 -.125 .625


ExampleExample

Standardized Effect

Perc

ent

76543210-1-2

99

95

90

80

70

605040

30

20

10

5

1

Factor NameA AB BC C

Effect TypeNot SignificantSignificant

B

A

Normal Probability Plot of the Standardized Effects(response is Pulse, Alpha = .05)


Effect sizes depend on the Effect sizes depend on the measurement scalemeasurement scale

Statistical tests are based on Statistical tests are based on standardizedstandardized effects effects

To compute standardized effects, To compute standardized effects, start with an estimate of start with an estimate of experimental errorexperimental error


Experimental error can be Experimental error can be summarized by the square root of summarized by the square root of the variance of the background the variance of the background noise (the standard deviation)noise (the standard deviation)

The experimental error measures The experimental error measures variation in a single observationvariation in a single observation


The variance is best estimated by The variance is best estimated by the Mean Square for Pure Error the Mean Square for Pure Error (MSPE)(MSPE)

8

) variancescell of Sum(MSPE


ExampleExample

The standard deviation for each run The standard deviation for each run is ~3 beats per minuteis ~3 beats per minute

26.3

62108

9221146258

MSPE

....

MSPE


While the standard deviation for a While the standard deviation for a single response is the square root of single response is the square root of MSPE, the standard deviation of an MSPE, the standard deviation of an effect (its standard error) is:effect (its standard error) is:

n

MSPEk 22


We divide an effect in a k-factor We divide an effect in a k-factor experiment with n replications (e.g., experiment with n replications (e.g., A) by its standard error to compute a A) by its standard error to compute a t-test statistic :t-test statistic :

nMSPE

EffectT

k 22


Test statistics for other effects are Test statistics for other effects are computed similarlycomputed similarly

U-do-it: Calculate the T-statistics of U-do-it: Calculate the T-statistics of all effects for the Exercise dataall effects for the Exercise data


When an effect is negligible, T has a When an effect is negligible, T has a t-distributiont-distribution

The shape of the t-distribution curve The shape of the t-distribution curve depends on the number of replicates depends on the number of replicates (“degrees of freedom”=2(“degrees of freedom”=2kk(n-1))(n-1))

The t-distribution curves have The t-distribution curves have slightly more spread than the bell-slightly more spread than the bell-shaped (“normal”) curveshaped (“normal”) curve

Statistical tests for Statistical tests for replicated experiments--t replicated experiments--t curve for 3-factor designcurve for 3-factor design

43210-1-2-3-4-5

0.4

0.3

0.2

0.1

0.0

4 reps2 reps

2 replications

4 replications

Statistics tests for replicated Statistics tests for replicated experimentsexperiments

If |T| is larger than the 99.5If |T| is larger than the 99.5thth or 97.5 or 97.5thth percentile of the t distribution, an percentile of the t distribution, an effect is significanteffect is significant

These percentiles are commonly These percentiles are commonly found in textbooks (but found in textbooks (but please please use a use a computer package instead)computer package instead)

Statistical tests for Statistical tests for replicated experiments--t replicated experiments--t critical value for 3-factor critical value for 3-factor

design (n=4)design (n=4)

43210-1-2-3-4

0.4

0.3

0.2

0.1

0.0

.025

t dist. with 3 reps (24 df)

2.064


Sometimes, twice the area to the Sometimes, twice the area to the right of |T| is reported as a right of |T| is reported as a p-valuep-value. . Small p-values suggest that a Small p-values suggest that a standardized effect is distinguishable standardized effect is distinguishable from background noise from background noise

You definitely need a computer to You definitely need a computer to compute p-values--in the following compute p-values--in the following example, the p-value for the M effect example, the p-value for the M effect is 2*.122=.244is 2*.122=.244

Statistical test for replicated Statistical test for replicated experiments--Exampleexperiments--Example

43210-1-2-3-4-5

0.4

0.3

0.2

0.1

0.0

1.19

0

.122

t distribution with 24 degrees of freedom


ExampleExample U-do-it: Compute p-values for the U-do-it: Compute p-values for the remaining effects. Which effects are remaining effects. Which effects are significant? Are these the same significant? Are these the same effects that the probability plot effects that the probability plot detected?detected?


F tests for individual effects are F tests for individual effects are equivalent to t-testsequivalent to t-tests

F tests allow several comparisons to F tests allow several comparisons to be tested simultaneouslybe tested simultaneously

The t-test can be used to help in The t-test can be used to help in computing the number of replications computing the number of replications needed in a factorial experimentneeded in a factorial experiment


Hypothesis testing can be extended Hypothesis testing can be extended to combine estimates of error from to combine estimates of error from both both pure errorpure error and and negligible negligible effectseffects– Negligible effects can be selected Negligible effects can be selected a a

prioripriori or from effects plots or from effects plots– Degrees of freedom for t-tests and F-Degrees of freedom for t-tests and F-

tests should be adjusted accordinglytests should be adjusted accordingly


Error estimated from negiglible terms Error estimated from negiglible terms (Lack of Fit) is similar to MSPE(Lack of Fit) is similar to MSPE

Source DF SS MS Source DF SS MS

Residual Error 28 300.38 10.73Residual Error 28 300.38 10.73

Lack of Fit 4 45.38 11.34Lack of Fit 4 45.38 11.34

Pure Error 24 255.00 10.63Pure Error 24 255.00 10.63

Documents

Statistical tests for replicated experiments Normal probability plots are a less formal diagnostic tool for detecting effects Normal probability plots