Experimental Design, Statistical Analysis

CSCI 4800/6800University of GeorgiaMarch 7, 2002Eileen Kraemer

Research Design

Elements: Observations/Measures Treatments/Programs Groups Assignment to Group Time

Observations/Measure

Notation: ‘O’ Examples:

Body weight Time to complete Number of correct response

Multiple measures: O1, O2, …

Treatments or Programs

Notation: ‘X’ Use of medication Use of visualization Use of audio feedback Etc.

Sometimes see X+, X-

Groups

Each group is assigned a line in the design notation

Assignment to Group

R = randomN = non-equivalent groupsC = assignment by cutoffs

Time

Moves from left to right in diagram

Types of experiments

True experiment – random assignment to groupsQuasi experiment – no random assignment, but has a control group or multiple measuresNon-experiment – no random assignment, no control, no multiple measures

Design Notation ExampleR O1 X O1,2

R O1 O1,2

Pretest-posttest treatment

comparison group

randomized experiment

Design Notation Example

N O X O

N O O

Pretest-posttest

Non-Equivalent Groups

Quasi-experiment

Design Notation ExampleX O

Posttest Only

Non-experiment

Goals of design ..

Goal:to be able to show causalityFirst step: internal validity: If x, then y AND If not X, then not Y

Two-group Designs

Two-group, posttest only, randomized experiment

R X O

R O

Compare by testing for differences between means of groups, using t-test or one-way Analysis of Variance(ANOVA)

Note: 2 groups, post-only measure, two distributions each with mean and variance, statistical (non-chance) difference between groups

To analyze …

What do we mean by a difference?

Possible Outcomes:

Measuring Differences …

Three ways to estimate effect

Independent t-testOne-way Analysis of Variance (ANOVA)Regression Analysis (most general)

equivalent

Computing the t-value

Computing the variance

Regression Analysis

Solve overdetermined system of equations for β0 and β1, while minimizing sum of e-terms

Regression Analysis

ANOVA

Compares differences within group to differences between groupsFor 2 populations, 1 treatment, same as t-testStatistic used is F value, same as square of t-value from t-test

Other Experimental Designs

Signal enhancers Factorial designs

Noise reducers Covariance designs Blocking designs

Factorial Designs

Factorial Design

Factor – major independent variable Setting, time_on_task

Level – subdivision of a factor Setting= in_class, pull-out Time_on_task = 1 hour, 4 hours

Factorial Design

Design notation as shown2x2 factorial design (2 levels of one factor X 2 levels of second factor)

Outcomes of Factorial Design Experiments

Null caseMain effectInteraction Effect

The Null Case

The Null Case

Main Effect - Time

Main Effect - Setting

Main Effect - Both

Interaction effects

Interaction Effects

Statistical Methods for Factorial Design

Regression AnalysisANOVA

ANOVA

Analysis of variance – tests hypotheses about differences between two or more meansCould do pairwise comparison using t-tests, but can lead to true hypothesis being rejected (Type I error) (higher probability than with ANOVA)

Between-subjects design

Example: Effect of intensity of background

noise on reading comprehension Group 1: 30 minutes reading, no

background noise Group 2: 30 minutes reading,

moderate level of noise Group 3: 30 minutes reading, loud

background noise

Experimental Design

One factor (noise), three levels(a=3)Null hypothesis: 1 = 2 = 3

Noise None Moderate High

R O O O

Notation

If all sample sizes same, use n, and total N = a * nElse N = n1 + n2 + n3

Assumptions

Normal distributions

Homogeneity of variance Variance is equal in each of the

populations

Random, independent samplingStill works well when assumptions not quite true(“robust” to violations)

ANOVA

Compares two estimates of variance MSE – Mean Square Error, variances

within samples MSB – Mean Square Between, variance

of the sample means

If null hypothesis is true, then MSE approx = MSB, since

both are estimates of same quantity Is false, the MSB sufficiently > MSE

MSE

MSB

Use sample means to calculate sampling distribution of the mean,

= 1

MSB

Sampling distribution of the mean * nIn example, MSB = (n)(sampling dist) = (4) (1) = 4

Is it significant?

Depends on ratio of MSB to MSEF = MSB/MSEProbability value computed based on F value, F value has sampling distribution based on degrees of freedom numerator (a-1) and degrees of freedom denominator (N-a)Lookup up F-value in table, find p valueFor one degree of freedom, F == t^2

Factorial Between-Subjects ANOVA, Two factors

Three significance tests Main factor 1 Main factor 2 interaction

Example Experiment

Two factors (dosage, task)3 levels of dosage (0, 100, 200 mg)2 levels of task (simple, complex)2x3 factorial design, 8 subjects/group

Summary tableSOURCE df Sum of Squares Mean Square F pTask 1 47125.3333 47125.3333 384.174 0.000 Dosage 2 42.6667 21.3333 0.174 0.841 TD 2 1418.6667 709.3333 5.783 0.006 ERROR 42 5152.0000 122.6667 TOTAL 47 53738.6667

Sources of variation: Task Dosage Interaction Error

Results

Sum of squares (as before)Mean Squares = (sum of squares) / degrees of freedomF ratios = mean square effect / mean square errorP value : Given F value and degrees of freedom, look up p value

Results - example

Mean time to complete task was higher for complex task than for simpleEffect of dosage not significantInteraction exists between dosage and task: increase in dosage decreases performance on complex while increasing performance on simple

Results