Sequential Sequential Experimentation: Experimentation: A A

Different Approach to Different Approach to

Prevention ResearchPrevention Research Annual Meeting of the Society for Prevention Research

Washington, DC

June, 2003

Linda M. CollinsThe Methodology Center and

Department of Human Development & Family Studies

Penn State

Susan A. MurphyDepartment of Statistics and Institute for Social Research

University of Michigan

Vijay NairDepartments of Statistics and Industrial & Operations Engineering

University of Michigan

OverviewOverview

• Why we are proposing a change in how we conduct prevention research

• Introduction to sequential experimentation

• Advantages we see to sequential experimentation

• Considerations and open questions

Two broad scientific issues facing Two broad scientific issues facing prevention scienceprevention science

How can we improve our understanding How can we improve our understanding of the mechanisms underlying of the mechanisms underlying

prevention?prevention?

• In early days of prevention, the field struggled to establish THAT prevention worked

• Now, the time is right to begin understanding HOW, WHY and FOR WHOM prevention works

• This will allow us to fine-tune, even optimize, interventions

How can we hasten the building of a How can we hasten the building of a coherent body of knowledge about coherent body of knowledge about

prevention?prevention?

• The field has moved forward in important ways – but slowly

• Solid, replicable results are mixed with unreplicable and contradictory findings

• We need to speed up the accumulation of consistent, replicable findings

We argue that a new approach is needed in order to address these issues.

But what is the difficulty with the current approach?

Two prevention program Two prevention program evaluation research agendasevaluation research agendas

• Answering the bottom-line question of whether a program is effective

• Informing theory and design of prevention programs

Two prevention program Two prevention program evaluation research agendas, evaluation research agendas,

one approachone approach

• We try to address both agendas with large, expensive prevention trials involving– lots of subjects

– power for detecting treatment effects

– lots of variables– post hoc analyses with the hope of informing theory and

figuring out how the program worked (or why it didn’t work)

Two prevention research Two prevention research program evaluation research program evaluation research

agendas, one approachagendas, one approach

• This approach tries to serve both agendas simultaneously, but is it the BEST way to serve either one?

How can we best inform How can we best inform theory?theory?

• Post-hoc or secondary analyses can be very misleading – e.g. post-hoc “dosage received” analyses

• Best way to inform theory: series of randomized experiments

How can we best answer How can we best answer questions about program questions about program

effectiveness?effectiveness?

• Even sound post-hoc analyses cannot help improve the prevention trial that generated the data

• Best way to maximize program effectiveness: obtain data to inform the prevention trial BEFORE it is conducted

We propose a different approach: sequential experimentation

OverviewOverview

• Why we are proposing a change in how we conduct prevention research

• Introduction to sequential experimentation

• Advantages we see to sequential experimentation

• Considerations and open questions

A different approach: A different approach: Sequential experimentationSequential experimentation

• Emerged from agriculture and engineering

• Philosophy: – scientific research is an iterative process of

deduction and induction– constant interplay among theory, experimentation,

data analysis

A different approach: Sequential A different approach: Sequential experimentationexperimentation

• In engineering

– theory suggests many possible treatment combinations that would require a prototype

– not practical to build ALL prototypes – each one is resource-intensive

A different approach: Sequential A different approach: Sequential experimentationexperimentation

• In prevention

– Theory suggests many candidate components and combinations of components that could make up a preventive intervention

– Not practical to bring EVERY possible combination of components to a prevention trial – each trial is resource-intensive

The objective of sequential The objective of sequential experimentationexperimentation

• Sort through the space of all possible treatment combinations and dosages in an efficient way, to select programs and dosages most likely to be successful

Sequential experimentation: A Sequential experimentation: A sequence of three phasessequence of three phases

• Screening of candidate program components

• Refining of program components

• Confirming the effectiveness of a refined program

Screening phaseScreening phase

• Purpose: to sort through candidate components to decide which are effective and should be investigated further

• Starting point: candidate intervention components suggested by theory– e.g. normative education, resistance training, parental component

• Examine each via a series of randomized controlled experiments

Screening Screening phasephase

• Components identified as effective in the screening phase are tentatively selected for inclusion in the intervention.

• Next, go to refining phase.

Refining phaseRefining phase

• Purpose: identify interactions, investigate blocking variables and putative moderators, refine dosage– e.g.:

examine whether normative education and parental component interact

examine effects of gender determine optimal number of resistance training lessons

• Starting point: components identified as effective in the screening phase

Refining phaseRefining phase

• Series of randomized experiments on components that passed the screening phase

• Depending on results, may go back to screening, or on to confirming phase

Confirming phaseConfirming phase

• Purpose: construct and test preventive intervention

• Starting point: active program components, appropriate dosages, and important moderators identified during screening and refining phases

Confirming phaseConfirming phase

• Develop a refined intervention, consisting of only active, effective components, delivered in appropriate amounts

• A full randomized prevention trial testing the refined intervention

A few key points about sequential A few key points about sequential experimentationexperimentation

• Theory plays a critically important role, particularly in the screening and refining phases– Selection of candidate components– Informing the designs used

A few key points about sequential A few key points about sequential experimentationexperimentation

• Extensive use made of designs such as balanced fractional factorials and response surfaces

• These designs use theory to inform choices about where to concentrate resources

• Much more efficient than full factorial designs– Number of cells and sample size requirements can be greatly

reduced, WHILE MAINTAINING STATISTICAL POWER

A few key points about sequential A few key points about sequential experimentationexperimentation

• The screening and refining phases are NOT pilot studies

• They are randomized, controlled studies

A few key points about sequential A few key points about sequential experimentationexperimentation

• Type II errors (overlooking active prevention component) may be worse here than Type I errors (mistakenly selecting inactive component)

• Type I error rate greater than usual p<.05 may be used

OverviewOverview

• Why we are proposing a change in how we conduct prevention research

• Introduction to sequential experimentation

• Advantages we see to sequential experimentation

• Considerations and open questions

Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation

approachapproach

•Screening, refining phases address “how, why, and for whom” questions

Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation

approachapproach

•Results of screening and refining phases will be replicable and will lead to a coherent body of knowledge

Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation

approachapproach

• Programs that go to prevention trials have been refined; the idea is to produce larger effect sizes

increased statistical powerincreased public health benefits

Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation

approachapproach

• It may allow for more creative risk-taking– it is possible to try a brand-new approach

in the screening phase without the investment of an entire prevention trial

Advantages we see to the Advantages we see to the sequential experimentation sequential experimentation

approachapproach

• Less likely to run into unanticipated problems

• Thus less likely to be presented with the familiar dilemma:– Midway through the prevention trial we realize

the program needs revision, but we don’t want to change the program and thereby make the evaluation difficult

OverviewOverview

• Why we are proposing a change in how we conduct prevention research

• Introduction to sequential experimentation

• Advantages we see to sequential experimentation

• Considerations and open questions

Considerations and open Considerations and open questionsquestions

• In most cases, the screening and refining phases can occur in a five-year funding cycle

• The confirming phase can occur in a second five-year funding cycle

• The support of NIH program staff will be needed

Considerations and open Considerations and open questionsquestions

• Hierarchical data– we are working on this

• Steady stream of research subjects needed– series of experiments may take place over

several years– possibility of cohort effects

Considerations and open Considerations and open questionsquestions

•You can think of many others!