PHASES OF RESEARCH AND DEVELOPMENT FOR HEALTH PROMOTION PROGRAMS

UIC University of

Illinois at Chicago

Health Research and Policy Centers

“Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003

PHASES OF RESEARCH AND DEVELOPMENT FOR HEALTH

PROMOTION PROGRAMS

Brian R. Flay, D.Phil., FAAHBHealth Research and Policy

CentersUniversity of Illinois at Chicago

Presented at the Annual Conference of the American Academy of Health Behavior

St. Augustine, Florida, March 16-19, 2003

UIC University of

Illinois at Chicago



Phases of Research: An Historical View

• FDA (19??)• OTA (1978)• Medical Innovations (McKinley, 1981)• NHLBI (1983)• NCI (Greenwald & Cullen, 1984)• Smoking Prevention (Flay & Best,

1982; Flay, 1986)• Expedited FDA (for AIDS) (1990)• Alcohol Prevention (Holder, Boyd, et

al, 1995; Holder, Flay, et al, 1999)

UIC University of

Illinois at Chicago



Phases of Research in Intervention Development (Flay,

1986)1. Basic Research2. Hypothesis Development3. Component Development and Pilot Studies4. Prototype Studies of Complete Programs5. Efficacy Trials of Refined Programs

– Well controlled randomized trials

6. Treatment Effectiveness Trials– Generalizability of effects under standardized delivery

7. Implementation Effectiveness Trials– Effectiveness with real-world variations in implementation

8. Demonstration Studies– Implementation and evaluation in multiple whole systems

UIC University of

Illinois at Chicago



Limited Use of Phases• Few researchers or research groups have

followed the phases sequence systematically• Sussman comes closest at the earliest phases

– See Sussman et al., 1995 monograph– Used theory-driven program development– Conducted small-scale pilot studies of program components– Altered or added new activities as a result– Piloted full-scale intervention

• Botvin comes closest at the efficacy-effectiveness phases– Has conducted multiple efficacy trials– Has conducted multiple treatment effectiveness trials– Has conducted multiple implementation effectiveness trials

• To my knowledge, no one research group has done it all -- the substance abuse prevention field almost has.

UIC University of

Illinois at Chicago



Reasons for limited use of Phases

• Interventions are becoming more complex– Multi-component, involving multiple stake-holders

• Higher stages require large, rigorous studies– Often too costly for standard research grant funding

• Different research skills and interests are needed at the different phases

• Many interventions are developed outside the research environment– Developed by professionals in the field, communities or

legislators– Holder and colleagues have suggested modifications to

incorporate these types of interventions

UIC University of

Illinois at Chicago



Complex Interventions• Always thought of as curricula, or whole programs,

not separate components– Few field-based tests of efficacy of separate components to date– But curricula/programs based on basic and hypothesis-driven

research

• Programs have grown more complex over the years• Multiple outcomes are becoming the norm

– E.g., multiple behaviors + Character + Achievement

• Also multiple ecologies are involved– School-wide– Involvement of parents/families– Involvement of community

• Therefore, multiple mediators, both distal and proximal– Distal: Family patterns, school climate, community involvement– Proximal: Attitudes, normative beliefs, self-efficacy, intentions

UIC University of

Illinois at Chicago



School-based Prevention/Promotion Studies

are Large and Complex• Large randomized trials

– With multiple schools per condition

• Comparisons with “treatment as usual”• Measurement of implementation process and

program integrity• Assessment of effects on presumed mediators

– Helps test theories

• Multiple measures/sources of data– Surveys of students, parents, teachers, staff, community– Teacher and parent reports of behavior– School records for behavior and achievement

• Multiple, independent trials of promising programs– At both efficacy and effectiveness levels

• Cost-effectiveness analyses

UIC University of

Illinois at Chicago



Holder et al (1999) modifications

• For field-developed programs and policy interventions– Often skip efficacy trials and use “natural experiments”

and quasi-experimental effectiveness trials.

• To distinguish between research-driven and program-driven interventions– Program-driven programs are not researcher-initiated– They are program- or policy-driven

• To allow for natural experiments or other quasi-experiments– Time-series analyses with natural variations– Comparison of states that adopt a policy with those that

don’t

UIC University of

Illinois at Chicago



Phases of research for program-driven prevention interventions (Holder

et al. 1999)1. Foundational Research

– Epidemiology, etiology and behavior change models

2. Developmental (Preliminary Effectiveness) Studies– Documentation and preliminary evaluation of an existing intervention– Should address safety and cost issues

3. Efficacy Studies– Quasi-experimental test of an optimally implemented intervention– Acceptance (compliance/readines) is also maximized when possible– Experimental or quasi-experimental test of well implemented

variations of the intervention

4. Effectiveness Studies– Evaluation of the quality and levels of acceptance, implementation

and overall effectiveness under real-world conditions

5. Diffusion Studies– Acceptance Evaluation -- of levels of acceptance and effects of– Implementation Evaluation -- of levels of implementation and effects

of– Monitoring -- Continued monitoring of program maintenance and

diffusion

UIC University of

Illinois at Chicago



Examples from AlcoholSee Holder et al. (1999) for more examples and

references

• DUI Enforcement• Ross evaluated the effects of DUI enforcement with time-series

analysis of archival data. • Short-term results approach an estimate of maximum effects

(I.e., efficacy). Long-term results (decay) represent effectiveness under on-going real-world conditions.

• Minimum drinking age• Natural variation in time of legislation and implementation

allowed for evaluation of the effects of changes in the minimum drinking age on youth alcohol use, DUI, crashes and other outcomes.

• Studies of high versus low implementation/enforcement states.

• Pricing Policies• Early cross-sectional studies led to time-series econometric

analyses.• Studies in “early change” States where enforcement was high• Studies comparing high and low enforcement states• Other national-level studies

UIC University of

Illinois at Chicago



Much of Prevention is STUCK … We’re Spinning our Wheels

• At the Efficacy Trial phase– How can we get more programs into

effectiveness trials

• At the Effectiveness Trial phase– How can we get more proven programs

adopted

• At the “Model Program” phase– How can we ensure the ongoing effectiveness

of model programs

UIC University of

Illinois at Chicago



What does the future hold?

• Where are the Dissemination Trials?– Deliberate studies of planned variations in

implementation/delivery, training, etc.

• Where are the Demonstration Studies?– Large scale studies of effects in the real world– On-going assessments of sustained effectiveness

• Alternatives: Where do we go next?

• Conclusions

UIC University of

Illinois at Chicago


Glasgow et al., 20023

• Pointed out that Greenwald & Cullen (1985 and Flay (1986) assumed that the best the candidates for effectiveness trials are interventions that have proven effective in efficacy trials

• And argued that this assumption, or the way it was operationalized, often led to to the development of interventions with low probability of success in the real world– Because of the strict standardization of the intervention

(to maximize internal validity) – And the limited conditions under which efficacy trials

are conducted (limited external validity)


UIC University of

Illinois at Chicago


RE-AIM

• REACH• Efficacy or Effectiveness• Adoption• Implementation• Maintenance and cost

• Strange order!• How about:

– Efficacy, Effectiveness, Dissemination, Implementation, Reach, Maintenance of effects and delivery


UIC University of

Illinois at Chicago


Glasgow et al Recommendations

• Increased attention to moderating factors– I.e., for whom and conditions under which it works

• Realize that impact involves more than just efficacy– Impact = Reach x Efficacy x Implementation– Or Impact = Effectiveness x Dissemination x

Maintenance

• Include external validity criteria in author guidelines

• Increase funding for research focused on moderating variables, external validity, and robustness– All of these get to generalizability


UIC University of

Illinois at Chicago



Selected ReferencesFlay BR (1986) Efficacy and effectiveness trials (and other phases of research)

in the development of health promotion programs. Prev Med, 15, 451-474, 1986.

Flay BR, Best JA (1982) Overcoming design problems in evaluating health behavior problems. Evaluation and Health Prof., 5, 43-69.

Food and Drug Administration (1990) From test tube to patient: New drug development in the United States. DHHS Pub. No. (FDA) 90-3168.

Greenwald P, Cullen JW (1984) The scientific approach to cancer control. CA Cancer J. Clin. 34, 328-332.

Holder H, Boyd G, Howard J, Flay B, Voas R, Grossman M (1995) Alcohol-related prevention research policy: The need for a phases research model. J of Public Health Policy, 16(3), 324-346.

Holder H, Flay B, Howard J, Boyd G, Voas R, & Grossman M (1999) Phases of alcohol problem prevention research. Alcoholism: Clinical and Experimental Research, 23(1), 183-194.

McKinlay JB (1981) From “promising report” to “standard procedure”: Seven stages in the career of a medical innovation. Milbank Memorial Fund Quarterly/Health Sot. 59, 374-411.

National Heart, Lung, and Blood Institute (1983) Guidelines for Demonstration and Education Research Grants. NHLBI, Washington, D.C.

Office of Technology Assessment (1978) Assessing the Efficacy and Safety of Medical Technologies. U.S. Govt. Printing Office, Washington, D.C.

Documents

PHASES OF RESEARCH AND DEVELOPMENT FOR HEALTH PROMOTION PROGRAMS