Study Design Considerations for Observational Comparative

Effectiveness Research

Prepared for:Agency for Healthcare Research and Quality (AHRQ)

www.ahrq.gov

This presentation will: Provide a rationale for study design choice

and describe key design features Define start of followup Define inclusion and exclusion criteria at

start of followup Define exposures of interest at start of

followup. Define outcome(s) of interest Define potential confounders

Outline of Material

Conventional designs Cohort Case-control Case-cohort

Self-controlled designs Case-crossover Case-time-controlled Self-controlled case series

Study Design Overview

Exposures or treatments are not assigned, a situation which leads to challenges ensuring internal validity, that is, the absence of bias. To ensure internal validity, treatment groups compared

must have the same underlying risk for outcome within subgroups definable by measured covariates (e.g., no unmeasured confounding). Confounding by indication leads to higher propensity

for/more intensive treatment in those with the most severe disease.

With confounding by frailty, frail patients (close to death) are less likely to be treated with preventive treatments.

Ensuring a study’s internal validity is a prerequisite for its external validity or generalizability.

Issues of Bias in ObservationalComparative Effectiveness Research

Cohorts are defined by their exposure at a certain point in time (i.e., baseline date) and are followed over time for the occurrence of the outcome.

Advantages: Has a clear timeline separating potential confounders from

the exposure and the exposure from the outcome Allows estimation of actual incidence (risk or rate) Can assess multiple outcomes Is easy to conceptualize

Limitations: Is inefficient for ad hoc studies when the incidence of the

outcome is low

Study Design: Cohort Study

Identifies all incident cases that develop an outcome and compares exposure history to controls

Samples controls at random from cohort members at risk for developing an outcome

Advantages: Oversampling cases increases computational efficiency

of ad hoc studies when compared with a cohort study Can assess multiple exposures

Limitations: Is difficult to conceptualize Has potential for recall bias in ad hoc studies

Study Design: Case-Control Study

Cohorts defined as in a cohort study Cohort members followed for incidence of

outcomes Additional information required for analysis

collected for a random sample of the cohort and all cases

Increased efficiency, when compared with a full-cohort design, if additional information needs to be collected

Decreased efficiency, when compared with a nested case-control design, unless studying multiple outcomes or estimating risk

Study Design: Case-Cohort Study

Prior exposure history of cases used as the control Removes confounding effect of measured and unmeasured

characteristics that are stable over time (e.g., genetics) Appropriate for studying acute effects of transient

exposures Advantages:

Self-controlled Ability to assess short-term reversible effects Ability to inform about the time window for these

effects Limitations:

Assumes constant prevalence of treatments over time Does not allow estimation of treatment effect in a

population

Study Design: Case-Crossover Study

Adjusts for calendar time trends in the prevalence of treatments, which can bias the case-crossover design

Divides the case-crossover odds ratio by the equivalent odds ratio estimated in controls

Advantages: Not dependent on assumption of no temporal

changes in the prevalence of treatment Limitations:

Need for controls adds complexity Control for time trend can introduce confounding

Study Design: Case-Time-Controlled Study

Estimates the immediate effect of treatment in those treated at least once

Dependent on cases that have changes in treatment during a defined observation period

Advantages: Controls for factors that are stable over time Cohort design has the potential to increase efficiency Well suited for rare adverse events in vaccine safety

studies Limitations:

Limited applicability in many comparative effectiveness research studies

Study Design: Self-Controlled Case-Series

Study setting Consideration of the study population and data

source(s) Inclusion and exclusion criteria

Should be clearly defined Include details about the study time period

Choice of comparators Reduces potential for confounding by comparing

treatment of interest with a different treatment for the same indication or an indication with the same potential for confounding

Study Design Features

New-User Design The conventional prevalent user design is prone to

confounding and selection bias as a result of changes in treatment effects over time. Including only new users reduces bias and confounding

associated with inclusion of prevalent users. There must be a clear starting point for followup under

similar conditions of medicalization. Immortal Time Bias

Occurs as a result of defining the exposure during the followup time rather than before followup

New-user design and use of comparator treatments reduce potential for this bias

Other Study Design Considerations

Knowledge of study design options is essential to increase internal and external validity of observational comparative effectiveness research.

Biases introduced by suboptimal study design cannot usually be removed by statistical analysis.

Cohort design is preferred when data have already been collected; the validity of a nested case-control study is equivalent, given proper control selection and timing of exposures and covariates.

It is important to define the start of followup, inclusion and exclusion criteria, outcome of interest, and potential confounders at the outset.

Conclusions

Summary Checklist (1 of 3)

Guidance Key ConsiderationsProvide rationale for study design choice and describe key design features

• Cohort study proposals should clearly define the cohort entry date (baseline date), employ a new-user design (or provide a rationale for including prevalent users), and plans for reporting losses to followup.

• Case-control study proposals should clearly describe the control sampling method, employ a new-user design (or provide a rationale for assessing confounders at the index date), and assess potential for recall bias (if applicable).

• Case-cohort study proposals should include how the sampling scheme will be accounted for during analysis.

• Case-crossover study proposals should discuss the potential for confounding by time-varying factors and clearly state how the resulting effect estimate can be interpreted.

• Case-time-controlled study proposals should clearly weigh the pros and cons of accounting for calendar time trends in the prevalence of exposure.

Summary Checklist (2 of 3)

Guidance Key Considerations

Define start of followup (baseline)

• The time point for start of followup should be clearly defined, meaningful, and ideally anchored to the time of a medical intervention (e.g., initiation of drug use).

• If alternative approaches are proposed, the rationale should be provided and implications discussed.

Define inclusion and exclusion criteria at start of followup

• Exclusion and inclusion criteria should be defined at the start of followup (baseline) and solely based on information available at this point in time (i.e., ignoring potentially known events after baseline).

• The definition should include the time window for assessment (usually the same for all cohort members).

Summary Checklist (3 of 3)

Guidance Key Considerations

Define exposure (treatments) of interest at start of followup

• Use an active comparator (indicated as alternative treatment at same stage of disease progression) when possible.

Define outcome(s) of interest

• Provide information on measures of accuracy if possible.

Define potential confounders

• Potential confounders known to be associated with treatment and outcome should be prespecified when possible.

• Confounders should be assessed before exposure or treatment initiation to ensure they are not affected by the exposure.

• Approaches to empirical identification of confounders should be described if planned.