What is “sufficient” evidence to inform combination HIV prevention programs

Stefan Baral

Evidence Supporting Interventions Donabedian Approach

Process The Traditional Gold Standard for M&E Is system efficient?

Counting the actual products distributed, people trained, etc

Condoms, Peer Educators, Paralegals, etc. Structure

Structural Outcomes of the Intervention Health Systems, Health Policies, etc

Outcome Emerging Gold Standard… What is happening with outcome of interest?

Impact! Efficacy vs Effectiveness

2

What is Sufficient Evidence?

Evidence-based medicine is a global standard Double-Blinded (DB) RCT is gold standard

Evidence-based PH interventions should also be a global standard Often limited evidence, PH decision still needs to be

made Precautionary Principle for PH?

When there are threats of serious or irreversible damage, lack of full scientific certainty shall not be used as a reason for postponing cost-effective measures to prevent environmental degradation

To develop guidelines Need to characterize

Efficacious Effective Sustainable and Scalable programs

3

Agency for Healthcare Research and Quality (AHRQ)

US Preventive Services Task Force (USPSTF) Three-Step System

Strength of Recommendation Letter (A-D, I)

Level of Certainty Low, Medium, High

Suggestions for Practice

http://www.ahrq.gov/clinic/uspstfix.htm

Strength of Recommendation

Rating Strength of Recommendation Practice Recommendations

A • Recommends the Service• High Certainty that net benefit is substantial

Offer/Provide This Service

B • Recommends the Service• High certainty that net benefit is moderate• Moderate certainty that net benefit is substantial

Offer/Provide This Service

C • Recommends against Routine Provision of this service

• Special considerations for or against in each patient

• Moderate certainty that net benefit is low

Offer/Provide only if special considerations support in individual

D • Recommends against the service• Moderate or high certainty that service has no

net benefit or harms outweigh the benefits

Discourage the use of this service

I • Current Evidence is Insufficient to assess balance between benefits and harms

• Evidence is : lacking, poor quality, conflicting

Never be offered

Level of Certainty

Level of Certainty

Description

High Evidence from:• Methodologically Sound Studies in Primary Care Populations

(Generalizability)• Health outcomes evaluation (effectiveness)

Unlikely to be affected by future studies

Moderate Enough evidence to determine effect of service on health outcomes, but limited confidence in estimate

• Evidence constrained by• Number/size/quality of studies, Inconsistency, limited

generalizabilityRecommendation may change based on future results

Low Evidence is insufficient to assess health outcomesInsufficient because:

• Limited number/size of studies, flaws in study design, inconsistency, gaps in chain of evidence, not generalizable, inadequate info

More data will allow estimation of effects on health outcomes

CDC Prevention Research Synthesis (PRS) Project

3 Domains Study Design Study Implementation and Analysis Strength of Evidence

2 Tiers Tier 1 – Best Evidence Tier 2 – Good Evidence

Separate Criteria for Individual-level interventions (ILI) and group-level

interventions (GLI) Community-Level Interventions (CLI)

http://www.cdc.gov/hiv/topics/research/prs/index.htm

CDC

Prevention and Treatment of Opportunistic Infections Two Step System

Letter – Strength of Recommendation related to Practice Recommendation

Efficacy Data Clinical Benefit

Roman Numeral – Quality of Evidence

Source: Kaplan, et al. Guidelines for Prevention and Treatment of Opportunistic Infections in HIV-Infected Adults and Adolescents. MMWR. 2009

Strength of Recommendation

Rating Strength of Recommendation Use

A • Strong Efficacy Data • Substantial Clinical Benefit

Should always be offered

B • Moderate Efficacy/Substantial Clinical Benefit• High Efficacy/Limited Clinical Benefit

Generally be offered

C • Insufficient Efficacy Data• Good Efficacy Data/Efficacy Data does not

outweigh adverse effects/actual cost/opportunity cost

Optional

D • Lack of Efficacy Data/Moderate adverse outcome data

Generally not be offered

E • Good evidence for lack of efficacy or adverse outcome

Never be offered

Quality of Evidence

Rating Quality of Evidence

I Evidence from at least one high-quality DB RCT

II Evidence from at least one• Quasi-experimental clinical trial

• no randomization, no blinding, etc• Cohort/Case-control data

• Ideally multiple centers• Multiple Time-Series Studies• Dramatic Results from Uncontrolled Experiments

III Evidence from• Expert Opinion

• Clinical experience• Reports of Expert Committees/Authoritative Bodies• Descriptive Studies

UK National Institute for Clinical Excellence (NICE) (Formerly the Health Development Agency)

Oxford Center For Evidence-Based Medicine

Level of Evidence

Rating within Level

Details

1 ABC

SR of RCTs (homogeneity)Individual RCTAll or None (no outcome either before or after intervention—ie parachutes)

2 ABC

SR of Cohort Studies (with homogeneity)Individual CohortOutcomes Research/Ecological Work

3 AB

SR Of Case-Control Studies (with homogeneity)Individual Case-Control Studies

4 Case-Series (or poor quality cohort/case-control)

5 Expert Opinion

12

CEBM Grade of Recommendations

13

Grade Characteristics

A Consistent Level 1 Study

B Consistent Level of 2 or 3 Studies or Extrapolating from Level 1 Studies (ie off label use)

C Level 4 Studies or Extrapolations from Level 2/3

D Level 5 Evidence or troublingly inconsistent or inconclusive studies of any level

Grading of Recommendations Assessment, Development and Evaluation (GRADE)

Score-based system designed for clinical interventions Type of Evidence Quality Consistency Directedness Effect Size

Includes Values and Preferences PICOTS Questions

Population, Intervention/Exposure, Comparison/Control, Outcome, Timing, Setting (PC, Specialty, In-Patient)

GRADE Characteristics

Type of Evidence RCT or SR of RCT

+4 Observational Evidence

+2 Quality

Blinding, retention, subjective outcomes 0 to -3

Directness Generalizability

0 to -2 Effect Size

Measure of association >2 or >5 0 to +2

GRADE Score

Strength of Recommendation High

> or = 4 Medium

3 Low

2 Very Low

< or = 1 Values and Preferences

Highest Attainable Standard of Evidence System for HIV Interventions (HASTE)

http://www.ncbi.nlm.nih.gov/pmc/articles/PMC3461350, http://www.plosmedicine.org/article/info%3Adoi%2F10.1371%2Fjournal.pmed.1001469

Tension Between Internal and External Validity Challenges for Evidence Combination Prevention

Internal Validity Minimal study biases suggesting confidence in

ultimate conclusion of the study External Validity

Generalizability of ultimate findings to broader population

Traditional Question for Clinicians/Programmers Does it work? What is effect size?

Should I use it? Implementation Questions

How, when, why, and where does it work? What factors influence effectiveness?

Should I use it? How should I use it?

Tension in Research about Validity

Traditional Approach is to establish internal validity with certain study designs and then have studies focused on external validity Internal Validity

Phase 1 (Safety), Phase 2a/b (tolerability, TOC), Phase 3 (Efficacy)

External Validity Phase 4 (Post-Marketing)

Traditional Research Pathway

Effectiveness Research (and guideline development) generally happens prior to implementation research Are there more time-effective approaches to integrate

implementation research with effectiveness/efficacy research

Assess barriers/facilitators to intervention uptake acceptance/adoption/routinization

Diagnose quality gaps Fidelity

Characterize Sustainability Maintenance, Cost-Effectiveness