Will It Work?Efficacy Studies and Clinical Research
Glenn Pransky, M.D., M.Occ.H.
Director, Liberty Mutual Center for Disability Research
Associate Professor, UMass Medical School
How Do Doctors Decide?
• It ought to work - inductive reasoning
• Others vouch for it - abdication
• Demonstrated effect - deduction
Gastric Freezing
• Cooling to -5° C decreased secretions
• President of ACS tried -10° C; published case series of 20 pts: symptoms, X-ray healing
• 2500 gastric freezing machines sold 1962-66
• 15,000 treatments in US by 1969
Randomized Trial (Ruffin et al, NEJM, 1969; 281, p.16)
• Double - blinded intervention - treating MD and patient didn’t know
• Results:
– Recurrence in 30 (44%) of placebo (warm)
– Recurrence in 35 (51%) of treated patients
Efficacy
• For a given medical problem, efficacy is the probability that treatment significant improvement in outcome, under ideal conditions
• Effectiveness = efficacy in usual conditions of practice
Efficacy Studies
• Phase I - human dosage / toxicity
• Phase II - uncontrolled trial
• Phase III - randomized, controlled trial
RCT Strengths
• Show maximum effect vs. placebo
• Strongest design to bias from several potential sources
• Prove causation
• Compare treatments
RCT Weaknesses
• Infrequent side-effects missed
• Long latency effects often attenuated
• Community usage unknown
• problems with placebo definition
Home vs. Hospital Care for Suspect MI (Hill, Lancet, 1978, l:837)
• RCT, 6 week follow-up after acute ER visit
• Sickest patients excluded
• Results:
– HOME: 20% of 79 died
– HOSPITAL: 18% of 71 died
• Need 261 / grp. - detect 25% RR difference
• Need 45 / grp. - detect 50% RR difference
RCT Sample Size
• Type I error: false optimism; usually = 5% (p-value)
• Type II error: false pessimism; usually = 20%
• Power = prob. of finding A if A is true
• Required parameters:expected control outcome, expected intervention effect, rate of outcome / length follow-up
Preventive Intervention, Uncommon Adverse Outcome
• New treatment for HTN in pregnancy
• Goal: risk from 2% 1%
• Type 1 error 5%, type 2 error 20%
• 2511 individuals needed in each group!
Selection - Exclusions
• Old, young, demented, minorities, pregnant
• Liver or kidney disease - drug metabolism
• Noncompliant patients (run-in period)
Selection - Inclusions
• Volunteers - sicker, compliant
• Referral (selected) population
Treatment problems in efficacy studies
• Atypical = non-generalizable
• Randomized allocation with complex significant factor patterns– Propensity scoring as alternative
• Ethics of no rx. vs. comparison rx.
• Placebo effect
• Multifaceted intervention - investigator attributes to one element
Control
• Blinding - ideally pt., treaters, evaluators
– When is this not feasible?
• Surgical treatments - randomization, but patients & treaters know
Control
• Contamination type II error– sympathy, community care
• Cointervention type I error (unblinding)– ‘more is better,’ support of tertiary care
center vs usual care
Follow-up
• Adequate for stability of 1/ 2outcomes
• Multiple assessments
• Loss - sicker, toxicity, high study burden
• Assume the worst?
Surgery vs. Medical Rx for Bilateral Carotid Stenosis (Fields & Maslenikov, JAMA, 1970, 211: p1993)
• 79 surg, 79 med @ F/U 1 year later
• Surgery: 27% risk stroke/death (p=0.02)
• Of 16 LTFU, 15 were allocated surgery - all had early death or stroke
• Intention-to-treat analysis: risk 16%, p = 0.09
Outcomes
• Clinical / physiologic
• Function
• Employment
• Satisfaction, quality of life
• Side-effects
• Value of surrogate endpoints?
Analytic Methods
• Variable length follow-up Kaplan-Meier survival curve (for terminal event)
• Assumes constant probability
Problems
• Multiple comparisons
• Large trial: statistical but not practical significance
Typical Results
1 = follow-up too short (Type 2 error)
2 = follow-up too long; few subjects or overwhelming additional influences plateau usually = low numbers
0
50
100
t1 t2 t3 t4 t5
t
% a
live 1
2
Adherence Chola, Mortality in CDP (NEJM 303: 1038, 1980)
Risk reduction= .26-.16/.26= 38%
Convincing?
Post-MI secondary prevention trial with clofibrate.
CDP Study (2)
• Same results in placebo group
• Conclusion: compliance survival effect!
Are All Relevant Results Reported?
Measure Placebo Clofibrate
Avg. change serum chol. +1% -9%
Nonfatal MI/1000 7.2 5.8
Fatal MI/1000 1.7 1.6
Total deaths/1000 5.2 6.2
(Oliver et al WHO trial on primary prevention IHD using databrate Lancet 1980; 2, p. 379)
Generalizability
• Adequate description?
• Typical cohort, setting?
• Typical treatment; acceptable, cost, available
• Sideffects - effort to obtain information
Functional Restoration: Pitfalls in Evaluating Efficacy (Gatchel, Mayer, Hazard et al, Spine, 1991, 17:988)
• Staff Training
• Program duplication, same selection criteria & patients
• Consistent evaluation methods, close follow-up
• Report important details (drop-out, total costs)
• Understand key differences - jurisdiction, etc.
Placebo Effects, Other Issues in Pain Treatment (Turner, Deyo, et al., JAMA, 1994, 271:1609)
• Regression to mean apparent benefit regardless of treatment– Patients enroll @ worst point in cyclical course– Also may reflect meas. error, random variation
• Nonspecific Rx effect– Attention, concern; expectations of healing;
“facilitated” reporting
Placebo Effects
• “Change in illness attributable to symbolic import of treatment.”
• Up to 70% of responses to Rx’s initially thought to be efficacious eventually defined as placebo effects
• IMA procedure: 56% significant improvement, 42% NTG use – (Cobb, NEJM, 1959, 260, 1115-8: skin incision
placebo)
• Diskectomy - negative surgical exploration: 37% no sciatica, 42% no LBP
– (Spangfort, Acta Orth Scand, 1972, 142:1)
Factors Influencing Placebo Response
• Patient attitude to provider and treatment
• More compliance
• Provider attitude, unbliding
Placebo Effects in Pain Treatment
• Mimic expected dose-response relationship (pain medications)
• Mimic expected side-effects (drowsiness, nausea)
• Perception: large or injected = strong;
yellow = stimulant
• Suggestion worse condition (Nocebo effect)
Questions
1. Randomization?
2. Blinding?
3. Adequate data collection / reporting?
4. Statistical, clinical significance?
5. All participants included
6. Generalizable
7. Feasible?