Lecture 17 (Oct 28,2004) 1

Lecture 17: Prevention of bias in RCTs

• Statistical/analytic issues in RCTs– Measures of effect– Precision/hypothesis testing– Compliance/intention to treat– RCTs of effectiveness of screening

• Effects of study design (Schultz paper)• Strengths and weaknesses of RCTs

Lecture 17 (Oct 28,2004) 2

Analysis of RCTs

• Planning stage:– Pre-specified hypotheses– Primary and secondary outcomes– Measure of effect– Sample size calculation

Lecture 17 (Oct 28,2004) 3

Analysis of RCTs

• Analysis stage:– Check on success of randomization– Analyze adherence to interventions – Intention to treat - why?– Should the analyses be blinded?

Lecture 17 (Oct 28,2004) 4

MRFIT study(Multiple Risk Factor Intervention Trial)• Prevention of coronary heart disease (CHD)

– Followed Framingham and other observational studies• Multi-site RCT • High-risk men age 35-57 (Framingham algorithm)

– N = 12,866• Comparison groups:

– Special intervention (SI): • Reduction of serum cholesterol via smoking cessation,

hypertension treatment, dietary modification– Usual care (UC):

• Notification of physician of results of risk status

Lecture 17 (Oct 28,2004) 5

MRFIT study (cont)

• Primary outcome: Death from CHD– Method of analysis?

• Secondary outcomes: – Death from any cardiovascular disease

– Death from any cause

– Overall CHD incidence (fatal and non-fatal cases)

• Intermediate outcomes:– Risk factor levels

Lecture 17 (Oct 28,2004) 6

MRFIT study(Multiple Risk Factor Intervention Trial)

• Sample size estimation:– Expected 6-year CHD death rate = 29.0/1,000– Hypothesized rate in SI group = 21.3/1,000 (26.6% reduction)– P (type 1 error) = 0.05 (one-sided test)– Power = 0.88

• Basis for projection:– 10% reduction of serum cholesterol if >220 mg/dL (vs no change in UC)– Reduction in smoking rate:

• 25% for smokers of 40+ cigs/day (vs 5% UC)• 40% for smokers of 20-39 cigs/day (vs 10% UC)• 55% for smokers of <20 cigs/day (vs 15% UC)

• Sub-group hypotheses:– Formulated during trial, blind to interim mortality data– Example: SI would be especially effective in men with normal resting

ECGs

Lecture 17 (Oct 28,2004) 7

Lecture 17 (Oct 28,2004) 8

Lecture 17 (Oct 28,2004) 9

MRFIT: explanations?

• Intervention not effective• Intervention is effective, but less than expected

because:– Lower than expected mortality in UC group– Risk reduction in UC group

• Positive effect in some sub-groups offset by negative effect in others– In subgroup with hypertension and ECG abnormalities,

higher death rate in SI vs UC– Possibly unfavorable response to antihypertensive drug

therapy?

Lecture 17 (Oct 28,2004) 10

MRFIT - lessons

• Consider “contamination” and “compensatory” effects in study design

• Clear specification a priori of planned sub-group analyses (with sample size calculations)

• (Reference: JAMA 1982, 248: 1465-1477)

Lecture 17 (Oct 28,2004) 11

Measures of effect

• Types of data to be analyzed:– incidence rate of an adverse event (death, etc)

• It = incidence rate in treatment group

• Ic = incidence rate in control group

• Example (mammography and mortality):• It = 2/10,000/year

• Ic = 4/10,000/year

Lecture 17 (Oct 28,2004) 12

Risk difference and ratio

Risk difference = Ic - It/units

– usually easier to express as risk reduction

– 4 - 2/10,000/year = 1/10,000/year

Risk ratio (relative risk) = Ic = 4/2 = 2.0

It

Alternatively: = It = 2/4 = 0.50

Ic

Lecture 17 (Oct 28,2004) 13

Relative risk reduction

• Analogous to attributable risk percent

• Sometimes called percent effectiveness

= risk difference = Ic - It

risk in control group Ic

= 2/4 = 50%

• Can be computed from the risk ratio: 1 - 1

RR

= 1 -1/2

Lecture 17 (Oct 28,2004) 14

Example from GUSTO trial

• tissue plasminogen activator (TPA) vs streptokinase (SK) as thrombolytic strategy in treatment of AMI.

30-day mortality in TPA group = 6.3%

• 30-day mortality in SK group = 7.3%

Lecture 17 (Oct 28,2004) 15

Measures of effect

RATE/RISK RATIO

SK rate = 7.3 = 1.16

TPA rate 6.3

RELATIVE RISK REDUCTION

SK rate – TPA rate = 7.3 – 6.3 = 14%

SK rate 7.3

[also calculated as 1 – (1/rate ratio)]

Lecture 17 (Oct 28,2004) 16

Measures of effect (cont)

ABSOLUTE RISK REDUCTION (rate/risk difference; attributable risk)

SK rate – TPA rate = 7.3% – 6.3% = 1.0%

NUMBER NEEDED TO TREAT (NNT)(Reciprocal of risk difference)

1 = 1 = 100

SK rate – TPA rate .01

Lecture 17 (Oct 28,2004) 17

SELECTION OF EFFECT MEASURES

Ratio measures assess strength of effect - how effective is the treatment?

Difference measures take into account frequency of the outcome – can assess whether it is worthwhile (allocation of time and $$)

Both ratio and difference measures are needed

All these measures are estimates and are subject to sampling error – need confidence intervals to determine their precision

All the measures are limited by the study(ies) that generated them – they may vary by patient characteristics, adherence to treatment, duration of follow-up, etc)

Measures consider only beneficial and not adverse effects of treatment.

Lecture 17 (Oct 28,2004) 18

Aspirin in prevention of MI among male smokers

(data from Physicians’ Health Study)

5-year incidence of MI:

aspirin group = 1.2%

placebo group = 2.2%

Risk ratio = 1.8

Relative risk reduction = 45%

Absolute risk reduction = 1.0% in 5 years

NNT = 100 for 5 years (to prevent 1 MI)

Lecture 17 (Oct 28,2004) 19

Antihypertensive treatment in 75-year old women with BP of 170/80

(data from SHEP study)

• 5-year incidence of stroke:

treatment group = 5.2%

placebo group = 8.2%– Risk ratio = 1.6– Relative risk reduction = 37%– Absolute risk reduction = 3.0% in 5 years– NNT = 33 / 5 years (to prevent 1 stroke)

Lecture 17 (Oct 28,2004) 20

Measures of effect in RCTs: continuous outcomes

• Example: RCT of antidepressant vs placebo:

• Measures on depression scale at baseline and at follow-up

• Possible measures:– Difference in mean scores at follow-up – Difference in change scores from baseline to

follow-up

Lecture 17 (Oct 28,2004) 21

Adherence to interventions

• Possible outcomes:– Low adherence in one or both study groups

• E.g. St John’s wort vs sertaline

– Cross-over • E.g., RCTs of medical vs surgical treatment of CHD

• How should results be analyzed?– By intervention to which randomized (“intention-to-

treat”)

– By intervention actually received?

Lecture 17 (Oct 28,2004) 22

RCTs of screeningExample: evaluation of the effectiveness of

breast cancer screening (HIP study)

• 1st RCT of breast cancer screening – Study population: Members of HMO

– Intervention: Invitation to receive annual mammography and clinical exam (3 years)

• Possible outcomes:– survival rate (1 year, 5 year)

– case-fatality rate

– mortality rate

• Which would you use?

Lecture 17 (Oct 28,2004) 23

Lecture 17 (Oct 28,2004) 24

Bias in RCTs of screening

• Definition of time zero?– Date of first symptoms?

– Date of detection?

– Date of diagnosis?

• Bias if difference in “time zero”between study groups:

– screening/early detection intervention shifts time zero

– intervention appears to lengthen time to outcome without real change in prognosis

– “lead time” bias

– “length” bias

Lecture 17 (Oct 28,2004) 25

Other types of bias in RCTs

• Hawthorne effect:– Non-specific effect of being in a study– Prevention?

• Contamination bias:– Control group receives some component(s) of

intervention– Prevention?

• Confounding variables – Variables associated with intervention group and

outcome, not in causal chain– Prevention?

Lecture 17 (Oct 28,2004) 26

Internal vs external validity

• Internal validity – Lack of bias in study

• External validity– Generalizability– Representativeness of study sample