Gatekeeping Testing Strategies in Clinical Trials Alex Dmitrienko, Ph.D. Eli Lilly and Company FDA/Industry Statistics Workshop September 2004

Gatekeeping Testing Strategies in Clinical Trials

Alex Dmitrienko, Ph.D.Eli Lilly and Company

FDA/Industry Statistics WorkshopSeptember 2004

Slide 2

Outline

Gatekeeping strategies• account for the hierarchical structure of multiple analyses

(comparisons) and preserve the overall false positive rate

Clinical trial examples• registration trials with multiple primary and secondary

endpoints (trials in patients with depression and acute lung injury)

• dose-finding trials (trial in patients with hypertension)

Slide 3

Primary versus secondary findings

Findings with respect to secondary endpoints provide much useful information • useful to prescribing physicians and patients

FDA guidance “Clinical studies section of labeling for prescription drugs and biologics”• “The CLINICAL STUDIES section should present those

endpoints that are essential to establishing the effectiveness of the drug (or that show the limitations of effectiveness) and those that provide additional useful and valid information about the activities of the drug”

Slide 4

Primary versus secondary findings

Dilemma• regulatory agencies and pharmaceutical companies have

long debated what secondary findings should be included in the product label

• regulatory agencies are concerned that pharmaceutical companies tend to present favorable data and ignore unfavorable data

Gatekeeper strategies offer one potential solution to the dilemma

Slide 5

Types of gatekeeping strategies

Two types of gatekeeping strategies• sequential strategies have been used for 10 years• parallel strategies were introduced by Dmitrienko, Offen

and Westfall (2003)

Introduce general gatekeeping framework• based on the closed testing principle (Marcus et al, 1976)• focus on strategies derived using Bonferroni’s test• easily extended to more powerful tests that account for

the correlation among the endpoints (Dunnett’s test, resampling tests)

Slide 6

Example 1: One primary endpoint

Depression trial• Experimental drug is compared to placebo• Single primary endpoint

– 17-item Hamilton depression rating scale (HAMD 17 score)

• Trial is declared successful if the drug is superior to placebo

• Two important secondary endpoints– response rate based on the HAMD 17 score

– remission rate based on the HAMD 17 score

• Can the secondary findings be included in the product label?

Slide 7

Sequential gatekeeping strategy

Step 1: Perform the primary analysis

Step 2: Perform the secondary analyses with an adjustment for multiplicity if the primary analysis yielded a significant result

A1: HAMD17A2: Response rate

A3: Remission rate

Family 2Family 1 (gatekeeper)

Slide 8

Sequential gatekeeping strategy

Endpoint Raw p Adjusted p Primary: HAMD 17 0.046 0.046 Secondary: Response rate 0.048 0.048 Secondary: Remission rate 0.021 0.042

Primary analysis: No adjustment for multiplicitySecondary analyses: Stepwise Holm’s test

All primary and secondary findings are significant at 5% level

Slide 9

Example 2: Multiple primary endpoints

Clinical trial in patients with acute lung injury• Experimental drug is compared to placebo• Two primary endpoints

– number of days patients are off mechanical ventilation (vent-free days)

– 28-day all-cause mortality rate

• Trial is declared successful if the drug is superior to placebo with respect to either endpoint

• Two important secondary endpoints– number of days patients are out of ICU (ICU-free days)

– overall quality of life at the end of the study

• Can the secondary findings be included in the product label?

Slide 10

Parallel gatekeeping strategy

Step 1: Perform the primary analyses with an adjustment for multiplicity

Step 2: Perform the secondary analyses with an adjustment for multiplicity if at least one primary analysis yielded a significant result

A1: Vent-free days

A2: Mortality

A3: ICU-free days

A4: Quality of life

Family 1 (gatekeeper)

Family 2

Slide 11




A1: Vent-free days

A2: Mortality

A3: ICU-free days

A4: Quality of life


Family 2

Slide 12




A1: Vent-free days

A2: Mortality

A3: ICU-free days

A4: Quality of life


Family 2

Slide 13

Statistical details

Sequential families of null hypotheses• Null hypotheses H11, H12, H21, H22 grouped into two

families F1={H11, H12} and F2 ={H21, H22}

Parallel gatekeeping testing procedure1. Overall Type I error rate is no greater than (e.g., 0.05)

2. The null hypotheses in F2 can be tested if at least one hypothesis in F1 has been rejected [parallel testing]

3. Adjusted p-values for the hypotheses in F1 do not depend on the p-values associated with the hypotheses in F2 [primary analyses cannot depend on secondary analyses]

Slide 14

Closed testing

Closed family of hypotheses• Consider all 15 intersections of the null hypotheses

– {H11 or H12 or H21 or H22}, {H11 or H12 or H21}, etc

• Specify a test for each intersection hypothesis that controls the Type I error rate, e.g., Bonferroni test

• Establish implication relationships– Intersection hypothesis {H11 or H12} implies H11 and H12, etc

• Tests for original hypotheses– Reject a null hypothesis if all intersection hypotheses implying it have

been rejected

Slide 15

Decision matrixIntersection hypothesis Rejection rule H11 or H12 or H21 or H22 Z11>z/2 or Z12>z/2 H11 or H12 or H21 Z11>z/2 or Z12>z/2 H11 or H12 or H22 Z11>z/2 or Z12>z/2 H11 or H12 Z11>z/2 or Z12>z/2 H11 or H21 or H22 Z11>z/2 or Z21>z/4 or Z22>z/4 H11 or H21 Z11>z/2 or Z21>z/2 H11 or H22 Z11>z/2 or Z22>z/2 H11 Z11>z/2 H12 or H21 or H22 Z12>z/2 or Z21>z/4 or Z22>z/4 H12 or H21 Z12>z/2 or Z21>z/2 H12 or H22 Z12>z/2 or Z22>z/2 H12 Z12>z/2 H21 or H22 Z21>z/2 or Z22>z/2 H21 Z21>z H22 Z22>z

Slide 16


Two scenarios• significant improvement in the mean number of ventilator-

free days and 28-day all-cause mortality• significant improvement in 28-day all-cause mortality but

not in mean number of ventilator-free day

Weighted analyses• primary endpoints are unequally weighted to reflect their

relative importance (likelihood of success)– weight=0.9 for number of ventilator-free days

– weight=0.1 for 28-day all-cause mortality

Slide 17


Scenario 1• significant improvement in the mean number of ventilator-

free days and 28-day all-cause mortality

Endpoint Raw p Adjusted p Primary: Vent-free days 0.024 0.027 Primary: Mortality 0.003 0.030 Secondary: ICU-free days 0.026 0.029 Secondary: Quality of life 0.002 0.027

Primary analyses: Bonferroni’s testSecondary analyses: Stepwise Holm’s test

All primary and secondary findings are significant at 5% level

Slide 18


Scenario 2• significant improvement in 28-day all-cause mortality but

not in mean number of ventilator-free day

Endpoint Raw p Adjusted p Primary: Vent-free days 0.084 0.093 Primary: Mortality 0.003 0.030 Secondary: ICU-free days 0.026 0.093 Secondary: Quality of life 0.002 0.040

Primary analyses: Bonferroni’s testSecondary analyses: Stepwise Holm’s test

The primary mortality analysis and secondary quality of life analysis are significant at 5% level

Slide 19

Example 3: Dose-finding study

Clinical trial in patients with hypertension• Four doses of an experimental drug are compared to

placebo– doses are labeled as D1, D2, D3 and D4

• Primary endpoint– reduction in diastolic blood pressure

• Objectives of the study– find the doses with a significant reduction in diastolic blood pressure

compared to placebo

– study the shape of the dose-response curve

Slide 20


Step 1: Compare doses D3 and D4 to placebo

Step 2: Compare doses D1 and D3 to placebo if at least one comparison at Step 1 is significant

Step 3: Perform various pairwise dose comparisons if at least one comparison at Step 2 is significant

A1: D4 vs. P

A2: D3 vs. P

A3: D2 vs. P

A4: D1 vs. P

Pairwise comparisons



Family 3

Slide 21





A1: D4 vs. P

A2: D3 vs. P

A3: D2 vs. P

A4: D1 vs. P




Family 3

Slide 22





A1: D4 vs. P

A2: D3 vs. P

A3: D2 vs. P

A4: D1 vs. P




Family 3

Slide 23

Example 3: Dose-finding studyM

ean

redu

ctio

n in

DB

P (m

mH

g)

0

5

10

Dose0 1 2 3 4

Slide 24

Parallel gatekeeping strategyComparison Raw p Adjusted p

Gatekeeping Holm Dunnett

procedure procedure procedure

D4 vs. P 0.0008 0.0016 0.0055 0.0030

D3 vs. P 0.0135 0.0269 0.0673 0.0459

D2 vs. P 0.0197 0.0394 0.0787 0.0656

D1 vs. P 0.7237 1.0000 1.0000 0.9899

D4 vs. D1 0.0003 0.0394 0.0021

D4 vs. D2 0.2779 1.0000 0.8338

D3 vs. D1 0.0054 0.0394 0.0324

D3 vs. D2 0.8473 1.0000 1.0000

Doses D2, D3 and D4 are significantly different from placebo at 5% level

Slide 25

Summary

Gatekeeping strategies can be successfully used in• pivotal trials with multiple primary and secondary

endpoints • dose-finding studies

Registration trials• a priori designation of gatekeeping strategy allows

additional data useful to physician and patient to be presented in the product label

Dose-finding studies• efficient tests of dose-response relationship

Slide 26

Extensions

More powerful gatekeeping tests• based on more powerful tests, e.g., Simes test• based on tests accounting for the correlation among the

endpoints (exact parametric tests such as Dunnett’s test and approximate resampling-based Westfall-Young tests)

Software implementation• SAS programs for gatekeeping tests can be found in

Dmitrienko, Molenberghs, Chuang-Stein, Offen. (2004). Analysis of Clinical Trials: A Practical Guide. SAS Publishing, Cary, NC.

Slide 27

References

Papers• Dmitrienko, Offen, Westfall. (2003). Gatekeeping

strategies for clinical trials that do not require all primary effects to be significant. Statistics in Medicine. 22, 2387-2400.

• Marcus R, Peritz E, Gabriel KR. (1976). On closed testing procedure with special reference to ordered analysis of variance. Biometrika. 63, 655-660.

• Westfall, Krishen. (2001). Optimally weighted, fixed sequence, and gatekeeping multiple testing procedures. Journal of Statistical Planning and Inference. 99, 25-40.

Documents

Gatekeeping Testing Strategies in Clinical Trials Alex Dmitrienko, Ph.D. Eli Lilly and Company FDA/Industry Statistics Workshop September 2004