THE USE OFHISTORICAL CONTROLS

IN DEVICE STUDIES

Vic HasselbladDuke Clinical Research Institute

HISTORICAL CONTROLS:THE SETTING

• New trial will have a single experimental arm• The endpoint is dichotomous• Comparison will be to either

• summary rates from other studies• another single arm using patient level analyses

HISTORICAL CONTROLSNO PATIENT LEVEL DATA

• Each historical arm has to be treated as a sample• Results are usually calculated from a random effects model• The distribution for the “next” sample is estimated• This requires that the between study variance be estimated specifically

AN EXAMPLE FROM DISTAL PROTECTION DEVICES

• A new distal protection device proposed using data from existing distal protection devices as historical controls• The endpoint was major adverse cardiac events (MACE)• The results from these three arms appeared

to be very consistent• The estimation proved difficult (as we shall

see)

DISTAL PROTECTION DEVICES

FilterWire (FIRE Trial)

GuardWire (FIRE Trial)

GuardWire (SAFER Trial)

0.0 0.1 0.2 0.3 0.4

MACE Rate at 30 Days

DISTAL PROTECTION DEVICES

• The prior for the mean rate was a non-informative (Jeffries) prior

• The prior for the study-to-study variation (τ2) was assumed to be non-

informative (1/τ2)• The expected distribution of the log-odds

of the event rate was assumed to be normal

CONSTRUCTING AHIERARCHICAL BAYES

RANDOM EFFECTS MODEL

POSTERIOR FOR VARIANCE (τ2)

0 1 2 3 4 5

MACE Rate

POSTERIOR FOR MEAN RATE

0.000 0.050 0.100 0.150 0.200 0.250 0.300

MACE Rate

Hierarchical Bayes Standard Bayes

In order to use the results from a small number of arms, one has to assume that the variation between arms is quite small.

In other words, one has to add subjective information to the prior.

CONCLUSIONS FORHISTORICAL CONTROLS

NO PATIENT LEVEL DATA

HISTORICAL CONTROLSWITH PATIENT LEVEL DATA

• Propensity scores are used to correct for the fact that the two populations are not guaranteed to be similar• Patients are stratified by their propensity to get

a particular treatment• Patients within a given propensity score are compared and the results are pooled across propensity categories

AN EXAMPLE WITH STENTS

• The object was to compare two different stent methodologies, one of which was a historical one• The safety endpoint was MACE at 30 days• The comparison was based on non-inferiority• Propensity scores were used to make the comparison: two different models were used as a sensitivity analysis

FIRST PROPENSITY SCOREUsed vessel diameter, lesion length, and presence of diabetes as predictors.

SECOND PROPENSITY SCOREUsed vessel diameter, lesion length, presence of diabetes plus several others factors including smoking and EF as predictors.

AN EXAMPLE WITH STENTS

First propensity score

Second propensity score

Delta

Covariate Effect Confounder Effect None Moderate Strong Moderate -9.0 -12.7 -16.0 Strong -16.1 -22.3 -28.2

Percent Bias in the Estimated Treatment EffectBased on a Stratified Propensity Score

(from Lunceford and Davidian, 2004)

STRATIFIED PROPENSITY SCORES CAN HAVE DIFFICULTIES

Covariate Effect Confounder Effect None Moderate Strong Moderate 8.3 11.1 8.9 Strong 12.0 16.2 12.8

Ratio of Means Squared ErrorsStratified Propensity Score Versus

Doubly Robust Estimator(from Lunceford and Davidian, 2004)

STRATIFIED PROPENSITY SCORES CAN HAVE DIFFICULTIES

HISTORICAL CONTROLSWITH PATIENT LEVEL DATA

• Even with the use of propensity scores, the results of a historical control analysis may not be definitive• The use of stratified propensity scores is not always the solution• In certain situations doubly robust estimators are better as long as they have the correct model (for propensity or risk)• If the models are wrong, all bets are off

SUMMARY

• Historical control analyses are fraught with difficulties

• In many cases you don’t know if problems exist until after the data have been collected