12/10/02Harry Bushar1 Computerized Thermal Imaging Breast Cancer System 2100 (CTI BCS2100) Radiological Devices Advisory Panel December 10, 2002 Statistical

12/10/02 Harry Bushar 1

Computerized Thermal Imaging Breast Cancer System 2100

(CTI BCS2100)Radiological Devices Advisory Panel

December 10, 2002Statistical PresentationHarry F. Bushar, PhD


Outline of Statistical Presentation

• Clinical Study Protocol• Objective• Design• Population• Demographics• Evaluation (Effectiveness + Safety)

• PMA Clinical Study• Effectiveness + Safety

• Amendment 4 Clinical Study• Effectiveness

• Amendment 5 Clinical Study• Effectiveness

• Amendment 7 Adjustment• Effectiveness

• Statistical Conclusions


Clinical Study Protocol

• Study Objective – “The objective of the study is to determine if

the CTI System, when used in conjunction with clinical examination and/or diagnostic mammography, increases the ability of physicians to differentiate benign from malignant, or suspicious, breast abnormalities.”



• Study Design – prospective– blinded-to-histology– multi-center– intended to compare

• Level of Suspicion (LOS) (0-5) of malignancy of suspicious breast lesions for clinical examination or diagnostic mammography before BCS to

• LOS + (BCS Index of Suspicion (IOS) (0.00 – 100.00) of malignancy of suspicious breast lesions)

– biopsy as the “gold standard” for pathology.



• Study Population– Original study population = 600 patients with

biopsy.– Actual study population = 2,407 patients with

biopsy.


Clinical Study Protocol• Study Demographics

– Gender• 2,364 Female• 15 Male• 28 Unknown

– Ethnicity• 53% Caucasian• 30% African American• 13% Latino• 2% Asian• 1% Other• 1% Unknown

– Age• 12% (< 40)• 55% (40 – 60)• 32% (> 60)• 1% Unknown



• Primary Effectiveness on Overall Population– Evaluation

• Area Under the ROC* Curve (AUC) to compare results of diagnostic mammography (LOS) without BCS (IOS) and diagnostic mammography with BCS (LOS + IOS).

• Sensitivity• Specificity

– “The CTI System will be considered effective if its performance in conjunction with diagnostic mammography and/or clinical examination is clinically better than mammography and/or clinical examination alone.”

*ROC = Receiver Operating Characteristic



• Secondary Effectiveness on Subpopulations– Mammographic Lesion Type

• Calcifications• Masses• Distortions

– Mammographic Lesion Size• < 0.5 cm• 0.5 – 1.0 cm• > 1.0 cm

– Mammographic Lesion Depth• As available



• Safety Evaluation – occurrence of adverse events.


PMA Clinical Study Population

• The sponsor acquired BCS images from 2,407 patients at 6 US clinical sites from 12/20/96 through 4/30/01.

• The sponsor actually analyzed only those patients with both– mammography, not those with just clinical

examination, and – biopsy within 60 days.


PMA Training Clinical Study

• 700 patients, consisting of the first 220 patients + 480 patients randomly selected from among the next 1,912 patients, were used to set the following BCS IOS cut-off: ≥ 20.59 implies a recommendation for biopsy of

a given lesion and

< 20.59 implies a recommendation for short-interval follow-up of a given lesion.


PMA Testing Clinical Study

• 1,432 patients, enrolled from 12/20/96 through 10/30/00, were initially available to test the effectiveness of the BCS both in the original PMA and in Amendment 4.

• 769 patients, with 187 malignant lesions + 688 benign lesions, were actually included in the effectiveness evaluation both in the original PMA and in Amendment 4.

• Note that each patient had 1 to 4 lesions.• Note also that the sponsor assumes that lesions

within patient are independent.


PMA Clinical Study Results

• Primary Effectiveness by ROC AUC– The sponsor found, after excluding calcifications

alone, a statistically significantly greater ROC AUC for (IOS + LOS1) than for mammography LOS1 scores (1-4) alone at significance level = 0.05.

– The sponsor found, after again excluding calcifications alone, but expanding mammography LOS1 scores beyond (1-4) to include 2 additional intermediate categories (3.50 & 3.75), no statistically significant difference between ROC AUC for (IOS + LOS2) & (LOS2 alone) at significance level = 0.05.


ROC Plot of IOS+LOS1 Versus LOS1

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

1 - Specificity

Sens

itivi

ty IOS+LOS1

LOS1


ROC Plot of IOS+LOS2 Versus LOS2

0

0.2

0.4

0.6

0.8

1

0 0.2 0.4 0.6 0.8 1

1-Specificity

Sen

sitiv

ity

IOS+LOS2LOS2


PMA Clinical Study Results

• Safety– The following 4 adverse events occurred out

of 2,407 subjects (12/20/96 through 4/30/01):• 2 “mild”, possibly-related, resolved adverse events

were both associated with patient discomfort during positioning.

• 1 “serious” and 1 “mild”, not-likely-related, resolved adverse events were hospitalization for treatment of a pre-existing metabolic disorder and dizziness when sitting up after thermal imaging, respectively.


Amendment 4 Clinical Results

• Overall (187 malignant + 688 benign)– Sensitivity = 97.1% with

95% CI* = (94.1%, 98.8%)– Specificity = 14.3% with

95% CI = (12.1%, 16.6%)*CI = Confidence Interval


Amendment 4 Clinical Results

• Calcifications* (105 malignant + 368 benign)• Sensitivity = 94.8% with 95% CI = (89.6%, 97.8%)• Specificity = 9.2% with 95% CI = ( 6.9%, 12.1%)

• Masses (90 malignant + 322 benign)• Sensitivity = 100% with 95% CI = (96.7%, 100%) • Specificity = 18.0% with 95% CI = (14.6%, 21.9%)

• Distortions (16 malignant + 15 benign)• Sensitivity = 100% with 95% CI = (82.9%, 100%)• Specificity = 16.7% with 95% CI = ( 3.9%, 40.2%)

*The sponsor interpreted these effectiveness clinical results by lesion type to specifically exclude calcifications alone.


Amendment 4 Interpretation

• The sponsor’s initial rejection of overall effectiveness, followed by the sponsor’s differential findings among the 3 lesion type sub-populations clearly indicates exploration: – which does require confirmation, – which must be based on new data.


PPMA* Amendment 5 Population

• Study population = 275 additional patients.– Gender

• 274 female• 1 male

– Ethnicity• 65% Caucasian• 31% African American• 3% Latino• 1% Asian or Unknown

– Age• 13% (< 40)• 60% (40 – 60)• 27% (> 60)

*PPMA = Post-PMA


PPMA Amendment 5 Study

• 275 additional patients, enrolled from 11/1/00 through 4/30/01 at 3 of the 6 original US clinical sites, were initially available for confirmation of the effectiveness of the BCS in Amendment 5.

• 173 patients, with 43 malignant + 159 benign lesions, were actually included in the Amendment 5 effectiveness evaluation.

• Note that each patient had 1 to 3 lesions. • Note also that the sponsor assumes that lesions

within patient are independent.


PPMA Amendment 5 Results

• Overall (43 malignant + 159 benign)– Sensitivity = 93.8% with

95% CI = (84.0%, 98.5%)– Specificity = 20.0% with

95% CI = (14.9%, 26.0%)


PPMA Amendment 5 Results

• Calcifications (24 malignant + 87 benign)• Sensitivity = 93.1% with 95% CI = (77.9%, 99.0%)• Specificity = 17.4% with 95% CI = (11.1%, 25.5%)

• Masses* (15 malignant + 63 benign)• Sensitivity = 93.3% with 95% CI = (72.1%, 99.7%)• Specificity = 25.4% with 95% CI = (16.6%, 36.0%)

• Distortions (8 malignant + 4 benign)• Sensitivity = 75.0% with 95% CI = (40.0%, 95.4%)• Specificity = 25.0% with 95% CI = ( 1.3%, 75.1%)

*The sponsor interpreted these effectiveness results by lesion type to specifically include only masses.


Amendment 7 Adjustment• The sponsor’s attempted Bonferroni adjustment, in response to FDA

deficiency 1.a), by – widening sensitivity/specificity CI estimates, which are – based on a simple direct combination of exploratory + confirmatory

clinical data (PMA/Amendment 4 + PPMA Amendment 5), to– test a theoretically possible hypothetical set of hypotheses for

• 7 (lesion types) or • 63 (lesion types, sizes, & depths),

– which hypotheses are not explicitly included in the protocol, • Is not statistically acceptable, because

– the sponsor simply estimates sensitivity/specificity CI’s for various sub-populations,

– without actually statistically testing any hypotheses.• Therefore, there are no multiple comparisons requiring adjustment.


Statistical Conclusions• Diagnostic mammography, not just clinical examination, is required

for use of BCS.• The sponsor’s primary effectiveness demonstration, using ROC

AUC, loses statistical significance, when mammography LOS (1-4) is expanded by just 2 additional intermediate categories (3.50 & 3.75), after excluding calcifications alone.

• The sponsor’s initial rejection of overall sensitivity, followed by rejection of calcification-alone sensitivity, indicates exploration:– which requires confirmation, – which requires new data.

• The sponsor’s attempted Bonferroni adjustment, by widening sensitivity/specificity CI estimates, is not statistically acceptable.

Documents

12/10/02Harry Bushar1 Computerized Thermal Imaging Breast Cancer System 2100 (CTI BCS2100) Radiological Devices Advisory Panel December 10, 2002 Statistical