2/3/04 Sacks 1

CLINICAL CONCLUSIONS

CLINICAL CONCLUSIONS

2/3/04 Sacks 2

Granting the Granting the statisticalstatistical

significancesignificance of a gain in Aof a gain in Azz

of 0.02of 0.02 (95% CI: 0.01, 0.04), (95% CI: 0.01, 0.04), what what is the is the clinical clinical

significancesignificance? ?

Granting the Granting the statisticalstatistical

significancesignificance of a gain in Aof a gain in Azz

of 0.02of 0.02 (95% CI: 0.01, 0.04), (95% CI: 0.01, 0.04), what what is the is the clinical clinical

significancesignificance? ?

2/3/04 Sacks 3

Clinical UtilityClinical Utility... CAD intended to reduce ... CAD intended to reduce missed nodules; i.e., missed nodules; i.e., intended to increaseintended to increase

user’s user’s SENSITIVITYSENSITIVITY ... ...

Clinical UtilityClinical Utility... CAD intended to reduce ... CAD intended to reduce missed nodules; i.e., missed nodules; i.e., intended to increaseintended to increase

user’s user’s SENSITIVITYSENSITIVITY ... ...

2/3/04 Sacks 4

Gain of 0.02 in AGain of 0.02 in AZ Z

understatesunderstatesrelative gain in relative gain in sensitivity. sensitivity. Why? Why?

Gain of 0.02 in AGain of 0.02 in AZ Z

understatesunderstatesrelative gain in relative gain in sensitivity. sensitivity. Why? Why?

2/3/04 Sacks 5

When CAD is used When CAD is used according to instruction to according to instruction to retain all judgments of retain all judgments of actionability even if actionability even if unmarked by CAD...unmarked by CAD...

When CAD is used When CAD is used according to instruction to according to instruction to retain all judgments of retain all judgments of actionability even if actionability even if unmarked by CAD...unmarked by CAD...

2/3/04 Sacks 6

... user always ... user always maintains or increases maintains or increases sensitivitysensitivity, and always , and always maintains or increases maintains or increases FPF, as well.FPF, as well.

... user always ... user always maintains or increases maintains or increases sensitivitysensitivity, and always , and always maintains or increases maintains or increases FPF, as well.FPF, as well.

2/3/04 Sacks 7

                                                                                                                       

                                 

                                                                                                                       

                                 

FPRFPR00 1100

11

TP

RT

PR

UnaidedUnaided

AidedAided

ΔSe =ΔFNRΔSe =ΔFNR

ΔFPRΔFPRFNRUFNRU

SeUSeU

Chance line (guessing)Chance line (guessing)

ROCROC

2/3/04 Sacks 8

                                                                                                                       

                                 

FPR0 10

1

TP

R

Unaided

Aided

Chance line (guessing)

Loss of Se only possible if instruction not followed

Realistic

ROC

2/3/04 Sacks 9

So any statistically So any statistically significant improvement significant improvement in Ain Azz means an even means an even

greater relative gain in Se, greater relative gain in Se, and one achieved without and one achieved without falling to a lower ROC falling to a lower ROC curve.curve.

So any statistically So any statistically significant improvement significant improvement in Ain Azz means an even means an even

greater relative gain in Se, greater relative gain in Se, and one achieved without and one achieved without falling to a lower ROC falling to a lower ROC curve.curve.

2/3/04 Sacks 10

The real question for The real question for judging the safety judging the safety and effectiveness of and effectiveness of a CAD is...a CAD is...

The real question for The real question for judging the safety judging the safety and effectiveness of and effectiveness of a CAD is...a CAD is...

2/3/04 Sacks 11

Can we infer from Can we infer from improved average improved average user performance (Auser performance (Azz) )

in a clinical in a clinical studystudy ... ...

Can we infer from Can we infer from improved average improved average user performance (Auser performance (Azz) )

in a clinical in a clinical studystudy ... ...

2/3/04 Sacks 12

... that the average ... that the average user will improve user will improve performance (Aperformance (Azz) with ) with

CAD in clinical CAD in clinical practice practice ......

... that the average ... that the average user will improve user will improve performance (Aperformance (Azz) with ) with

CAD in clinical CAD in clinical practice practice ......

2/3/04 Sacks 13

... i.e., improve over ... i.e., improve over her/his performance her/his performance (A(Azz) in ) in currentcurrent clinical clinical

practice?practice?

... i.e., improve over ... i.e., improve over her/his performance her/his performance (A(Azz) in ) in currentcurrent clinical clinical

practice?practice?

2/3/04 Sacks 14

To put it another way, is To put it another way, is the unaided reading in a the unaided reading in a clinical study a good clinical study a good surrogate for current surrogate for current (CAD-less) clinical (CAD-less) clinical practice?practice?

To put it another way, is To put it another way, is the unaided reading in a the unaided reading in a clinical study a good clinical study a good surrogate for current surrogate for current (CAD-less) clinical (CAD-less) clinical practice?practice?

2/3/04 Sacks 15

Az

Current clinical practice

Unaided reading

Clinical study

Aided reading

CAD-less reading

2/3/04 Sacks 16

For example, in actual For example, in actual clinical practice clinical practice with CADwith CAD, , the the unaidedunaided A Azz could be could be

lowered by failure to read lowered by failure to read first as one would first as one would normally read (i.e., with normally read (i.e., with adequate vigilance).adequate vigilance).

For example, in actual For example, in actual clinical practice clinical practice with CADwith CAD, , the the unaidedunaided A Azz could be could be

lowered by failure to read lowered by failure to read first as one would first as one would normally read (i.e., with normally read (i.e., with adequate vigilance).adequate vigilance).

2/3/04 Sacks 17

If this were to happen, If this were to happen, then the then the aidedaided A Azz

could also be lower could also be lower than current (CAD-than current (CAD-less) practice.less) practice.

If this were to happen, If this were to happen, then the then the aidedaided A Azz

could also be lower could also be lower than current (CAD-than current (CAD-less) practice.less) practice.

2/3/04 Sacks 18

Az

Current clinical practice

Unaided reading

Clinical study

Aided reading

CAD-less reading

2/3/04 Sacks 19

What would be the What would be the implications of such implications of such lowering of vigilance lowering of vigilance for judging the safety for judging the safety and effectiveness of and effectiveness of the CAD?the CAD?

What would be the What would be the implications of such implications of such lowering of vigilance lowering of vigilance for judging the safety for judging the safety and effectiveness of and effectiveness of the CAD?the CAD?

2/3/04 Sacks 20

Can labeling help Can labeling help prevent this?prevent this?Can labeling help Can labeling help prevent this?prevent this?

2/3/04 Sacks 21

LABELING ISSUELABELING ISSUE

Two rules, if followed by CAD Two rules, if followed by CAD user in future clinical practice, user in future clinical practice, will help prevent missing will help prevent missing moremore nodules than former reading nodules than former reading without CAD:without CAD:

LABELING ISSUELABELING ISSUE

Two rules, if followed by CAD Two rules, if followed by CAD user in future clinical practice, user in future clinical practice, will help prevent missing will help prevent missing moremore nodules than former reading nodules than former reading without CAD:without CAD:

2/3/04 Sacks 22

• AlwaysAlways read unaided first, read unaided first, and and as carefully as if you as carefully as if you had no had no CAD.CAD.

(This would help keep Az of(This would help keep Az of aided reading higher than Az of aided reading higher than Az of current CAD-less reading.) current CAD-less reading.)

• AlwaysAlways read unaided first, read unaided first, and and as carefully as if you as carefully as if you had no had no CAD.CAD.

(This would help keep Az of(This would help keep Az of aided reading higher than Az of aided reading higher than Az of current CAD-less reading.) current CAD-less reading.)

2/3/04 Sacks 23

• Never Never back off from unaided back off from unaided judgment of actionability of a judgment of actionability of a nodule if CAD fails to mark it.nodule if CAD fails to mark it.

(This would prevent sensitivity (This would prevent sensitivity from falling below that of current from falling below that of current CAD-less sensitivity. I.e., it would CAD-less sensitivity. I.e., it would prevent missing more, rather thanprevent missing more, rather thanfewer, nodules.)fewer, nodules.)

• Never Never back off from unaided back off from unaided judgment of actionability of a judgment of actionability of a nodule if CAD fails to mark it.nodule if CAD fails to mark it.

(This would prevent sensitivity (This would prevent sensitivity from falling below that of current from falling below that of current CAD-less sensitivity. I.e., it would CAD-less sensitivity. I.e., it would prevent missing more, rather thanprevent missing more, rather thanfewer, nodules.)fewer, nodules.)