Panel: Image Interpretation
Challenges and Approaches to Standardization
Lawrence Schwartz, MDDepartment of Radiology
Columbia Presbyterian Medical [email protected]
Imaging Committee Chair for CALGB Wright Center of Innovation
in Biomedical Imaging
Imaging Modalities Used in Clinical TrialsImage Interpretation Standardization
CT
o The need for standardization varies by imaging modality, technique and potentially therapeutic optionMRI
PET
therapeutic option
o The need and degree of standardization is clearly related to the magnitude of the therapeutic effect which is to be measured
The Need for Interpretation StandardizationWhat are sources of variability ?
o Target lesion selection
o Image acquisition protocols
o Measurement of target lesions
Interpretation of “clear unequivocal progression o Interpretation of “clear unequivocal progression of non-target disease”
o Identification of new lesions
o Primary tumor type
Overall responses for all possible combinationsTarget lesions
Nontargetlesions
Newlesions
Overallresponse
CR CR No CR
CR Incomplete response/SD No PR
Table of Response AssessmentRECIST
CR Incomplete response/SD No PR
PR Non-PD No PR
SD Non-PD No SD
PD Any Yes or No PD
Any PD Yes or No PD
Any Any Yes PD
Response assessment Response Rank
Patient
No.
Total No .
of lesions
No. of
groupingsCR PR SD PD 1 2 3
No. of response
categories
1 7 21 0 0 0 21 1 0 0 1
2 16 4368 0 0 3697 671 0.85 0.15 0 2
3 10 252 0 100 152 0 0.6 0.4 0 2
4 10 252 1 232 19 0 0.98 0.08 0.004 3
The Need for Interpretation Standardization Variability - Target Lesion Selection
4 10 252 1 232 19 0 0.98 0.08 0.004 3
5 12 792 0 0 31 761 0.96 0.04 0 2
6 15 3003 0 0 3003 0 1 0 0 1
Calculated tumor response assessments, response ranks, and response
categories for one patient, analyzing 10 lesions with RECIST criteria
Clin Cancer Res. 2003 Oct 1;9(12):4318-23
STANDARIZED AVERAGE
RESPONSE VARIANCE
36 36 36 33 24 22 19 18 16 10 9 6
0.0 0.2 0.4 0.6 0.8 1.0
The Need for Interpretation Standardization Target Lesion Selection
NUMBERS OF LESIONS
PER GROUP
STANDARIZED AVERAGE
RESPONSE VARIANCE
1 2 3 4 5 6 7 8 9 10 11 12
0.0 0.2 0.4 0.6 0.8 1.0
Clin Cancer Res. 2003 Oct 1;9(12):4318-23
The Need for Interpretation Standardization Image Acquisition - Contrast Administration
Larry Schwartz
Memorial Sloan Kettering Cancer Center
Pre-walking CT Post-walking CT
Sources of VariabilityModality Acquisition and Measurement of target lesions
1.25-mm 1.25-mm
Uni-dimension (mm):
Bi-dimension (mm2):
Volume (mm3):
Pre-walking Post-walking Variation
27.6 27.8 0.7%
552 597.7 7.9%
4957.1 4852.3 2.1% Zhao Radiology
Concordance
correlation coefficient
Mean
% relative difference
95% Limits of
agreement� 95% CI
Sources of VariabilityModality Acquisition and Measurement of target lesions
Uni-
dimensional1.00 (1.00, 1.00) -0.6% -7.3 %, 6.2 %
Bi-
dimensional1.00 (0.99, 1.00) 1.1% -17.6 %, 19.8 %
Volume 1.00 (1.00, 1.00) 0.7% -12.1%, 13.4 %
Zhao Radiology
Sources of VariabilityInterpretation of “clear unequivocal progression of non-target
disease”
o There is no clear definition or interpretation of “clear unequivocal progression of non-target disease” in RECISTdisease” in RECIST
o This may result in variable interpretations impacting TTP image analysis especially in diseases with more extensive non target component
Sources of VariabilityIdentification of new lesions
Frequency of pulmonary nodules detectionNo. of Nodules Observer A Observer B
1.25 mm 5 mm 1.25 mm 5 mm
2-5 mm 28 13 36 22
Impact on lung lesion detection for time to progression analysis
2-5 mm 28 13 36 22
6-10 mm 18 14 20 18
11-30 mm 9 9 9 9
Total 55 36 65 49
What is an “optimal” or “acceptable”Agreement among observers?
o Consideration of the Primary Tumor and type of metastatic disease
o Mesothelioma
o Ovariano Colorectal
Renalo Ovarian
o Pancreas
o Gastric
o Others – Prostate, lymphoma
A single, standard agreement/adjudication rate would not reflect the
variability in assessments across clinical trials
o Renal
o Breast
Acceptable Adjudication Rate?Number of Modalities Assessed
o Case Study 1
o Nonsmall Cell Lung Cancer
CT – Chest
o Case Study 2
o Ovarian Cancer
o CT – Chest / Abdomen / Pelviso CT – Chest
/Abdomen
Abdomen / Pelvis
o FDG-PET
o CA-125
o QOL assessment
o Paracentesis for ascites
A single, standard agreement/adjudication rate would not reflect the
variability in assessments across clinical trials
Acceptable Agreement Rate?Each Adjudicated Case may not be Equal
SD SD PR PR PR PD
0 1 2 3 4 5 6 7 8months
Reader 1PR PR
SD SD PR PD
monthsReader 2
PDPD PDPD
Case 1
0 1 2 3 4 5 6 7 8months
SD SD PR PR PR PD
0 1 2 3 4 5 6 7 8months
Reader 1PR PR
SD SD PR PD
0 1 2 3 4 5 6 7 8months
Reader 2PR PR PR PD
Case 2
CALGB
o Founded 1956
o The unit of membership is the
US Cooperative Groups
o Cooperative groups are consortia of institutions that o The unit of membership is the
institution; 28 main members, 14 CCOPs, 225 affiliates
o (Headquarters): University of Chicago; Statistical Center: Duke University
institutions that conduct research in cancer treatment, prevention, biology and health outcomes
Treatment (Intervention) Trials @ CALGB
o Phase I or limited access n = 3
o Phase II n = 22
o Phase III n = 18
o Breast
o Lymphoma
o GI
o Colorectal, esophagus, o Phase III n = 18
o Registration directed (prospective) n = 4
o Several retrospective registration directed trials
o Colorectal, esophagus, rectal
o GU
o Kidney, bladder, prostate
o Pathology
o Imaging
Setting Standards of Care
o FDA approvals based on cooperative group data:
-cisplatin for NSCLC
-paclitaxel for ovarian and NSCLC
-paclitaxel as adjuvant therapy for breast cancer
-tamoxifen for breast cancer prevention-tamoxifen for breast cancer prevention
-interferon for high risk melanoma
-5-azacytidine for MDS
-oxaliplatin for met. CRC
-bevacizumab in 2nd line therapy for mCRC
New CALGB Trials Utilizing Imaging
Protocol Study Chair Imaging Co-Chair
CALGB40502 Hope Rugo, M.D. Deanna L. Kroetz, Ph.D.
CALGB40503 Maura Dickler, M.D. Federico Innocenti, M.D.
CALGB50303Wyndham H. Wilson, M.D., Ph.D. Andrew D. Zelentz, M.D., Ph.D.
Heiko Schoder, M.D.
CALGB50701 Barbara Grant, M.D. Lale Kostakoglu, M.D.
CALGB80302 David H. Ilson, M.D., Ph.D. Nathan Hall, M.D., Ph.D.CALGB80302 David H. Ilson, M.D., Ph.D. Nathan Hall, M.D., Ph.D.
CALGB140503 Nasser Altorki, M.D. Ernest Scalzetti, M.D.
CALGB80802 Ghassan Abou-Alfa, M.D. Lawrence Schwartz, M.D.
SWOG0816 Oliver W. Press, M.D., Ph.D., Heiko Schoder, M.D.
CALGB30803 Sarita Dubey, M.D. Ernest Scalzetti, M.D
CALGB50604 David J. Straus, M.D. Lale Kostakoglu, M.D.
CALGB50801 Ann S. LaCasce, M.D. Lale Kostakoglu, M.D.
CALGB30901 Arkadiusz Z. Dudek M.D., Ph.D. Ernest Scalzetti, M.D
CALGB50602 Sonali M. Smith, M.D. Heiko Schoder, M.D.
CALGB50201 Thomas Shea, M.D. Lawrence Schwartz, M.D.
CALGB50203 David J. Straus, M.D. Malik Juweid, M.D.
CALGB50404 Barbara Grant, M.D. Malik Juweid, M.D.
St u dy N u m be r St u dy Na m e p ts a c c u ra l To t a l St u d i e sRe ce iv e dC A L G B 1 4 0 5 0 3 A P ha s e I I I Ra n do m iz e d Tr ia l o f Lo b e c to m y v e r s u s S u b l o ba r R e s e c t io n fo rS ma l l (< /= 2 c m ) P e r ip h e r a l N o n- s ma l l C e l l L u n g Ca n c e r 9 9 2 4 1C A L G B 8 0 3 0 2 A P ha s e I I Tr ia l o f P r eo p e r a t i v e I r i n o t e ca n, C i sp la t i n a n d Ra d ia t io n i nE so p ha g ea l Ca n c e r 4 5 1 3 7C A L G B 5 0 7 0 1 A P ha s e I I Tr ia l o f Ex t e n d e d I n d u c t io n Ep r a t uz u ma b ( A n t i- C D 2 2M o n o c l o na l A n t i bo d y ) ( C A L G B I N D # 1 0 1 2 4 1 ) P l u s R i t u x i ma b i nP r e v io u s l y U n t r ea t e d Fo l l i c u la r N o n- H o d g k i n ' s L y m p h o ma ( N H L ) 6 1 1 0 4C A L G B 5 0 6 0 2 A P ha s e I I S t u d y o f Ga l i x i ma b ( A n t i- C D 8 0 ) fo r Pa t i e n t s w i t hR e la p s e d / R e fr a c to r y H o d g k i n L y m p h o ma 1 4 2 5C A L G B 5 0 3 0 3 P ha s e I I I Ra n do m iz e d S t u d y o f R- C H O P v. Do s e- A d j u s t e d E P O C H- R w i t hM o l e c u la r P r o f i l i n g i n U n t r ea t e d D e N o vo D i f f u s e La r g e B- C e l lL y m p h o ma s 5 3 1 5 1C A L G B 5 0 2 0 3 P ha s e I I Tr ia l o f Do x o r u b i c i n, V i n b la s t i n e a n d G e m c i ta b i n e ( A V G )C h e m o t h e r a p y fo r N o n- B u l k y S ta g e I a n d I I H o d g k i n L y m p h o ma 1 0 5 4 0 9C A L G B 4 0 5 0 3 E n do c r i n e T h e r a p y i n Co m b i na t io n w i t h A n t i- V E G F T h e r a p y : ARa n do m iz e d, Do u b l e- B l i n d, P la c e bo - Co n t r o l l e d P ha s e I I I Tr ia l o fE n do c r i n e T h e r a p y A l o n e o r E n do c r i n e T h e r a p y P l u s B e va c iz u ma b( N S C 7 0 4 8 6 5 : I N D 7 9 2 1 ) fo r Wo m e n w i t h H o r m o n e R e c ep to rPo s i t v e A d va n c e d B r ea s t Ca n c e r 5 7 1 3 3
C A L G B 4 0 5 0 2 A Ra n do m iz e d P ha s e I I I Tr ia l o f W e e k l y Pa c l i ta x e l Co m p a r e d to W e e k l yNa n o p a r t i c l e A l b u m i n Bo u n d N A B- Pa c l i ta x e l o r I x a b ep e l o n eCo m b i n e d w i t h B e va c iz u ma b a s F i r s t- L i n e T h e r a p y fo r Lo ca l l yR e c u r r e n t o r M e ta s ta t i c B r ea s t Ca n c e r 1 1 2 2 8 5C A L G B 5 0 2 0 1 A P ha s e I I S t u d y to E va l u ta e t h e Sa f e t y a n d E f f i ca c y o f Z e va l i n ( I N D # B BI N D 1 1 0 2 3 ) T h e r a p u t i c R e g i m e n i n Pa t i e n t s w i t h Tr a n s fo r m e dC D 2 0+ B- c e l l N o n- H o d g k i n ' s L y m p h o ma 7 2 4S W O G 0 8 1 6 A P H A S E I I T R I A L O F R E S P O N S E- A D A P T E D T H E R A P Y O F S T A G E I I I- I VH O D G K I N L Y M P H O M A U S I N G E A R L Y I N T E R I M P D G- P E T I M A G I N G 3 0 4 2
ICL
VT
NH
MA
•10.
•30.
•105.
•14.
•26.
•21.
•39.
•40.
•53.
�ICL – Collaborative Sites [106] at USA
•34.
•12.
•46.
•66.
•70.
•76.
•103.
•31.
•37.
•45.
•47.
•69.
•77.
•79.
•33.
•36.
•48.
•52.
•57. •38.
•49.
•1.
•7.
•16.
•22.
•25.
•28.
•44.
•82.
•98.
•4.
•9.
•64.
WA
OR
ID
MT
CAWY
ND
SD
MN
MINY
MEVT
NH
MA
RI
WI•52.
•54.
•85.
•99.
•58.
•68.
•71.
•90.
•75.
•92. •89.
CALGB Imaging Core Lab Overview Procedures and Services
Jun Zhang, PhD; Nathan C. Hall, MD, PhD; Michael V. Knopp, MD, PhD
The Ohio State University, Columbus
MA
CT
RI
NJ
DE
MD
DC
•11.
•19.
•26.
•42.
•60.
•80.
•32.
•91.
•104.
•34.
•51.
•62.
•84.
•2.
•35.
•55.
•101.
•23.
•43.
•50.
•86.
•8.
•6.
•30.
•96.
•13.
•15.
•17.
•18.
•20.
•46.
•56.
•61.
•81.
•88.
•106
CA
NV
UT
AZNM
CO
NE
KS
OK
TXLA
AR
MO
IA
INILWV
KY
TN
MSAL
FL
GA
SC
NC
VA
PAOH
•41.
•24.
•59.
•63.
•65.
•72.
•74.
•87.
•27.
CTRI
DE
MD
•5.
•29.
DC
NJ
•67.
•102
•73
•83.
•95.
•78.
Imaging Core Service
Clinical Trials Quality Control
Infrastructure Administrative
ICR
•Imaging Core Facilities
•Vendor Imaging Systems
•Vendor Workstations
•Dedicated Workstations
•Director
•Project Leader
•Project Manager
•Dedicated Individuals
SOP Audit
ICR
•Data receipt confirmation
•Data quality check report
•DCIOM De-identification
•ICR database
•Site education/training/approval
•Overall communication
•Regular trial report
•Lab meetings
•Training sessions
•Site credentialing
•Compliance monitoring
•Protocol Amendment
•Site Technical Manual
•Trial E-mail
•Web/FTP transfer
•Data management
•Post-processing
•Central review
•WebEx
•De-identification
•Equipment validation
ICL
Communication
Data
Submission
CD
FTP
Imaging
Complete
Sites
Discussion
Y
N
PhoneEmail WebEx
ICL
Committee
Decision
Bland
Review
Review Panel
+Problem ?
Quality Control Workflow in Clinical
Cancer Trials
Data
Management
Quality
Check
Clinical QC
DICOM
QC
Protocol
Compliance
Receipt
Confirmatio
n
Imaging Core Laboratory
Endpoint
Analysis
Regular
Review &
Report
QC Report
Storage
Archiving
Database
Database
N
≤ 24 hours ≤ 72 hours
De-
Identification
Y
Consult
Committee
Database
Update
Notify Site of
Decision
Compliant ?
o The need for standardization varies by imaging modality and potentially therapeutic option
o The need and degree of standardization is clearly related to the magnitude of the therapeutic effect which is to be measured
Centralized Data with Remote Review
• Vendor Advanced Workstation based
• Extended Brilliance Workspace
• Multi-Modality Workplace
Centralized Data Review• Centralized Data Review
• Data in one system
• Multiple reviewers
• Easy and Real-Time Access – Internet
PET-response (≥ 35%
SUV): 5-FU and Oxaliplatin
with concurrent RT
FDG PET/CT after induction chemo can identify
patients who benefit from changing chemo resulting in
improved response rates and PFS
Imaging Adaptive Trials
T3/4 or N1
Esophageal
Cancer
Pre-tx PET/CT
then Induction
Chemotherapy
with concurrent RT
PET day 22-
28 to
evaluate
response
Referral for
surgical resection
6 weeks post-RT
PET- no response (< 35%
SUV): CPT-11, Taxotere with
concurrent RTIntenseIntense need for real-time standardization ofAcquisition and INTERPRETATION
Real-time Adaptive Trial Support -
1. New studies received? - Monitor trial Email and Workstation for the Review
2. New Pt registration? – Monitor trial email and remind sites of data submission
3. Data Receipt Confirmation within 24 hours upon data receipt
4. Quality Check Report notification within 48 hours for ‘baseline’ and ‘final’, 24 hours for ‘interim’
5. For ‘non-compliant’ studies, contact imaging committee for a final decision.5. For ‘non-compliant’ studies, contact imaging committee for a final decision.
6. DICOM image De-identification
7. Remote Review Scheduling with Central Readers
8. Prepare the review form for readers
9. Real-time Data Review with reader(s)
10. Request for review results from readers
11. Notification of central review results to sites and Central Office
ACADEMIC EXPERT PANEL
REVIEW
72 HOUR TURN AROUND FROM
ACQUISITION TO INTERPRETATION
Panel: Image InterpretationChallenges and Approaches to Standardization
o Interpretation by its nature is both quantitative as well qualitative
o Critical is standardization of acquisition, analysis and results reporting
o Expert interpretationo Expert interpretation
o Training, education, experience – imaging and therapeutic specific
o The need for standardization varies by imaging modality and potentially therapeutic option
o The need and degree of standardization is clearly related to the magnitude of the therapeutic effect which is to be measured