Upload
london-nunley
View
216
Download
0
Embed Size (px)
Citation preview
Agenda:
- Hippocampal Label Expansion project- Certification Platform: performance/criteria- Questionnaire- Congress presentations- Papers
XV PMT meeting – Jan 30, 2013
Label Expansion - Design
Variable Value range Levels N Subjects
x level
N Hippos
x level
MTA 0-4 5 27 54
Scanner Siemens, GE, Philips
3 ~ 45 90
Magnet field strength
1.5 T, 3T 2 66 + 132 +
Diagnosis CTRL, MCI, AD 3 ~ 45 ~ 90
Age 60-70, 70-80, 80+ 3 ~ 45 ~ 90
Sides Right, Left 2 135 135
Subjects = 135 Hippocampi = 270
4
Hippo Label Expansion
USD 95239AA (50%), Bioclinica, Brain Image Analysis, Ixico, Roche, Synarc, True Positive Medical Devices (Louis Collins, Jan 29) (50%)
Laval: OK managing labels for qualification of AlgorithmsFund from Can AA will better enable that, and completion of current platform/certification criteria for human tracers (Wide logistical analysis for certification criteria can be carried out by Abderazzak at Laval)
Deadline: 3 months from $ receipt
Tracers:
1) Masami (Jaccard=83 in the whole Certification phase) 60 images (overlap with his beta tester project for Jap VSRAD algorithm)2) Martina 19 images
Contacted: Rossana (Master; 28 img?), Dominik (Master; 28 img?)
Publication policy
AVAILIABILITY OF DELIVERABLES:Needs being changed?
NO for the qualification labels (they should not be delivered anyway)
Maybe for the labels from the validation. Anyway, SD notes that these labels are FEW, and from few subjects: developers will have all what they need from the Expansion project, no need to change policy on availability of labels
Publication policy
AVAILIABILITY OF DELIVERABLESThe Harmonized Protocol (i.e. the text description and illustrative material), the digital master tracer masks, and digital training sets (masks developed by the 5 best naïve tracers) will be available only to beta-test users in the period (about 1 year) between the end of the Delphi Panel and the publication of the full validation resultsOr ADFEQUATE QC (i.e. the analysis of the whole set of hippocampal tracings by all naïve tracers).To qualify as beta-testers, users will submit proposals to the SC; if accepted, beta-testers will be granted access to deliverables under conditions dictated by a written cooperation agreement that will be signed by the user and the Chair of the SC. Assuming mutual benefit, the cooperation agreement will include provisions related to scientific publications, donations, and feedback that beta users will provide on their exploitation of the Harmonized Protocol.
ADNI Publication policy
AVAILIABILITY OF LABELS ON THE “SODIUM” WEBSITE
This would soon be an issue for the Expansion
Expanded labels may be back-transformed to native space: users will only need the ADNI code
Platform: Training Phase
2 Images (Scheltens=0,3) Feedback
6 Images (Scheltens=0,1,2,3,4,4) Feedback
10 Images (2 x Scheltens=0,1,2,3,4) Feedback CERTIFICATION
VALIDATION
TRAINING
Training Performance & Validation Tracer I round II round III round QUALIFICATION Valid phase 1 Valid phase 2
By End of June?
BL
1 Kristian Complete Complete Complete Complete END June 15
2 Corinna Complete Complete Complete Complete 31 genn
3 Mariangela Complete Complete Complete Complete
4 Tim Swih. Complete Complete Complete Complete Feb 6 fase 2? (100 img)
5 Masami Complete Complete Complete Complete END ON
6 Michel Complete Complete Complete Complete End March fase 2? (100 img)
7 Travis Complete Complete Complete Complete Feb 6
8 Marileen Complete Complete Complete Complete 31 genn fase 2? (100 img)
9 Enrica Complete Complete Complete Complete 31 genn
10 Adam Complete Complete 1 Febbraio
11 Chadwich Complete Complete 15 feb
12 Margo Complete Complete 13 genn
13 Yawu Complete Complete Complete Complete Feb 6
14 Melanie Complete Complete 10 genn
15 Emma Complete Feb 11
16 Oliver Complete 31 jan
17 Gregory QUALIFIED QUALIFIED QUALIFIED Jan 31 fase 2? (100 img)
Ros Proposed max 30 sbj
Dominik Proposed max 28 sbj
Martina Max 19 sbj
Melanie, img: bq_f_15T, total Jaccard: 0.76She does not separate properly the indusium griseum, and sometimes includes fornix. These errors are repeated in more slices where the criterion applies. All the rest is ok or really minor.
Masami, img: bq_a_3T, total Jaccard: 0.82Minor mistake in applying the HP criteria, or small inconsistencies in potentially ambiguous boundaries
These are all of his divergence points. We can not really say that he is “not compliant” with the HP
Michel 0,85
Masami 0,83
Marileen 0,81
Tim 0,81
Kristian 0,80
Corinna 0,80
Yawu 0,80
Travis 0,79
Enrica 0,79
Mariangela 0,78
Jaccard indices in Qualification
Certification Criteria (to be defined)
Hypothesis so far (generated versus quality check) is that:
Jaccard < 70 denotes major problems in hippocampal segmentationJaccard < 80 denotes incomplete compliance with the HP criteria for normal subjects (may be admissible for atrophic subjects: Enrica, bq_r_15T, J=0.724, tracings ok)
Combination of values needed
(Wide statistical analysis needed. See slide 3, Abderazzak at Laval and Can AA funds for Laval)
Questionnaire
How long does it take to you to segment one hippo based on the HP?
Do you believe that your comprehension of the hippocampus and your way to segment has improved after the training based on HP?Which are the parts of the HP that were less clear to you?
Were there regions in the Hippocampus where you did not know how to segment due to lack of instruction for that specific point or case? (if yes, please attach figures illustrating them)
Questions regarding the platform as required by SD
We can use SD’s Survey Monkey’s account
- Biomarkers for Brain Disorders (Cambridge, February 3-5, 2013)Definition of harmonized protocol for hippocampal segmentation
- AD/PD (Florence, March 6-10, 2013)Definition of Harmonized Protocol for Hippocampal Segmentation
- AAN (San Diego, March 16-23, 2013) EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation
- IEEE Medical Measurement and Applications conference (Gatineau –Quebec-, May 4-5, 2013.Platform for Training and Qualification
- AIC-AAIC (Boston, July 13-18, 2013) Training of Naïve Tracers; Statistics for Certification Platform
-(SINDEM (Ita Soc for Neurology of Dementias) Perugia March 13-15 Definition of harmonized protocol for hippocampal segmentation
Congress Presentations
Papers describing the project
Survey of protocols (preliminary phase; Published, JAD 2011)Operationalization (preliminary phase; Accepted, Alzheimer’s & Dementia, MS n. ADJ-D-12-00094)Axes check short report (Brescia Team, ADJ-D-13-00030)
Delphi consensus (Brescia Team, in progress)Master tracers’ practice and reliability (Brescia Team, in progr)Development of certification platform (Duchesne and coll)
Validation data (Brescia Team – companion paper 1)Protocol definition (Brescia Team – companion paper 2)Validation vs pathology (TBD)
DO
NE
IN P
RO
GR
ES
S P
LA
NN
ED
VALIDATION VS CURRENT PROTOCOLSASSESSMENT OF SOURCES OF
VARIANCE TRAINING SET DEVELOPMENT
VALIDATION VS PATHOLOGY
GOLD STANDARD
Harmonized ProtocolADNI scans: 2 x 5 Scheltens’s
atrophy score x 2 sides x 2 magnet strengths (1.5-3T)
Total per rater: 40 hippos
Harmonized ProtocolADNI scans: 2 sides x 5 Scheltens’s atrophy scores
x 3 time points (bl-1y-2y) x 3 scanners (+ retracing @ bl)
x 2 magnet strengths (1.5-3T)Total per rater: 240 hippos
Assessment of variance due to rater and center
Local ProtocolADNI scans: 2 x 5 Scheltens’s atrophy scores
x 2 sides x 2 magnet strengths (1.5-3T)
Harmonized ProtocolADNI scans: 2 x 5 Scheltens’s atrophy score x
2 sides x 2 magnet strength (1.5-3T)Total per rater: 40 hippos Harmonized Protocol:
Pathological datasets: Mayo Clinic and NYU
Total: about 40 hippos
TrainingADNI scans: 10 at 1.5T x 2 sides x 7
SUs x 2 tracing roundsTotal per rater: 40 hippos
20 naïve tracers 5 master tracers 1 tracer
REFERENCE PROBABILISTIC MASKS
with 95% C.I.
QUALIFICATION QUALIFICATION
Best 5 naïve tracers
Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater
TRAINING SET
Assessment of agreement with volume on pathology or ex vivo
MRI and correlation with neuronal density