Agenda:

- Hippocampal Label Expansion project- Certification Platform: performance/criteria- Questionnaire- Congress presentations- Papers

XV PMT meeting – Jan 30, 2013

Label Expansion - Design

Variable Value range Levels N Subjects

x level

N Hippos

x level

MTA 0-4 5 27 54

Scanner Siemens, GE, Philips

3 ~ 45 90

Magnet field strength

1.5 T, 3T 2 66 + 132 +

Diagnosis CTRL, MCI, AD 3 ~ 45 ~ 90

Age 60-70, 70-80, 80+ 3 ~ 45 ~ 90

Sides Right, Left 2 135 135

Subjects = 135 Hippocampi = 270

4

Hippo Label Expansion

USD 95239AA (50%), Bioclinica, Brain Image Analysis, Ixico, Roche, Synarc, True Positive Medical Devices (Louis Collins, Jan 29) (50%)

Laval: OK managing labels for qualification of AlgorithmsFund from Can AA will better enable that, and completion of current platform/certification criteria for human tracers (Wide logistical analysis for certification criteria can be carried out by Abderazzak at Laval)

Deadline: 3 months from $ receipt

Tracers:

1) Masami (Jaccard=83 in the whole Certification phase) 60 images (overlap with his beta tester project for Jap VSRAD algorithm)2) Martina 19 images

Contacted: Rossana (Master; 28 img?), Dominik (Master; 28 img?)

Publication policy

AVAILIABILITY OF DELIVERABLES:Needs being changed?

NO for the qualification labels (they should not be delivered anyway)

Maybe for the labels from the validation. Anyway, SD notes that these labels are FEW, and from few subjects: developers will have all what they need from the Expansion project, no need to change policy on availability of labels

Publication policy

AVAILIABILITY OF DELIVERABLESThe Harmonized Protocol (i.e. the text description and illustrative material), the digital master tracer masks, and digital training sets (masks developed by the 5 best naïve tracers) will be available only to beta-test users in the period (about 1 year) between the end of the Delphi Panel and the publication of the full validation resultsOr ADFEQUATE QC (i.e. the analysis of the whole set of hippocampal tracings by all naïve tracers).To qualify as beta-testers, users will submit proposals to the SC; if accepted, beta-testers will be granted access to deliverables under conditions dictated by a written cooperation agreement that will be signed by the user and the Chair of the SC. Assuming mutual benefit, the cooperation agreement will include provisions related to scientific publications, donations, and feedback that beta users will provide on their exploitation of the Harmonized Protocol.

ADNI Publication policy

AVAILIABILITY OF LABELS ON THE “SODIUM” WEBSITE

This would soon be an issue for the Expansion

Expanded labels may be back-transformed to native space: users will only need the ADNI code

Platform: Training Phase

2 Images (Scheltens=0,3) Feedback

6 Images (Scheltens=0,1,2,3,4,4) Feedback

10 Images (2 x Scheltens=0,1,2,3,4) Feedback CERTIFICATION

VALIDATION

TRAINING

Training Performance & Validation Tracer I round II round III round QUALIFICATION Valid phase 1 Valid phase 2

By End of June?

BL

1 Kristian Complete Complete Complete Complete END June 15

2 Corinna Complete Complete Complete Complete 31 genn

3 Mariangela Complete Complete Complete Complete

4 Tim Swih. Complete Complete Complete Complete Feb 6 fase 2? (100 img)

5 Masami Complete Complete Complete Complete END ON

6 Michel Complete Complete Complete Complete End March fase 2? (100 img)

7 Travis Complete Complete Complete Complete Feb 6

8 Marileen Complete Complete Complete Complete 31 genn fase 2? (100 img)

9 Enrica Complete Complete Complete Complete 31 genn

10 Adam Complete Complete 1 Febbraio

11 Chadwich Complete Complete 15 feb

12 Margo Complete Complete 13 genn

13 Yawu Complete Complete Complete Complete Feb 6

14 Melanie Complete Complete 10 genn

15 Emma Complete Feb 11

16 Oliver Complete 31 jan

17 Gregory QUALIFIED QUALIFIED QUALIFIED Jan 31 fase 2? (100 img)

Ros Proposed max 30 sbj

Dominik Proposed max 28 sbj

Martina Max 19 sbj

Emma, round 1 (1.5T), Jaccard: 0.69 left and 0.65 right

Yawu, round 1 (1.5T), tot Jaccard: 0.71does not distinguish amy from hippo; includes fornix

Melanie, img: bq_f_15T, total Jaccard: 0.76She does not separate properly the indusium griseum, and sometimes includes fornix. These errors are repeated in more slices where the criterion applies. All the rest is ok or really minor.

Masami, img: bq_a_3T, total Jaccard: 0.82Minor mistake in applying the HP criteria, or small inconsistencies in potentially ambiguous boundaries

These are all of his divergence points. We can not really say that he is “not compliant” with the HP

Michel, subject bq_F_3T (Qualific) Jaccard: left 0.873, right 0.870

Michel 0,85

Masami 0,83

Marileen 0,81

Tim 0,81

Kristian 0,80

Corinna 0,80

Yawu 0,80

Travis 0,79

Enrica 0,79

Mariangela 0,78

Jaccard indices in Qualification

Reliability in III round

Certification Criteria (to be defined)

Hypothesis so far (generated versus quality check) is that:

Jaccard < 70 denotes major problems in hippocampal segmentationJaccard < 80 denotes incomplete compliance with the HP criteria for normal subjects (may be admissible for atrophic subjects: Enrica, bq_r_15T, J=0.724, tracings ok)

Combination of values needed

(Wide statistical analysis needed. See slide 3, Abderazzak at Laval and Can AA funds for Laval)

Questionnaire

How long does it take to you to segment one hippo based on the HP?

Do you believe that your comprehension of the hippocampus and your way to segment has improved after the training based on HP?Which are the parts of the HP that were less clear to you?

Were there regions in the Hippocampus where you did not know how to segment due to lack of instruction for that specific point or case? (if yes, please attach figures illustrating them)

Questions regarding the platform as required by SD

We can use SD’s Survey Monkey’s account

- Biomarkers for Brain Disorders (Cambridge, February 3-5, 2013)Definition of harmonized protocol for hippocampal segmentation

- AD/PD (Florence, March 6-10, 2013)Definition of Harmonized Protocol for Hippocampal Segmentation

- AAN (San Diego, March 16-23, 2013) EADC-ADNI Benchmark Images of Harmonized Hippocampal Segmentation

- IEEE Medical Measurement and Applications conference (Gatineau –Quebec-, May 4-5, 2013.Platform for Training and Qualification

- AIC-AAIC (Boston, July 13-18, 2013) Training of Naïve Tracers; Statistics for Certification Platform

-(SINDEM (Ita Soc for Neurology of Dementias) Perugia March 13-15 Definition of harmonized protocol for hippocampal segmentation

Congress Presentations

Papers describing the project

Survey of protocols (preliminary phase; Published, JAD 2011)Operationalization (preliminary phase; Accepted, Alzheimer’s & Dementia, MS n. ADJ-D-12-00094)Axes check short report (Brescia Team, ADJ-D-13-00030)

Delphi consensus (Brescia Team, in progress)Master tracers’ practice and reliability (Brescia Team, in progr)Development of certification platform (Duchesne and coll)

Validation data (Brescia Team – companion paper 1)Protocol definition (Brescia Team – companion paper 2)Validation vs pathology (TBD)

DO

NE

IN P

RO

GR

ES

S P

LA

NN

ED

VALIDATION VS CURRENT PROTOCOLSASSESSMENT OF SOURCES OF

VARIANCE TRAINING SET DEVELOPMENT

VALIDATION VS PATHOLOGY

GOLD STANDARD

Harmonized ProtocolADNI scans: 2 x 5 Scheltens’s

atrophy score x 2 sides x 2 magnet strengths (1.5-3T)

Total per rater: 40 hippos

Harmonized ProtocolADNI scans: 2 sides x 5 Scheltens’s atrophy scores

x 3 time points (bl-1y-2y) x 3 scanners (+ retracing @ bl)

x 2 magnet strengths (1.5-3T)Total per rater: 240 hippos

Assessment of variance due to rater and center

Local ProtocolADNI scans: 2 x 5 Scheltens’s atrophy scores

x 2 sides x 2 magnet strengths (1.5-3T)

Harmonized ProtocolADNI scans: 2 x 5 Scheltens’s atrophy score x

2 sides x 2 magnet strength (1.5-3T)Total per rater: 40 hippos Harmonized Protocol:

Pathological datasets: Mayo Clinic and NYU

Total: about 40 hippos

TrainingADNI scans: 10 at 1.5T x 2 sides x 7

SUs x 2 tracing roundsTotal per rater: 40 hippos

20 naïve tracers 5 master tracers 1 tracer

REFERENCE PROBABILISTIC MASKS

with 95% C.I.

QUALIFICATION QUALIFICATION

Best 5 naïve tracers

Assessment of variance due to side, trace-retrace, atrophy, time, scanner, rater

TRAINING SET

Assessment of agreement with volume on pathology or ex vivo

MRI and correlation with neuronal density

GANTT