IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 1 … · Landmark Detection Fig. 2. Illustration of the proposed anatomical Landmark based Deep Feature Learning (LDFL) framework

2168-2194 (c) 2017 IEEE. Personal use is permitted, but republication/redistribution requires IEEE permission. See http://www.ieee.org/publications_standards/publications/rights/index.html for more information.

This article has been accepted for publication in a future issue of this journal, but has not been fully edited. Content may change prior to final publication. Citation information: DOI 10.1109/JBHI.2018.2791863, IEEE Journal ofBiomedical and Health Informatics

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 1

Anatomical Landmark based Deep FeatureRepresentation for MR Images in Brain Disease

DiagnosisMingxia Liu†, Jun Zhang†, Dong Nie, Pew-Thian Yap, Dinggang Shen, Fellow, IEEE

Abstract—Most automated techniques for brain disease di-agnosis utilize hand-crafted (e.g., voxel-based or region-based)biomarkers from structural magnetic resonance (MR) images asfeature representations. However, these hand-crafted features areusually high-dimensional or require regions-of-interest (ROIs)defined by experts. Also, because of possibly heterogeneousproperty between the hand-crafted features and the subsequentmodel, existing methods may lead to sub-optimal performancesin brain disease diagnosis. In this paper, we propose a landmarkbased deep feature learning (LDFL) framework to automaticallyextract patch-based representation from MRI for automaticdiagnosis of Alzheimer’s disease (AD). We first identify discrim-inative anatomical landmarks from MR images in a data-drivenmanner, and then propose a convolutional neural network (CNN)for patch-based deep feature learning. We have evaluated theproposed method on subjects from three public datasets, in-cluding the Alzheimer’s Disease Neuroimaging Initiative (ADNI-1), ADNI-2, and the Minimal Interval Resonance Imaging inAlzheimer’s Disease (MIRIAD) dataset. Experimental results ofboth tasks of brain disease classification and MR image retrievaldemonstrate that the proposed LDFL method improves theperformance of disease classification and MR image retrieval.

Index Terms—Anatomical landmarks; convolutional neuralnetwork; classification; image retrieval; brain disease diagnosis.

I. INTRODUCTION

Alzheimer’s disease (AD) is an increasingly prevalent dis-ease, characterized by the accumulation of amyloid-β (Aβ)and hyperphosphorylated tau in the brain that eventually leadsto neurodegeneration [1]. The accurate diagnosis of AD is ofgreat importance for possible improvement in the treatment ofthe disease and is expected to help reduce costs associated withlong-term care for patients. To support AD diagnosis, manycomputer-aided approaches have been proposed using variousbiomarkers. Compared with the accumulation of Aβ detectedin cerebrospinal fluid (CSF) or by using positron emissiontomography (PET) [2], [3], biomarkers based on structuralmagnetic resonance imaging (MRI) could suggest structuralchanges of the brain in a more sensitive manner [4], [5].

Currently, many global and relatively local biomarkers(shown in Fig. 1) have been proposed for AD diagnosis with

This study was partly supported by NIH grants (EB006733, EB008374,EB009634, MH100217, AG041721, AG042599, AG010129, and AG030514).Corresponding author: Dinggang Shen.

M. Liu, J. Zhang, D. Nie, P.-T. Yap, and D. Shen are with the Departmentof Radiology and BRIC, University of North Carolina at Chapel Hill, NorthCarolina 27599, USA. D. Shen is also with Department of Brain and CognitiveEngineering, Korea University, Seoul 02841, Republic of Korea.

† M. Liu and J. Zhang contributed equally to this study.

ROI-based measure

Global

Voxel-based measure Patch-based measure

Local

Fig. 1. Illustration of MRI biomarkers for brain disease diagnosis shown ina local-to-global manner, including high-dimensional-morphological measure(e.g., voxel-based representation), ROI-based measure, and the proposedpatch-based measure.

MRI data, including global region-of-interest (ROI) based vol-umetric measures and local high-dimensional-morphological-analysis (HDMA) based measures. Specifically, ROI-basedmeasures (e.g., cortical thickness [6]–[10], hippocampal vol-ume [11]–[13], and gray matter volume [14], [15]) are tradi-tionally adopted to measure regionally anatomical volume andto investigate abnormal tissue structures in the brain. However,the definition of ROIs usually requires a priori hypothesis onthe abnormal regions from a structural/functional perspective,requiring expert knowledge in practice [16]. Also, an abnormalregion might be only a small part of a pre-defined ROIor span over multiple ROIs, thereby leading to loss of dis-criminative information. In addition, ROI-based measurementsdepend largely on two time-consuming steps that reduce thefeasibility of timely AD diagnosis: 1) non-linear registrationacross subjects, and 2) brain tissue segmentation [17]. As analternative solution, HDMA-based measures capture localizedstructural changes in a hypothesis-free manner to quantifybrain atrophy, among which voxel-based morphometry (VBM)[15], [18]–[20], deformation-based morphometry (DBM) [21],and tensor-based morphometry (TBM) [22] are the typicalexamples. Specifically, VBM directly measures local tissue(e.g., gray matter, white matter and cerebrospinal fluid) densityof a brain via voxel-wise analysis, DBM detects morphologicaldifferences from non-linear deformation fields that align/warpimages to a common anatomical template, and TBM identifiesregionally structural differences from local Jacobians of de-formation fields, respectively. While the number of subjects isoften limited (e.g., in hundreds), HDMA-based measurementis generally of very high dimension (e.g., in millions), leadingto the over-fitting problem in subsequent learning models [23].Also, the HDMA-based measure is usually limited by registra-tion errors or inter-subject anatomical variations. Therefore, itis desirable to extract discriminative biomarkers from MRI in a



2 IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS

...

Training Data

Patch-based Feature Learning

Convolutional Neural

Network 1

Landmark Discovery

Disease

ClassificationTesting Data

Landmark-based Patch Extraction

Convolutional Neural

Network P

...

Image Retrieval

...Deep Feature

Representation 1

Deep Feature

Representation P

Patch-based

Features

MR Image Pre-processing

...

Landmark Detection

...

Fig. 2. Illustration of the proposed anatomical Landmark based Deep Feature Learning (LDFL) framework. There are four main components: 1) landmarkdiscovery, 2) landmark-based patch extraction, 3) patch-based feature learning, and 4) applications of disease classification and image retrieval.

semi-global manner, independent of any pre-defined ROIs andtime-consuming pre-processing procedures, which may resultin better performance in brain disease diagnosis.

In this study, we propose an anatomical landmark baseddeep feature learning (LDFL) framework for AD diagnosis(see Fig. 2), by extracting patch-based measures from struc-tural MR imaging data. Different from conventional ROI-basedand voxel-based feature representations for MR images, wedevelop a novel patch-based feature extraction method forcomputer-aided brain disease diagnosis with MRI data. Specif-ically, we first identify discriminative anatomical landmarksvia group comparison between AD and normal control (NC)subjects, and then learn patch-based feature representationsfrom each landmark location via a deep convolutional neuralnetwork (CNN) [24]. The effectiveness of the proposed LDFLmethod is validated in both tasks of brain disease classificationand MR image retrieval. The major contributions of this workcan be summarized as follows. First, the proposed featurerepresentation can be automatically learned from MR imagingdata, without using pre-defined ROIs and time-consuming pre-processing procedures (e.g., brain tissue segmentation). Sec-ond, we propose to locate the most informative image patchesfrom MRI with millions of patches based on anatomical land-marks. Third, we use the Alzheimer’s Disease NeuroimagingInitiative (i.e., ADNI-1) [25] as the training set, and ADNI-2and MIRIAD as independent testing sets.

II. MATERIALS AND METHODS

A. Data Preparation

Our analysis is based on three public datasets, includingthe Alzheimer’s Disease Neuroimaging Initiative (ADNI-1)dataset [25], ADNI-2, and the Minimal Interval ResonanceImaging in Alzheimer’s Disease (MIRIAD) dataset1. Thereare a total of 199 AD and 229 NC subjects with 1.5T T1-weighted structural MRI data in the baseline ADNI-1 dataset,

1http://www.ucl.ac.uk/drc/research/methods/miriad-scan-database

whereas the baseline ADNI-2 dataset contains 159 AD and200 NC subjects with 3T T1-weighted structural MRI data.Note that in our experiments, several subjects that appear inboth ADNI-1 and ADNI-2 are removed from ADNI-2 forindependent testing. The general inclusion/exclusion criteriaused by ADNI-1 are summarized as follows: 1) NC subjects:Mini-Mental State Examination (MMSE) scores between 24-30 (inclusive), a Clinical Dementia Rating (CDR) of 0, non-depressed, non-MCI and non-demented; 2) mild AD: MMSEscores between 20-26 (inclusive), CDR of 0.5 or 1.0 andmeets NINCDS/ADRDA criteria for probable AD. There are23 NCs and 46 AD patients with baseline 1.5T T1-weightedMRI in MIRIAD. In the main experiments, ADNI-1 is usedas the training set, while ADNI-2 and MIRIAD are adoptedas independent testing sets. The demographic and clinicalinformation of subjects is reported in Table I.

In this study, we process all MR images using a standardpipeline. More specifically, we first perform anterior commis-sure (AC)-posterior commissure (PC) correction for each MRimage, using the MIPAV software package. We then re-sampleeach MR image to have a resolution of 256× 256× 256, fol-lowed by intensity inhomogeneity correction for images usingthe N3 algorithm [26]. Also, we perform skull stripping [27]for all MR images, as well as a process of manual editing, toensure that both skull and dura are cleanly removed. We finallyremove the cerebellum from each MR image, by warping alabeled template to each skull-stripped image.

B. Method

Figure 2 illustrates a general framework of our LDFL pro-posed method, including four main components: 1) landmarkdiscovery, 2) landmark-based patch extraction, 3) patch-basedfeature learning, and 4) applications of disease classificationand image retrieval. For landmark discovery, we first identifyAD landmarks that have statistically significant differencesbetween AD and NC subjects in the training set and thenapply a pre-trained landmark detection model to automatically



LIU et al.: ANATOMICAL LANDMARK BASED DEEP FEATURE REPRESENTATION FOR MRI 3

TABLE IDEMOGRAPHIC AND CLINICAL INFORMATION OF SUBJECTS IN THE BASELINE ADNI-1 AND ADNI-2 DATABASE. VALUES ARE REPORTED AS

MEAN±STANDARD DEVIATION (STD); EDU: EDUCATION YEARS; MMSE: MINI-MENTAL STATE EXAMINATION; CDR-SB: CLINICAL DEMENTIARATING-SUM OF BOXES.

Dataset Category Male/Female Edu (Mean±Std) Age (Mean±Std) MMSE (Mean±Std) CDR-SB (Mean±Std)

ADNI-1AD 106/93 13.09 ± 6.83 69.98 ± 22.35 23.27 ± 2.02 0.74 ± 0.25

NC 127/102 15.71 ± 4.12 74.72 ± 10.98 29.11 ± 1.01 0.00 ± 0.00

ADNI-2AD 91/68 14.19 ± 6.79 69.06 ± 22.04 21.66 ± 6.07 4.16 ± 2.01

NC 113/87 15.66 ± 3.46 73.82 ± 8.41 27.12 ± 7.31 0.05 ± 0.22

MIRIADAD 19/27 - 69.95 ± 7.07 19.19 ± 4.01 -NC 12/11 - 70.36 ± 7.28 29.39 ± 0.84 -

(a) 3D illustration of all identified AD-related landmarks (b) Top 50 selected landmarks

6080

100120

140160

180200

0

50

100

150

200

25020

40

60

80

100

120

140

160

180

0.001

0

Top 50 landmarks

Bottom 1691 landmarks

x

y

z

0.001

0

Sagittal view Coronal viewAxial view

Fig. 3. Illustration of (a) all identified AD-related anatomical landmarks from AD and NC subjects in ADNI-1, and (b) top 50 selected landmarks forpatch-level feature learning in AD diagnosis. Here, different colors denote p-values in group comparison between AD and NC groups in ADNI-1.

detect landmarks in each testing image [28]. In the stage oflandmark-based patch extraction, we extract multiple patchesfrom each training image based on each of multiple anatomicallandmarks. In the patch-based feature learning stage, wedevelop a CNN model to learn morphological features, wherepatches extracted from each landmark are used as input, andsubject-level labels for patches are adopted as output. Notethat each CNN model is corresponding to a specific landmarkposition. Finally, the patch-based deep feature representationslearned from multiple landmarks are fed into subsequentdisease classification and image retrieval models.

1) Anatomical Landmark Discovery: Our goal is to identifyregions that have group differences in local brain structuresbetween AD patients and normal controls (NCs). Follow-ing previous studies [28], [29], we perform a voxel-wisegroup comparison between AD and NC groups in ADNI-1.Specifically, we first linear-aligned all training images to theColin27 template [30] to remove global translation, as wellas the scale and rotation differences of MR images. Also,we re-sampled all images to have the same spatial resolution(i.e., 1 × 1 × 1mm3). Then, we non-linearly aligned alltraining images to the template, to build the correspondenceamong voxels from different images, using the HAMMERalgorithm [31]. By using the deformation field from non-linearregistration, we can establish the correspondence between eachvoxel in the template and those in the linearly-aligned images.For each voxel in the template, we extracted two groups ofmorphological features (i.e., local energy pattern [32]) fromcorresponding voxels in all training images from the group of

AD patients and the group of NCs, respectively. Then, basedon the morphological features for each voxel, we performeda multivariate statistical test (i.e., Hotelling’s T2 [33]) onAD and NC groups, and thus can obtain a p-value for eachvoxel in the template space. Given all voxels in the template,we generated a p-value map corresponding to the template.Finally, the local minima from the p-value map were identifiedas locations of discriminative anatomical landmarks in thetemplate space. Finally, we projected these landmark locationsto all linearly-aligned training images using their respectivedeformation fields (generated in the non-linear registration).

To fast locate anatomical landmarks for testing images, wefurther trained a regression forest based landmark detector,with landmark as output and those linearly aligned trainingimages as input. For a new testing MR image, we can alignit linearly to the template space and then use our trainedlandmark detector to identify landmarks in the linearly-alignedtesting image. In this way, both training images and testingimages would have the same landmarks. For each MR image,there are approximately 1700 landmarks identified from ADand NC subjects in the ADNI-1 dataset, shown in Fig. 3 (a).

From Fig. 3 (a), we can observe that some landmarks arevery close to each other. In such a case, directly using thoselandmark for patch extraction may lead to overlapped patches.To address this issue, for all identified landmarks rankedaccording to p-values in descending manner, we further definea spatial Euclidean distance threshold (i.e., 16) to control thedistance between landmarks, to reduce the overlaps amongimage patches. Finally, we select the top 50 landmarks for deep




32@3*3*3

64@

2*2

*2

max-

pooling

19

19

19

19

19

19

32

3

3

3

19

19

19

32

10

10

10

32

3

3

3

10

10

10

64

3

3

3

10

10

10

64

Input Patch Conv1 Conv2 Conv4 Con5

2

2

2

2

2

2

32

5

5

564

5

5

532

Conv7FC11

2 Nodes

FC9

256 Nodes

…

FC10

…

128 Nodes

soft-max

Class Label

1/0

33

3

3

3

3

2

2

2

max-

pooling

22

2

fully connected convolutionfully connectedfully connected

Pool6Pool8

Pool3

max-

poolingconvolutionconvolution convolution convolution

32@3*3*3 32@2*2*2 64@3*3*3 64@2*2*2

32@3*3*332@2*2*2

Fig. 4. Schematic diagram of CNN for patch-based feature learning. Conv: Convolutional layer; Pool: Pooling layer; FC: Fully connected layer.

feature learning [34], and show these landmarks in Fig. 3 (b),Fig. S2 and Movie S1 of the Supplementary Materials.

2) Landmark-based Patch Extraction: To suppress the neg-ative influence of landmark localization errors introduced byimage registration, we sample multiple image patches (withthe size of 19 × 19 × 19) from each landmark location withdisplacements in a 3×3×3 cubic. Hence, there are 27 imagepatches extracted from a specific landmark position in an MRimage of a specific subject. The class label of each image patchis assigned as the same label of that subject (i.e., subject-level label). The influence of parameters (i.e., the numberof landmarks and size of patches) is shown in Fig. S1 inSupplementary Materials.

3) Patch-based Feature Learning: We develop a CNNmodel [35] to extract discriminative patch-based biomarkersfrom MRI for AD diagnosis, with a schematic diagram shownin Fig. 4. As shown in Fig. 4, the input of the network includemultiple image patches extracted from a specific landmarkposition in brain MR images, which are convolved by a seriesof 5 convolutional layers (i.e., Conv1, Conv2, Conv4, Conv5,and Conv7) with rectified linear unit (ReLU) activation. Here,Conv2, Conv5, and Conv7 are followed by max-pooling proce-dures to perform a down-sampling operation for their outputs,respectively. The size of a 3D kernel in each convolutionallayer is 3 × 3 × 3. The resulting 256-dimensional features,which are equal to the number of feature maps of the lastconvolutional layer (i.e., Conv7), are fed into a series of 3fully connected (FC) layers (i.e., FC9, FC10, and FC11) withdimensions of 256, 128, and 2, since 2 is the number of classesconsidered. The use of FC layers accelerates convergence,while the problem of over-fitting could be partly solved byadding a drop-out layer (with a ratio of 0.5) before FC10.The output of the last FC layer (i.e., FC11) is fed into a soft-max top-most output layer, to predict the probability of aninput image patch belonging to AD or NC group. Note thatthe information given by the subject-level class labels are usedin a back-propagation procedure.

Denote L as the number of landmarks, and the trainingset as X = {Xn}Nn=1 that contains N subjects with thecorresponding labels y = {yn}Nn=1. We denote the n-th

training image as Xn = {xn,1,xn,2, · · · ,xn,L} containingL image patches. As shown in Fig. 2, patches extracted fromtraining images are the basic training samples for our proposedCNN model, and the labels of those patches are subject-level labels. That is, the subject-level label information (i.e.,yn(n = 1, · · · , N)) is used as the supervision information fornetwork training. More specifically, the proposed CNN aimsto learn a mapping function Φ : X → y. For the l-th CNNmodel corresponding to the l-th (l = 1, · · · , L) landmark, theobjective function is shown as follows:

minWl

∑{xn,l∈Xn}N

n=1

−log (P(yn|xn,l;Wl)) (1)

where P(yn|xn,l;Wl) indicates the probability of the patchxn,l being correctly classified as the class yn using the networkcoefficients Wl.

The implementation of the proposed CNN model is basedon Caffe [36], and the computer we used in the experimentscontains a single GPU (i.e., NVIDIA GTX TITAN 12GB). Thenetwork is optimized by stochastic gradient descent (SGD)algorithm [37] with a momentum coefficient of 0.9. Thelearning rate is set to 10−2, and the weight updates areperformed in mini-batches of 30 samples per batch. Here, wefurther randomly select 10% subjects in ADNI-1 as validationdata, while the remaining in ADNI-1 are regarded as thetraining data. The training process ends when the network doesnot significantly improve its performance on the validation setwithin 60 epochs.

4) Applications of Disease Classification and MR ImageRetrieval: Given L (L = 50 in this study) anatomicallandmarks, we can train L CNN models, and each model iscorresponding to a specific landmark. In this way, for eachMR image, we can obtain L feature vectors via those CNNmodels. For brain disease classification, we simply combinethe estimated patch labels in 50 landmark locations achievedby CNNs using the majority voting strategy [38]. For the taskof MR image retrieval, we use the outputs from three FC layers(i.e., FC9, FC10, and FC11) of each CNN as patch-basedfeature representations, and compare them with conventionalrepresentations for MRI in the experiments.




III. EXPERIMENTS

A. Methods for Comparison

We compare our proposed LDFL method with three state-of-the-art feature representations for MR images, including1) ROI-based representation [10], [39], 2) voxel-based mor-phometry (VBM) representation [18], 3) landmark-based mor-phology (LMF) representation [28]. We briefly introduce thesethree methods as well as our method in the following.

1) ROI-based (ROI) representation. Following previousstudies [10], [39], we extract ROI-based features from MRimages in this method. To be specific, we first segment thestudied MR image into three tissue types, including gray mat-ter (GM), white matter (WM), and cerebrospinal fluid (CSF),by using the FAST algorithm in the FSL software. Then, wealign the anatomical automatic labeling (AAL) atlas [40] with(90 pre-defined ROIs in the cerebrum) to the native space ofeach subject using a deformable registration algorithm (i.e.,HAMMER [31]). We finally extract the volumes of GM tissuein 90 ROIs as feature representation for each MR image.Note that the volumes of GM tissue are normalized by thetotal intracranial volume (estimated by the summation of GM,WM, and CSF volumes). In the classification task, we feedthe ROI features into a linear support vector machine (LSVM)classifier [41] for disease classification. In the image retrievaltask, we adopt these ROI features to represent each MRI.

2) Voxel-based morphometry (VBM) representation [18]:Similar to the pre-processing procedure used in the ROI-basedmethod, we first normalize the studied MR images usingthe same registration and segmentation methods. Then, weextract the local tissue density of GM in a voxel-wise manneras feature representations for an MR images. To reduce thefeature dimension, we further perform the t-test algorithm toselect the most informative features. Those selected voxel-based features are finally fed into an LSVM for classificationand used as the representation for each MRI in the task ofimage retrieval.

3) Landmark based morphology (LMF) based repre-sentation [28] with engineered feature representations. InLMF, we first extract morphological features (i.e., local en-ergy pattern [32]) from each local image patch centered ateach landmark location, and then concatenates these featuresextracted from multiple landmarks, followed by a z-scorenormalization [42] process. Finally, the normalized featuresare used in both tasks of disease classification (via LSVM) andimage retrieval. As a landmark-based method, LMF shares thesame landmark pool as our proposed LDFL method, shown inFig. 3. It is worth noting that, different from our proposedLDFL approach that learns features automatically from MRI,LMF adopts human-engineered features for representing im-age patches around each landmark position.

4) Proposed patch-based representation. For each MRimage, we can obtain L feature vectors via the proposedCNN model in Fig.4. We then extract three types of featuresas the representation for MRI, including 1) features fromFC9 layer in the proposed CNN that is denoted as FC9 forshort, 2) features from FC10 layer (denoted as FC10), and 3)features from FC11 layer (denoted as FC11). Similar to LMF

method, we can simply concatenate the feature vectors fromL landmarks for subsequent model learning.

In the disease classification task, we feed those ROI andVBM features to an LSVM classifier, respectively. It is worthnoting that, for landmark-based features (i.e., LMF [28],FC9, FC10, and FC11), we further propose two strategies toutilize these features, including 1) feature concatenation and2) classifier ensemble. Specifically, in feature concatenationmethods (denoted as LMF con, FC9 con, FC10 con, andFC11 con), features learned from 50 landmarks are simplyconcatenated into a long feature vector, followed by an LSVMclassifier. In ensemble based methods (denoted as LMF ens,FC9 ens, FC10 ens, and FC11 ens), we first train an LSVMusing features obtained from each landmark position and canobtain 50 LSVMs given 50 landmarks. Then, the results ofthose LSVMs are combined by a majority voting strategy [38]for final classification. In the image retrieval task, we comparethe proposed three types of patch-based feature represen-tations via feature concatenation (i.e., FC9 con, FC10 con,and FC11 con) with three conventional representations forMRI (i.e., ROI, VBM, and LMF con). Specifically, we firstrepresent each MR image (w.r.t. a particular subject) usinga specific feature vector, and then compute the Euclideandistance between a new query MR image in ADNI-2 and eachof MRI in ADNI-1.

B. Experimental Settings

The performance of disease classification is evaluated bythe following metrics: 1) classification accuracy (ACC), 2)sensitivity (SEN), 3) specificity (SPE), 4) receiver operatingcharacteristic curve (ROC), 5) area under ROC (AUC), and 6)F-Measure [44]. We denote TP, TN, FP, FN and PPV as truepositive, true negative, false positive, false negative, and posi-tive predictive value, respectively. These evaluation metrics aredefined as: ACC= (TP+TN)

(TP+TN+FP+FN) , SEN= TP(TP+FN) , SPE= TN

(TN+FP) , F-Measure= (2×SEN×PPV)

(SEN+PPV) where PPV= TP(TP+FP) . In image retrieval

experiments, we utilize four metrics for performance evalu-ation, including 1) mean average precision (MAP) for a setof queries that is the mean of average precision scores foreach query (where each subject with MR image in ADNI-2 is used as a specific query), 2) F-Measure, 3) Matthewscorrelation coefficient (MCC) [45] that is a balanced measurefor binary classes, and 4) #Correct@K that is the number ofcorrect results in top K returned results. The most relevantresults should be ranked at top-most positions resulting inhigher #Correct@K values.

C. Results

In this section, we first show the learned features and thelearned kernels by the proposed LDFL method. Then, wereport the experimental results in both tasks of disease classi-fication and MR image retrieval, by comparing the proposedmethod with those competing methods.

1) Learned Feature Representations: It is worth noting thateach layer of CNN combines the extracted low layer featuremaps to learn higher level features at the next layer, in ahierarchical manner for describing more abstract anatomical




-30 -20 -10 0 10 20 30-30

-20

-10

0

10

20

30

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-30 -20 -10 0 10 20 30-20

-15

-10

-5

0

5

10

15

20

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-30 -20 -10 0 10 20 30-30

-20

-10

0

10

20

30

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-30 -20 -10 0 10 20 30-30

-20

-10

0

10

20

30

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-30 -20 -10 0 10 20 30-30

-20

-10

0

10

20

30

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-30 -20 -10 0 10 20 30-40

-30

-20

-10

0

10

20

30

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-40 -20 0 20 40-40

-30

-20

-10

0

10

20

30

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

-30 -20 -10 0 10 20 30 40-30

-20

-10

0

10

20

30

40

Reduced Dimension x

Red

uced

Dim

ensi

on y

NCAD

(a) (b)

(e) (f) (g) (h)

(c) (d)

Fig. 5. Manifold visualization of AD and NC subjects in the ADNI-2 dataset, by t-SNE projection [43] in learned 3D-CNN layers including (a) Conv1, (b)Conv2, (c) Conv4, (d) Conv5, (e) Conv7, (f) FC9, (g) FC10, and (h) FC11.

variations of a brain. To analyze the discriminative capabilityof the patch-based features learned from our method, wevisualize the learned features at 5 convolutional and 3 fullyconnected layers in CNN, by projecting those features downto 2 dimensions using the t-SNE dimension reduction algo-rithm [43]. As can be seen from Fig. 5 (a-e), the convolutionallayers (i.e., Conv1, Conv2, Conv4, Conv5, and Covn7) grad-ually enhance the discriminative power between AD and NCsubjects along the hierarchy. Also, Fig. 5 (f-h) indicate that thesubsequent task-specific classification layers (i.e., FC9, FC10,and FC11) further enhance the separability between AD andNC subjects, and features at the top-most FC layers (FC10,and FC11) are most discriminative.

2) Learned Kernels in CNN: Figure 6 shows 32 convo-lutional kernels learned at the Conv1 layer via the proposedCNN model (see Fig. 4). From Fig. 6, one may notice thedifferent natures of these kernels in capturing fundamental 3Dpatterns. These patterns vary complexity while passing throughconsecutive convolutional layers, so that the last layer couldhave the ability to describe the structural differences betweenAD and NC subjects. In addition, given an input image patch,we show the outputs of each convolutional layer in CNN inFigs. S6-S9 in the Supplementary Materials.

3) Results of Disease Classification: In the task of braindisease classification (i.e., AD vs. NC classification), wefirst learn features from MRI data in a supervised mannervia our LDFL framework. In this group of experiments, wetreat subjects in ADNI-1 as training data, and use subjectsin ADNI-2 as independent testing data. Figure 7 reportsclassification performances achieved by methods using con-ventional features and our patch-based features, as well asour LDFL method in AD vs. NC classification. As could beseen from Fig. 7, methods using our patch-level deep featuresusually achieve much better diagnosis results, compared withthose using conventional feature representations. For instance,LDFL achieves the best accuracy of 90.56% and the bestF-Measure of 89.10%, while ROI, VBM, LMF con, and

LMF ens generally result in worse performance. On the otherhand, ensemble based methods usually perform better thanfeature concatenation based methods. For instance, FC9 ensachieves an AUC of 94.72%, while the AUC of its counterpart(i.e., FC9 con) is only 91.40%. Also, among three patch-basedmeasures learned from different FC layers in the proposedCNN model, the method using features extracted from thetop-most FC layer (i.e., FC11) results in the best result. Thatis, FC11 ens achieves an accuracy of 88.61% and an AUC of95.83%. It is worth noting that MR images in the training set(i.e., ADNI-1) were acquired by 1.5T T1-weighted scanners,while those in the testing set (i.e., ADNI-2) were acquired by3T T1-weighted scanners. Despite the different signal-to-noiseratios of MRI in the training and the testing set, our LDFLmethod still achieves good results in AD classification. Thisimplies that LDFL has good generalization capability.

In Fig. 8, we further plot the ROC curves achieved by theproposed methods and conventional approaches. As can beseen from Fig. 8, the overall best performances are achievedby the proposed FC11 con and LDFL methods among sixfeature concatenation based methods and the five ensemblebased approaches, respectively. It further demonstrates theeffectiveness of the proposed patch-based biomarkers learnedfrom MR imaging data. More results are given in Fig. S4-S6in the Supplementary Materials.

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

False Positive Rate (FPR)

Tru

e P

ositiv

e R

ate

(T

PR

)

LMF_ens

FC9_ens

FC10_ens

FC11_ens

LDFL

0 0.2 0.4 0.6 0.8 10

0.2

0.4

0.6

0.8

1

False Positive Rate (FPR)

Tru

e P

ositiv

e R

ate

(T

PR

)

ROI

VBM

LMF_con

FC9_con

FC10_con

FC11_con

(a) (b)

Fig. 8. ROC curves achieved by (a) feature concatenation based methods and(b) ensemble based methods in AD vs. NC classification. The classifiers aretrained on ADNI-1 and tested on ADNI-2.




-0.2-0.100.1

-0.100.1

-0.2

0

0.2

-0.100.1

-0.100.1

-0.100.10.2

-0.100.1

-0.100.1

-0.100.1

-0.100.1

-0.2-0.100.1

-0.2

0

0.2

-0.100.1

-0.100.1

-0.100.1

-0.2

0

0.2

-0.2

0

0.2

-0.100.10.2

-0.100.10.2

-0.2

0

0.2

-0.2

0

0.2

-0.2-0.100.1

-0.100.1

-0.100.1

-0.100.1

-0.100.1

-0.2-0.100.1

-0.100.1

-0.1

0

0.1

-0.100.1

-0.100.1

-0.100.1

Fig. 6. Illustration of the learned 32 convolutional kernels (with the size of 3× 3× 3) at the Conv1 layer in the proposed CNN architecture for AD vs. NCclassification.

79.17

78.62

79.60

86.73

76.92

80.50

77.35

83.00

87.62

77.84

82.22

77.36 86.07

88.11

79.35

77.78

71.07 83.08

85.72

73.8686.39

83.65

88.56

91.40

84.44

85.83

82.39

88.56

91.23

83.71

87.78

79.25

94.53

95.75

85.14

86.67

77.99

93.53

94.72

83.78

87.78

80.50 93.53

95.20

85.33

88.61

82.39 93.53

95.83

86.47

90.56

87.42

93.03

95.74

89.10

60

70

80

90

100

1 2 3 4 5

ROIF VBF LMF_con LMF_ens F9_con F10_con F11_con F9_ens F10_ens F11_ens LDFL

ACC SEN SPE AUC F-Measure

Cla

ssific

ation R

esults (

%)

Fig. 7. Results in AD vs. NC classification achieved by different methods, where subjects with MR imaging in ADNI-1 are used as training data and those inADNI-2 are used as independent testing data. ACC: accuracy, SEN: sensitivity, SPE: specificity, AUC: area under the receiver operating characteristic curve.

4) Results of MR Image Retrieval: In the task of MRimage retrieval, subjects in the ADNI-1 dataset are used asthe existing subjects’ MRI database, while all the MR imagesin the ADNI-2 dataset are alternated used as the query image.Figure 9 reports the performance comparison of methods usingdifferent feature representations. Figure 9 (a) implies thatmethods using our patch-based deep features (i.e., FC9 con,FC10 con, and FC11 con) usually outperform those usingconventional features (i.e., ROI, and LMF con) in MR imageretrieval. Also, the best mean average precision (MAP), F-Measure, and Matthews correlation coefficient (MCC) areachieved by the proposed FC11 con, FC11 con, and FC9 conmethods, respectively.

From Fig. 9 (b), we could observe that features learned fromthe proposed methods result in better #Correct@K values,compared with ROI and LMF con. For instance, FC11 concan return 8.27 relevant subjects in top 10 returned resultsand 38.23 relevant subjects in top 50 returned results. Thebetter performance on #Correct@K indicates that the proposedmethod can return more relevant medical records. This couldbe due to the discriminative features learned from landmark-based CNN models, which can precisely locate possible rele-vant subjects and then finally provide a fine ranking list. Givenan MR image of a query subject represented by the learnedpatch-based features, it is possible to provide physicians withuseful reference from historical data (e.g., existing patients’MRI database) to help design a subject-specific treatment plan.

IV. DISCUSSION

Although extensive studies investigate to extract differentfeature representations from structural MR imaging data forcomputer-aided AD diagnosis, most of them focus on globalor local measures that require time-consuming pre-processing

0

1

2

3

4

5

Image R

etr

ieval R

esults

ROI

LMF_con

FC9_con

FC10_con

FC11_con

1 3 5 10 20 50

0

10

20

30

40

#C

orr

ect@

K

ROI

LMF_con

FC9_con

FC10_con

FC11_con

MAPF-Measure

MCC

#Correct@3

#Correct@5K

Fig. 9. Image retrieval results achieved by different methods. All the MRimages in the ADNI-2 dataset are alternated used as the query image. MAP:mean average precision; MCC: Matthews correlation coefficient; #Correct@K:number of correct results in top K returned results.

procedures or depend on pre-defined ROIs, respectively. Tothis end, we develop a landmark based deep feature learning(LDFL) framework to automatically extract patch-level repre-sentations from MR images, based on anatomical landmarksdiscovered from data via a data-driven algorithm. In particular,we first propose to identify the most discriminative anatom-ical landmarks via group comparison between AD and NCsubjects. Given L (L = 50 in this study) landmarks, we thentrain L CNN models for patch-based deep feature learning,with each CNN corresponding to a specific landmark. Bothdisease classification and image retrieval experiments on threepublic datasets (i.e., ADNI-1, ADNI-2, and MIRIAD) suggestthat the proposed method could help promote the performanceof computer-aided AD diagnosis.

A. Discriminative Capability of Landmarks

Figure 10 illustrates the patch classification accuracy ineach landmark location achieved by our proposed LDFLmethod. One could see from Fig. 10 that the accuracy ofpatches in each CNN model is different from each other. For




50

60

70

80

90

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50

Index of Landmarks

ACC(%)

Fig. 10. Classification accuracies of patches in each landmark position for the testing data in the ADNI-2 dataset.

-0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

Kappa Diversity

Avera

ged C

lassifie

r E

rror

Kappa Measure

Ave

rag

ed

Cla

ssifie

r E

rro

r

0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.40.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34

0.36

0.38

0.4

0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.40.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34

0.36

0.38

0.4

0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.40.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34

0.36

0.38

0.4

0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.40.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34

0.36

0.38

0.4

0.2 0.22 0.24 0.26 0.28 0.3 0.32 0.34 0.36 0.38 0.40.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34

0.36

0.38

0.4

LMF_ens

FC9_ens

FC10_ens

FC11_ens

LDFL

Centroid of LMF_ens

Centroid of FC9_ens

Centroid of FC10_ens

Centroid of FC11_ens

Centroid of LDFL

-0.4 -0.2-0.6 0.2 0.40 0.8 10.6

0.2

0.1

0.4

0.3

0.6

0.5

0.8

0.7

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.1

0.2

0.3

0.4

0.5

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.1

0.2

0.3

0.4

0.5

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.1

0.2

0.3

0.4

0.5

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.1

0.2

0.3

0.4

0.5

0 0.02 0.04 0.06 0.08 0.1 0.12 0.14 0.16 0.18 0.20

0.1

0.2

0.3

0.4

0.5

Fig. 11. Kappa-error diagram achieved by five ensemble based methods in AD vs. NC classification. The value on the x-axis denotes the kappa measure [46]of a pair of classifiers in the ensemble, whereas the value on the y-axis is the averaged individual error of a pair of classifiers.

instance, the patch classification accuracies at the 1st landmarkand the 50th landmark positions are 80.80% and 70.87%,respectively. Among those 50 landmarks, the best accuracies(i.e., > 80.00%) are achieved in a subset of landmarks (i.e.,1, 2, 4, 5, 9, 11, 13, 14, 16, 17, 21, 24, 31), implying the struc-tural changes in these landmarks could be more discriminativein distinguishing AD from normal controls (NCs).

From Fig. 3 (b), one may observe that landmarks in thatsubset mainly locate in the areas of bilateral hippocampus,bilateral parahippocampus, and bilateral fusiform. These areasare reported to be related to AD in previous studies [13],[47]–[49]. More results can be found in Figs. S2-S3 in theSupplementary Materials. Besides, Fig. 10 suggests that theoverall performance of patch classification gradually becomesworse with the increase of landmark index. The underlyingreason could be that the p-values (in the group comparisonbetween AD and NC subjects) of those 50 landmarks graduallyincrease, and thus their discriminative capabilities becomeworse in group comparison in the landmark discovery pro-cess [28]. Also, the most discriminative features are learnedfrom patches located at the 13 th and the 14 th landmarks,other than from patches at the first landmark location shownin Fig. 3. The main reason is that we use hand-crafted features(i.e., local energy pattern [32]) of MRI to identify anatomicallandmarks, while the proposed landmark-based patch-levelfeatures are learned automatically from data for AD diagnosis.

B. Diversity AnalysisWe adopt a kappa measure [46] to analyze the diversity

of classifiers in 5 ensemble-based methods for AD vs. NC

classification. The kappa measure evaluates the level of agree-ment between the outputs of two classifiers. In Fig. 11, weshow a diversity-error diagram achieved by different methods.For each method, the corresponding ensemble contains 50individual classifiers for 50 landmarks. The value on the x-axisof a diversity-error diagram denotes the kappa measure of apair of classifiers in the ensemble, whereas the value on the y-axis is the averaged individual error of a pair of classifiers. Asa small value of kappa measure indicates better diversity anda small value of averaged individual error indicates a betteraccuracy, the most desirable pairs of classifiers will be closeto the bottom left corner of the graph. We further plot thecentroids of clouds achieved by different methods in Fig. 11for visual evaluation of relative positions of kappa-error points.

Figure 11 suggests that the proposed FC11 ens methodoutperforms the competing methods regarding the averagedclassification error, while our FC10 ens method achieves thebest diversity regarding kappa measure. Also, the proposedLDFL method gives the overall best trade-off between theclassification error and the diversity of multiple classifiers,compared with the other four methods. That is, LDFL buildsa classifier ensemble based on the reasonably accurate butmarkedly diverse individual components.

C. Computational Cost

We now analyze the computational costs of the proposedLDFL method. It is worth noting that, for our LDFL andthree competing methods (i.e., ROI, VBM, and LMF [28]), alltraining processes are performed off-line. Hence, we analyze




TABLE IICOMPUTATIONAL COSTS OF DIFFERENT METHODS IN AD VS. NC CLASSIFICATION FOR A NEW TESTING MR IMAGE.

Method Time of Each Process (Platform) Total Time

ROI

1) Linear alignment 5.00 s (C++)

∼ 32.00min2) Non-linear registration 32.00min (HAMMER [31])3) Feature extraction 3.00 s (Matlab)4) Classification 0.02 s (Matlab)

VBM


∼ 32.00min2) Non-linear registration 32.00min (HAMMER [31])3) Feature extraction 4.00 s (Matlab)4) Classification 0.05 s (Matlab)

LMF


∼ 20.00 s2) Landmark prediction 10.00 s (Matlab)3) Feature extraction 5.00 s (Matlab)4) Classification 0.03 s (Matlab)

LDFL (Ours)1) Linear alignment 5.00 s (C++)

∼ 15.00 s2) Landmark prediction 10.00 s (Matlab)3) Joint feature extraction and classification 0.31 s (Caffe [36])

the on-line computational cost for a new testing subject withan MR image. Specifically, in the ROI method, we first linearlyalign the testing image to the template, and then use the non-linear registration algorithm [31] to map segmentations of graymatter (GM) and the 90 ROIs from template image to thetesting image, followed by a linear support vector machine(SVM) classifier. Similar to ROI, we use the same registrationand segmentation strategies for voxel-based method (VBM). Inour LDFL method, we first linearly align the new testing MRimage to the template, and then predict the landmark positionsfor this image. We further extract image patches from eachlandmark location and feed them to the proposed CNN modelfor joint feature learning and disease prediction. For LMF [28]method, we extract morphological features [32] of the linearly-aligned testing image based on the same landmarks as LDFL,and then perform prediction using the linear SVM classifier.

Table II reports the computational costs of different meth-ods. From Table II, we can see that the total computationalcosts of both ROI and VBM are more than half an hour, whichis much slower than our method (15 s). Although LMF hassimilar computational cost (20 s), its learning performance isworse than our proposed LDFL method (see results in Fig. 7and Figs. S4-S6 in the Supplementary Materials).

D. Technical Limitations

Although the proposed LDFL method achieved promisingresults in both tasks of brain disease diagnosis and MR imageretrieval, several technical issues need to be considered. First,the anatomical landmarks used in this study is pre-definedin our previous study [28]. That is, the process of landmarkdefinition is independent of our proposed patch-level featurelearning, which may lead to sub-optimal performance. Areasonable solution is to integrate landmark identification andlandmark-based feature learning into a unified deep learningframework, which will be one of our future works. Second andmore generally, the anatomical landmarks used in this studywere discovered in a data-driven manner. However, it remainsunknown which subset of landmarks is the most informativefor subsequent feature learning. Therefore, it could be inter-esting to investigate an optimal subset of identified landmarksfor patch-based feature learning. As one of our future works,

we will let experts refine these landmarks to make the usedlandmarks more compact. Besides, only the baseline MRI datain three datasets (i.e., ADNI-1, ADNI-2, and MIRIAD) areused in this work. In these datasets, there exist longitudinalMRI data that may provide complementary information forthe proposed feature learning method. Furthermore, we learnmultiple CNN models (w.r.t. multiple landmarks) separately inthe current study, without considering the context information(e.g., spatial locations) of those identified landmarks. Hence,future work will cover the development of a joint deep learningmodel by considering the landmarks jointly and globally.

V. CONCLUSION

In this paper, we propose a landmark based deep featurelearning (LDFL) framework, to automatically extract patch-based representations from MR images for AD-related braindisease diagnosis. Experimental results on three cohorts (i.e.,ADNI-1, ADNI-2, and MIRIAD) demonstrate the effective-ness of the proposed method in both tasks of disease classifi-cation and MR image retrieval. This approach paves the wayto discriminative biomarkers for computer-aided diagnosis ofAD and the morphological analysis of MR images.

REFERENCES

[1] J. Hardy and D. J. Selkoe, “The amyloid hypothesis of Alzheimer’sdisease: Progress and problems on the road to therapeutics,” Science,vol. 297, no. 5580, pp. 353–356, 2002.

[2] C. Lian, S. Ruan, T. Denoeux, H. Li, and P. Vera, “Spatial evidentialclustering with adaptive distance metric for tumor segmentation in FDG-PET images,” IEEE Transactions on Biomedical Engineering, vol. 65,no. 1, pp. 21–30, 2017.

[3] C. Lian, S. Ruan, T. Denœux, F. Jardin, and P. Vera, “Selectingradiomic features from FDG-PET images for cancer treatment outcomeprediction,” Medical Image Analysis, vol. 32, pp. 257–268, 2016.

[4] G. B. Frisoni, N. C. Fox, C. R. Jack, P. Scheltens, and P. M. Thompson,“The clinical use of structural MRI in Alzheimer disease,” NatureReviews Neurology, vol. 6, no. 2, pp. 67–77, 2010.

[5] E. M. Reiman, J. B. Langbaum, and P. N. Tariot, “Alzheimer’s preventioninitiative: A proposal to evaluate presymptomatic treatments as quicklyas possible,” Biomarkers in Medicine, vol. 4, no. 1, pp. 3–14, 2010.

[6] B. Fischl and A. M. Dale, “Measuring the thickness of the humancerebral cortex from magnetic resonance images,” Proceedings of theNational Academy of Sciences, vol. 97, no. 20, pp. 11 050–11 055, 2000.

[7] R. Cuingnet, E. Gerardin, J. Tessieras, G. Auzias, S. Lehericy, M.-O.Habert, M. Chupin, H. Benali, O. Colliot, A. D. N. Initiative et al.,“Automatic classification of patients with Alzheimer’s disease fromstructural MRI: A comparison of ten methods using the ADNI database,”NeuroImage, vol. 56, no. 2, pp. 766–781, 2011.




[8] J. Lotjonen, R. Wolz, J. Koikkalainen, V. Julkunen, L. Thurfjell,R. Lundqvist, G. Waldemar, H. Soininen, and D. Rueckert, “Fast androbust extraction of hippocampus from MR images for diagnostics ofAlzheimer’s disease,” NeuroImage, vol. 56, no. 1, pp. 185–196, 2011.

[9] M. Liu, J. Zhang, P.-T. Yap, and D. Shen, “View-aligned hypergraphlearning for Alzheimer’s disease diagnosis with incomplete multi-modality data,” Medical Image Analysis, vol. 36, pp. 123–134, 2017.

[10] M. Liu, D. Zhang, and D. Shen, “Relationship induced multi-templatelearning for diagnosis of Alzheimer’s disease and mild cognitive im-pairment,” IEEE Transactions on Medical Imaging, vol. 35, no. 6, pp.1463–1474, 2016.

[11] C. R. Jack, R. C. Petersen, P. C. O’Brien, and E. G. Tangalos, “MR-based hippocampal volumetry in the diagnosis of Alzheimer’s disease,”Neurology, vol. 42, no. 1, pp. 183–183, 1992.

[12] C. Jack, R. C. Petersen, Y. C. Xu, P. C. OBrien, G. E. Smith, R. J.Ivnik, B. F. Boeve, S. C. Waring, E. G. Tangalos, and E. Kokmen, “Pre-diction of AD with MRI-based hippocampal volume in mild cognitiveimpairment,” Neurology, vol. 52, no. 7, pp. 1397–1397, 1999.

[13] M. Atiya, B. T. Hyman, M. S. Albert, and R. Killiany, “Structuralmagnetic resonance imaging in established and prodromal Alzheimer’sdisease: A review,” Alzheimer Disease and Associated Disorders, vol. 17,no. 3, pp. 177–195, 2003.

[14] H. Yamasue, K. Kasai, A. Iwanami, T. Ohtani, H. Yamada, O. Abe,N. Kuroki, R. Fukuda, M. Tochigi, S. Furukawa et al., “Voxel-basedanalysis of MRI reveals anterior cingulate gray-matter volume reductionin posttraumatic stress disorder due to terrorism,” Proceedings of theNational Academy of Sciences, vol. 100, no. 15, pp. 9039–9043, 2003.

[15] E. A. Maguire, D. G. Gadian, I. S. Johnsrude, C. D. Good, J. Ashburner,R. S. Frackowiak, and C. D. Frith, “Navigation-related structural changein the hippocampi of taxi drivers,” Proceedings of the National Academyof Sciences, vol. 97, no. 8, pp. 4398–4403, 2000.

[16] G. W. Small, L. M. Ercoli, D. H. Silverman, S.-C. Huang, S. Komo,S. Y. Bookheimer, H. Lavretsky, K. Miller, P. Siddarth, N. L. Rasgonet al., “Cerebral metabolic and cognitive decline in persons at geneticrisk for Alzheimer’s disease,” Proceedings of the National Academy ofSciences, vol. 97, no. 11, pp. 6037–6042, 2000.

[17] M. Liu, D. Zhang, E. Adeli, and D. Shen, “Inherent structure-based mul-tiview learning with multitemplate feature representation for Alzheimer’sdisease diagnosis,” IEEE Transactions on Biomedical Engineering,vol. 63, no. 7, pp. 1473–1482, 2016.

[18] J. Ashburner and K. J. Friston, “Voxel-based morphometry–The meth-ods,” NeuroImage, vol. 11, no. 6, pp. 805–821, 2000.

[19] J. Baron, G. Chetelat, B. Desgranges, G. Perchey, B. Landeau,V. De La Sayette, and F. Eustache, “In vivo mapping of gray matterloss with voxel-based morphometry in mild Alzheimer’s disease,” Neu-roImage, vol. 14, no. 2, pp. 298–309, 2001.

[20] S. Kloppel, C. M. Stonnington, C. Chu, B. Draganski, R. I. Scahill,J. D. Rohrer, N. C. Fox, C. R. Jack, J. Ashburner, and R. S. Frackowiak,“Automatic classification of MR scans in Alzheimer’s disease,” Brain,vol. 131, no. 3, pp. 681–689, 2008.

[21] C. Gaser, I. Nenadic, B. R. Buchsbaum, E. A. Hazlett, and M. S. Buchs-baum, “Deformation-based morphometry and its relation to conventionalvolumetry of brain lateral ventricles in MRI,” NeuroImage, vol. 13, no. 6,pp. 1140–1145, 2001.

[22] X. Hua, A. D. Leow, N. Parikshak, S. Lee, M.-C. Chiang, A. W.Toga, C. R. Jack, M. W. Weiner, and P. M. Thompson, “Tensor-basedmorphometry as a neuroimaging biomarker for Alzheimer’s disease: AnMRI study of 676 AD, MCI, and normal subjects,” NeuroImage, vol. 43,no. 3, pp. 458–469, 2008.

[23] J. Friedman, T. Hastie, and R. Tibshirani, The elements of statisticallearning. Springer series in statistics Springer, Berlin, 2001, vol. 1.

[24] J. Zhang, M. Liu, and D. Shen, “Detecting anatomical landmarks fromlimited medical imaging data using two-stage task-oriented deep neuralnetworks,” IEEE Transactions on Image Processing, vol. 26, no. 10, pp.4753–4764, 2017.

[25] C. R. Jack, M. A. Bernstein, N. C. Fox, P. Thompson, G. Alexander,D. Harvey, B. Borowski, P. J. Britson, J. L Whitwell, C. Ward et al., “TheAlzheimer’s disease neuroimaging initiative (ADNI): MRI methods,”Journal of Magnetic Resonance Imaging, vol. 27, no. 4, pp. 685–691,2008.

[26] J. G. Sled, A. P. Zijdenbos, and A. C. Evans, “A nonparametric methodfor automatic correction of intensity nonuniformity in MRI data,” IEEETransactions on Medical Imaging, vol. 17, no. 1, pp. 87–97, 1998.

[27] Y. Wang, J. Nie, P.-T. Yap, F. Shi, L. Guo, and D. Shen, “Ro-bust deformable-surface-based skull-stripping for large-scale studies,”in Medical Image Computing and Computer-Assisted Intervention–MICCAI 2011. Springer, 2011, pp. 635–642.

[28] J. Zhang, Y. Gao, Y. Gao, B. Munsell, and D. Shen, “Detectinganatomical landmarks for fast Alzheimer’s disease diagnosis,” IEEETransactions on Medical Imaging, vol. 35, no. 12, pp. 2524–2533, 2016.

[29] J. Zhang, M. Liu, L. An, Y. Gao, and D. Shen, “Alzheimer’s diseasediagnosis using landmark-based features from longitudinal structuralMR images,” IEEE Journal of Biomedical and Health Informatics,vol. 21, no. 6, pp. 1607–1616, 2017.

[30] C. J. Holmes, R. Hoge, L. Collins, R. Woods, A. W. Toga, andA. C. Evans, “Enhancement of MR images using registration for signalaveraging,” Journal of Computer Assisted Tomography, vol. 22, no. 2,pp. 324–333, 1998.

[31] D. Shen and C. Davatzikos, “HAMMER: Hierarchical attribute matchingmechanism for elastic registration,” IEEE Transactions on MedicalImaging, vol. 21, no. 11, pp. 1421–1439, 2002.

[32] J. Zhang, J. Liang, and H. Zhao, “Local energy pattern for texture classi-fication using self-adaptive quantization thresholds,” IEEE Transactionson Image Processing, vol. 22, no. 1, pp. 31–42, 2013.

[33] K. Mardia, “Assessment of multinormality and the robustness ofhotelling’s t2 test,” Applied Statistics, pp. 163–171, 1975.

[34] M. Liu, J. Zhang, E. Adeli, and D. Shen, “Landmark-based deep multi-instance learning for brain disease diagnosis,” Medical Image Analysis,vol. 43, pp. 157–168, 2018.

[35] A. Krizhevsky, I. Sutskever, and G. E. Hinton, “ImageNet classifica-tion with deep convolutional neural networks,” in Advances in NeuralInformation Processing Systems, 2012, pp. 1097–1105.

[36] Y. Jia, E. Shelhamer, J. Donahue, S. Karayev, J. Long, R. Girshick,S. Guadarrama, and T. Darrell, “Caffe: Convolutional architecture forfast feature embedding,” in Proceedings of ACM International Confer-ence on Multimedia. ACM, 2014, pp. 675–678.

[37] S. Boyd and L. Vandenberghe, Convex optimization. CambridgeUniversity Press, 2004.

[38] T. G. Dietterich, “Ensemble methods in machine learning,” in Interna-tional Workshop on Multiple Classifier Systems. Springer, 2000, pp.1–15.

[39] D. Zhang, Y. Wang, L. Zhou, H. Yuan, and D. Shen, “Multimodalclassification of Alzheimer’s disease and mild cognitive impairment,”NeuroImage, vol. 55, no. 3, pp. 856–867, 2011.

[40] N. Tzourio-Mazoyer, B. Landeau, D. Papathanassiou, F. Crivello,O. Etard, N. Delcroix, B. Mazoyer, and M. Joliot, “Automated anatom-ical labeling of activations in SPM using a macroscopic anatomicalparcellation of the MNI MRI single-subject brain,” NeuroImage, vol. 15,no. 1, pp. 273–289, 2002.

[41] C.-C. Chang and C.-J. Lin, “Libsvm: A library for support vectormachines,” ACM Transactions on Intelligent Systems and Technology,vol. 2, no. 3, p. 27, 2011.

[42] A. Jain, K. Nandakumar, and A. Ross, “Score normalization in mul-timodal biometric systems,” Pattern Recognition, vol. 38, no. 12, pp.2270–2285, 2005.

[43] L. v. d. Maaten and G. Hinton, “Visualizing data using t-SNE,” Journalof Machine Learning Research, vol. 9, no. Nov, pp. 2579–2605, 2008.

[44] N. Chinchor and B. Sundheim, “Muc-5 evaluation metrics,” in Proceed-ings of the 5th Conference on Message Understanding. Association forComputational Linguistics, 1993, pp. 69–78.

[45] B. W. Matthews, “Comparison of the predicted and observed secondarystructure of t4 phage lysozyme,” Biochimica et Biophysica Acta (BBA)-Protein Structure, vol. 405, no. 2, pp. 442–451, 1975.

[46] J. J. Rodriguez, L. I. Kuncheva, and C. J. Alonso, “Rotation forest: Anew classifier ensemble method,” IEEE Transactions on Pattern Analysisand Machine Intelligence, vol. 28, no. 10, pp. 1619–1630, 2006.

[47] B. T. Hyman, G. W. Van Hoesen, A. R. Damasio, and C. L. Barnes,“Alzheimer’s disease: Cell-specific pathology isolates the hippocampalformation,” Science, vol. 225, no. 4667, pp. 1168–1170, 1984.

[48] L. De Jong, K. Van der Hiele, I. Veer, J. Houwing, R. Westendorp,E. Bollen, P. De Bruin, H. Middelkoop, M. Van Buchem, and J. VanDer Grond, “Strongly reduced volumes of putamen and thalamus inAlzheimer’s disease: An MRI study,” Brain, vol. 131, no. 12, pp. 3277–3285, 2008.

[49] D. Chan, N. C. Fox, R. I. Scahill, W. R. Crum, J. L. Whitwell,G. Leschziner, A. M. Rossor, J. M. Stevens, L. Cipolotti, and M. N.Rossor, “Patterns of temporal lobe atrophy in semantic dementia andAlzheimer’s disease,” Annals of Neurology, vol. 49, no. 4, pp. 433–442,2001.

Documents

IEEE JOURNAL OF BIOMEDICAL AND HEALTH INFORMATICS 1 … · Landmark Detection Fig. 2. Illustration of the proposed anatomical Landmark based Deep Feature Learning (LDFL) framework