Content-based image retrieval and Computer-aided diagnosis systems Paulo Mazzoncini de Azevedo...

Content-based image retrieval Content-based image retrieval and Computer-aided diagnosis and Computer-aided diagnosis

systemssystems

Paulo Mazzoncini de Azevedo Marques - PhDPaulo Mazzoncini de Azevedo Marques - PhD (pmarques@fmrp.usp.br)(pmarques@fmrp.usp.br)

Science of Images and Medical Physics CenterScience of Images and Medical Physics CenterSchool of Medicine of Ribeirão PretoSchool of Medicine of Ribeirão Preto

University of São PauloUniversity of São Paulo

DIAGNOSISDIAGNOSIS

Signal Detection Theory – Decision Matrix

The Essential Physics Of Medical Imaging. Bushberg JT, Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott

Williams Wilkins, Philadelphia, USA, 2002.

DIAGNOSISDIAGNOSIS

PERFORMANCE MEASUREMENTSPERFORMANCE MEASUREMENTS

SensitivitySensitivity = TP/(TP+FN) = TPF = TP/(TP+FN) = TPF

Specificity Specificity = TN/(TN+FP) = (1-FPF)= TN/(TN+FP) = (1-FPF)

True Positive Fraction (TPF)True Positive Fraction (TPF)

TPF = TP/(TP+FN)TPF = TP/(TP+FN)

False Positive FractionFalse Positive Fraction (FPF) (FPF)

FPF = FP/(FP+TN)FPF = FP/(FP+TN)Accuracy Accuracy = (TP+TN)/(TP+TN+FP+FN)= (TP+TN)/(TP+TN+FP+FN)

ROC curves (receiver operating characteristic)

The Essential Physics of Medical Imaging. Bushberg JT, Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott

Williams Wilkins, Philadelphia, USA, 2002.

DIAGNOSISDIAGNOSIS

PERFORMANCE MEASUREMENTSPERFORMANCE MEASUREMENTS

AzCAD/CBIR

sortition

Definition:

A diagnosis made by a radiologist using the output of a computerized scheme for automated image analysis as a diagnostic aid (second opinion).

Computer-aided Diagnosis

(CAD)

K. Doi - Computerized Medical Imaging and Graphics 31 (2007) 198–211

With CAD, the performance by computers does not have to be comparable to or better than that by physicians, but needs to be complementary to that by physicians (synergy).

Nishikawa RM - Applied Radiology, Suplement November 2001:14-16

CADCAD

TYPES OF AIDTYPES OF AID Computer-aided Detection (CADe)Computer-aided Detection (CADe)

– usually confined to marking suspicious usually confined to marking suspicious structures and sections structures and sections

– Initially approved by FDA-USA in 1998 for mammographyInitially approved by FDA-USA in 1998 for mammography

CADCAD

TYPES OF AIDTYPES OF AID Computer-aided Diagnosis (CADx)Computer-aided Diagnosis (CADx)

– usually focused on to classify detected usually focused on to classify detected structures or regions (more academic). structures or regions (more academic).

CADCAD

KNOWLEDGE INVOLVEDKNOWLEDGE INVOLVED

Computer VisionComputer Vision (quantitative features (quantitative features extraction)extraction)

– Preprocessing (noise reduction and Preprocessing (noise reduction and enhancement) enhancement)

– Segmentation (regions, edges, structures)Segmentation (regions, edges, structures)– Structure/ROI Analyze (form, size and Structure/ROI Analyze (form, size and

location, texture, topology)location, texture, topology)

Artificial IntelligenceArtificial Intelligence (classification)(classification)

– Features selectionFeatures selection– ClassificationClassification

CAD- EXAMPLECAD- EXAMPLE

CAD in Orthopedic Radiology: CAD in Orthopedic Radiology: Quantitative Evaluation of Vertebral Morphometry Quantitative Evaluation of Vertebral Morphometry

Eduardo A. Ribeiro, Marcello H. Nogueira-Barbosa, Eduardo A. Ribeiro, Marcello H. Nogueira-Barbosa,

Rangaraj M. Rangayyan, Rangaraj M. Rangayyan, Paulo M. Azevedo-MarquesPaulo M. Azevedo-Marques

School of Medicine of Ribeirão Preto, University of São Paulo, School of Medicine of Ribeirão Preto, University of São Paulo,

Ribeirão Preto, São Paulo, BrazilRibeirão Preto, São Paulo, Brazil

Department of Electrical & Computer Engineering, University of Calgary, Department of Electrical & Computer Engineering, University of Calgary,

Calgary, Alberta, CanadaCalgary, Alberta, Canada

Vertebral fractures are important indicators of osteoporosis.

Insufficiency fractures of the vertebrae are usually seen as a partial collapse of the vertebral body.

Both semi-quantitative and quantitative analysis of spinal and vertebral deformities could assist in the diagnostic decision-making process and in guiding therapeutic procedures.

Grading of Grading of Vertebral Fractures Vertebral Fractures (Genant)(Genant) Genant HK et al. Journal of Bone and Mineral Research, 8:1137–1148,

1993.

Manual quantitative morphometric analysis is labor-intensive and subject to inter-observer and intra-observer variability

CAD - PipelineCAD - Pipeline

1212

Image Acquisition

(film digitization)

Vertebral Plateau Segmentation

(Gabor Filters and ANN)

Vertebral Morphometry

(vertebral height measurement)

Analysis of Vertebral Height

(rule-based classification)Genant Grading

Marking Reference PointsMarking Reference Points

Five points, P1–P5, were manually marked near the middle of the intervertebral spaces spanning the range of L1–L4 by using a pointer.

The distances between the points were calculated automatically:

D(1,2), D(2,3), D(3,4), and D(4,5).

Using 75% of each distance measure, the corresponding line joining the manually marked points was shifted in either direction along its perpendicular to create a quadrilateral for each vertebra.

SegmentationSegmentation

Segmentation is based on the detection and characterization of oriented edges using Gabor filters and classification using a neural network.

F. J. Ayres and R. M. Rangayyan. Journal of Electronic Imaging, 16(2):023007:1–12, 2007.

Each image was filtered with a bank of 180 Gabor filters (sinusoidally modulated Gaussian functions) in steps of 1 degree

Width = 4 pixels and elongation factor = 8.

For each pixel, the magnitude response and angle of the Gabor filter providing the highest output were used to compose a Gabor magnitude image and an orientation field.

F S

Result of Gabor Result of Gabor FiltersFilters

original image Gabor magnitude response

coherence image

Manual Delineation of Manual Delineation of Vertebral PlateausVertebral Plateaus

5-pixel thick lines drawn for L1-L4

Pixels in regions corresponding to L1-L4 were obtained from the original image, the Gabor magnitude response, and the coherence image for analysis using a logistic sigmoid neural network.

A leave-one-out training and testing procedure was used.

The output of the neural network for each pixel was used to label the pixel as belonging to a vertebral plateau or not.

Detection of Vertebral Plateaus

with a Neural Network

Detection of Vertebral Detection of Vertebral Plateaus Plateaus with a Neural Networkwith a Neural Network

Original image Output of neural networkManual annotation

Vertebral Vertebral MorphometryMorphometry

1919

skeletonconvex

hull

skeleton

remove

spurs

apply

skeleton

to plateaus

Measurement of Measurement of Vertebral HeightVertebral Height

Measures of height obtained for a normal

vertebral body

Measures of height obtained for an abnormal vertebral

body

Initial Results of CADInitial Results of CAD

Results of computer-aided grading of vertebral fracture using the method proposed by Genant.

Values along the main diagonal correspond to correct classification by the CAD (86%).

Content-Based Image RetrievalContent-Based Image Retrieval

CBIRCBIRDefinition:

Content-based image retrieval (CBIR), also known as query by image content (QBIC) is the application of computer vision techniques to the image retrieval problem, that is, the problem of searching for similar images in large databases.

Content-based means that the search will analyze the actual contents of the image rather than the metadata such as keywords, tags, and/or descriptions associated with the image.

Muller H. et al. International Journal of Medical Informatics (2004) 73, 1—23

CBIR FrameworkCBIR Framework

Features Features

ExtractionExtraction

Query

by Similarity

Module

SimilarSimilar

ImagesImages

Query ImageQuery Image

Features of the Features of the query objectquery object

Similar Features + Similar Features + distances + Images distances + Images

IDID

ID of retrieved ID of retrieved featuresfeatures

...

Color

texture

shape

Extracted Features

feedbackfeedbackIndexingIndexing

structurestructure

query object

query object

Image Processing TechniquesImage Processing Techniques– Feature ExtractionFeature Extraction

Feature Vector (based on shape, Feature Vector (based on shape, texture, color or others techniques) texture, color or others techniques)

X1X2...

XNOriginal Image

Feature Extraction

Feature Vector

Computer VisionComputer Vision

Similarity Searches Similarity Searches Data DomainsData Domains

– MAM-Metric Access MAM-Metric Access MethodsMethods Multi-dimensional DomainsMulti-dimensional Domains Adimensional DomainsAdimensional Domains

– Fingerprints, words and so on.Fingerprints, words and so on. ExampleExample

– mvp-treemvp-tree, , vp-treevp-tree, , M-treeM-tree, , Slim-Slim-TreeTree

SLIM-TREE armazenando 17 objetosSLIMSLIM--TREE armazenando 17 objetosTREE armazenando 17 objetos

(query by example)(query by example)

Metric SpaceMetric Space is a pair: is a pair: M=(M=(DD,d),d) where: where:– DD is the characteristic domain of objects is the characteristic domain of objects – dd is a metric distance function. is a metric distance function.

Properties of the distance function Properties of the distance function d()d()::– symmetry: symmetry:

d(x,y) = d(y,x)d(x,y) = d(y,x)– non-negativity: non-negativity:

0 < d(x,y) < 0 < d(x,y) < , , x x y y e e d(x,x) = 0d(x,x) = 0– triangle inequality: triangle inequality:

d(x,y) d(x,y) d(x,z) + d(z,y) d(x,z) + d(z,y) WhereWhere x, y x, y ee z z are objects of are objects of DD

Similarity Searches Similarity Searches MetricMetric SpaceSpace

Minkowski Minkowski FunctionFunction

Range QueryRange Query::““Find all the images that are Find all the images that are

within 10 units of distance within 10 units of distance from image1.from image1.””

Similarity Searches Similarity Searches Query Definitions Query Definitions

Nearest Neighbor Query (k-Nearest Neighbor Query (k-NNNN):):

"Find the 5 nearest images to image1"Find the 5 nearest images to image1””

CBIRCBIRPERFORMANCE PERFORMANCE

MEASUREMENTSMEASUREMENTS

Precision X Recall curvesPrecision X Recall curves

0.75

0.8

0.85

0.9

0.95

1

0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

f) 0.5% of database

Recall

MRHead500

0.75

0.8

0.85

0.9

0.95

1

0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

e) 1% of database

Recall

MRHead500

0.8

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

d) 2% of database

Recall

MRHead500

0.82

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

c) 5% of database

Recall

MRHead500

0.84

0.86

0.88

0.9

0.92

0.94

0.96

0.98

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

b) 10% of database

Recall

MRHead500

0.88

0.9

0.92

0.94

0.96

0.98

1

0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1

a) 15% of database

MRHead500

Recall

CBIR- EXAMPLECBIR- EXAMPLE

Content-based retrieval of color images of Content-based retrieval of color images of dermatological ulcers.dermatological ulcers.

Silvio Moreto Pereira, Marco Andrey C. Frade, Silvio Moreto Pereira, Marco Andrey C. Frade,

Rangaraj M. Rangayyan, Rangaraj M. Rangayyan, Paulo M. Azevedo-MarquesPaulo M. Azevedo-Marques

School of Medicine of Ribeirão Preto, University of São Paulo, School of Medicine of Ribeirão Preto, University of São Paulo,

Ribeirão Preto, São Paulo, BrazilRibeirão Preto, São Paulo, Brazil

Department of Electrical & Computer Engineering, University of Calgary, Department of Electrical & Computer Engineering, University of Calgary,

Calgary, Alberta, CanadaCalgary, Alberta, Canada

Dermatological Ulcers

Ulcers may appear on the legs due to chronic diseases such as diabetes and venous insufficiency.

Visual assessment of pathological regions and evaluation of macroscopic features are used for the diagnosis of skin lesions in clinical practice.

The appearance of a lesion provides important clues regarding the diagnosis, severity, and prognosis.

The red-yellow-black-white (RYKW) model of tissue composition is useful as a descriptive tool.

Ulcer Tissue Types

Granulation (red) Fibrin (yellow)

Scar or necrosis (black) Mixed

Imaging of Ulcers

3232

Representation of

Color ImagesEach color image was represented using the standard representations as

• [red, green, blue] or RGB,

• [hue, saturation, intensity] or HSI, and

• L*u*v* .

Segmentation of

Ulcer Images

Black regions Ulcer regions

Red regions Yellow regions

Original image Hue-saturation

histogram

S>0.4 and

H 300º to 0 to 30º

S>0.2 and

H 30º to 90º

S<0.2 and

I<0.25*max

Features ExtractionMultispectral cooccurrence matrices (CCMs) obtained from the

RGB, HS, u*v*, and a*b* components.

a total of 111 statistical features were extracted from the R, G, B, H, S, u*, v*, a*, and b* components to characterize each color image

KNN Based Retrieval using Cosine Distance

High-Speed

Network

HIS/MIS

Archive

Imaging

Modality

Web-based RIS/PACS/EMR

RIS

Visualization

Workstation

HL-7

DICOM

Firewall

PACS

DB

DICOM

RAID

Speech Recognition

PACS – Picture Archiving and Communication System

CAD-CBIR/PACS INTEGRATIONCAD-CBIR/PACS INTEGRATION

PACS AND IMAGING INFORMATICS: Basic Principles and

Applications - H.K. Huang, New Jersey - USA, 2004

Example of CAD/PACS integration framework:

– Communication services (DICOM functionalities)

– Image-processing pipeline (CAD-CBIR server)

Azevedo Marques PM et. al. International Journal of Computer Assisted Radiology and Surgery. 2009, v. 4. p. S-180-S-181.

Example of CAD-PACS integration cores.put("normal", Color.WHITE);

cores.put("ground-glass", Color.BLUE);

cores.put("reticular-linear", Color.GREEN);

cores.put("micronodules", Color.RED);

cores.put("honeycombing", Color.YELLOW);

cores.put("emphysematous", Color.MAGENTA);

cores.put("consolidation", Color.CYAN);

Azevedo Marques PM, et. al. International Journal of Computer Assisted Radiology and Surgery. 2009, v. 4. p. S-180-S-181.

CAD scheme using CBIR approach.CAD scheme using CBIR approach.

Example of applying a CAD scheme using CBIR approach to detect and classify a suspicious breast mass region. A suspicious mass is automatically detected by CAD scheme and queried by the observer (pointed by the arrow). In CAD workstation, the mass region segmentation (boundary contour), 12 CBIR-selected similar ROIs, and both detection and classification scores are displayed. Among the 12 similar ROIs, 8 depict malignant masses (marked by Red frame), 2 depict benign masses (marked by Green frame), and 2 depict CAD-cued false-positive regions (marked by Blue frame).

Bin Zheng. Computer-Aided Diagnosis in Mammography Using Content based Image Retrieval Approaches: Current Status and Future Perspectives.

Algorithms. 2009 June 1; 2(2): 828–849.

Example of CAD/CBIR-PACS integration

CONCLUSIONCONCLUSION

Computer-aided diagnosis has become a part of clinical work in the Computer-aided diagnosis has become a part of clinical work in the detection of breast cancer by use of mammograms, but is still in detection of breast cancer by use of mammograms, but is still in the infancy of its full potential for applications to many different the infancy of its full potential for applications to many different types of lesions obtained with various modalities.types of lesions obtained with various modalities.

Content-based image retrieval is an alternative and complementary Content-based image retrieval is an alternative and complementary approach for image retrieval based on key-words and metadata. approach for image retrieval based on key-words and metadata. Initial results are very promising about using CBIR as a Initial results are very promising about using CBIR as a diagnostic support tooldiagnostic support tool

In the future, it is likely that CAD and CBIR schemes will be In the future, it is likely that CAD and CBIR schemes will be incorporated into PACS incorporated into PACS

CAD and CBIR will be employed as useful tools for diagnostic CAD and CBIR will be employed as useful tools for diagnostic examinations in daily clinical work.examinations in daily clinical work.

THANK YOU!THANK YOU!

pmarques@fmrp.usp.br