Embed Size (px)
Citation preview
Content-based image retrieval Content-based image retrieval and Computer-aided diagnosis and Computer-aided diagnosis
systemssystems
Paulo Mazzoncini de Azevedo Marques - PhDPaulo Mazzoncini de Azevedo Marques - PhD ([email protected])([email protected])
Science of Images and Medical Physics CenterScience of Images and Medical Physics CenterSchool of Medicine of Ribeirão PretoSchool of Medicine of Ribeirão Preto
University of São PauloUniversity of São Paulo
DIAGNOSISDIAGNOSIS
Signal Detection Theory – Decision Matrix
The Essential Physics Of Medical Imaging. Bushberg JT, Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott
Williams Wilkins, Philadelphia, USA, 2002.
DIAGNOSISDIAGNOSIS
PERFORMANCE MEASUREMENTSPERFORMANCE MEASUREMENTS
SensitivitySensitivity = TP/(TP+FN) = TPF = TP/(TP+FN) = TPF
Specificity Specificity = TN/(TN+FP) = (1-FPF)= TN/(TN+FP) = (1-FPF)
True Positive Fraction (TPF)True Positive Fraction (TPF)
TPF = TP/(TP+FN)TPF = TP/(TP+FN)
False Positive FractionFalse Positive Fraction (FPF) (FPF)
FPF = FP/(FP+TN)FPF = FP/(FP+TN)Accuracy Accuracy = (TP+TN)/(TP+TN+FP+FN)= (TP+TN)/(TP+TN+FP+FN)
ROC curves (receiver operating characteristic)
The Essential Physics of Medical Imaging. Bushberg JT, Seibert JA, Leidholdt Jr. EM, Boone JM. Lippincott
Williams Wilkins, Philadelphia, USA, 2002.
DIAGNOSISDIAGNOSIS
PERFORMANCE MEASUREMENTSPERFORMANCE MEASUREMENTS
AzCAD/CBIR
sortition
Definition:
A diagnosis made by a radiologist using the output of a computerized scheme for automated image analysis as a diagnostic aid (second opinion).
Computer-aided Diagnosis
(CAD)
K. Doi - Computerized Medical Imaging and Graphics 31 (2007) 198–211
With CAD, the performance by computers does not have to be comparable to or better than that by physicians, but needs to be complementary to that by physicians (synergy).
Nishikawa RM - Applied Radiology, Suplement November 2001:14-16
CADCAD
TYPES OF AIDTYPES OF AID Computer-aided Detection (CADe)Computer-aided Detection (CADe)
– usually confined to marking suspicious usually confined to marking suspicious structures and sections structures and sections
– Initially approved by FDA-USA in 1998 for mammographyInitially approved by FDA-USA in 1998 for mammography
CADCAD
TYPES OF AIDTYPES OF AID Computer-aided Diagnosis (CADx)Computer-aided Diagnosis (CADx)
– usually focused on to classify detected usually focused on to classify detected structures or regions (more academic). structures or regions (more academic).
CADCAD
KNOWLEDGE INVOLVEDKNOWLEDGE INVOLVED
Computer VisionComputer Vision (quantitative features (quantitative features extraction)extraction)
– Preprocessing (noise reduction and Preprocessing (noise reduction and enhancement) enhancement)
– Segmentation (regions, edges, structures)Segmentation (regions, edges, structures)– Structure/ROI Analyze (form, size and Structure/ROI Analyze (form, size and
location, texture, topology)location, texture, topology)
Artificial IntelligenceArtificial Intelligence (classification)(classification)
– Features selectionFeatures selection– ClassificationClassification
CAD- EXAMPLECAD- EXAMPLE
CAD in Orthopedic Radiology: CAD in Orthopedic Radiology: Quantitative Evaluation of Vertebral Morphometry Quantitative Evaluation of Vertebral Morphometry
Eduardo A. Ribeiro, Marcello H. Nogueira-Barbosa, Eduardo A. Ribeiro, Marcello H. Nogueira-Barbosa,
Rangaraj M. Rangayyan, Rangaraj M. Rangayyan, Paulo M. Azevedo-MarquesPaulo M. Azevedo-Marques
School of Medicine of Ribeirão Preto, University of São Paulo, School of Medicine of Ribeirão Preto, University of São Paulo,
Ribeirão Preto, São Paulo, BrazilRibeirão Preto, São Paulo, Brazil
Department of Electrical & Computer Engineering, University of Calgary, Department of Electrical & Computer Engineering, University of Calgary,
Calgary, Alberta, CanadaCalgary, Alberta, Canada
Vertebral fractures are important indicators of osteoporosis.
Insufficiency fractures of the vertebrae are usually seen as a partial collapse of the vertebral body.
Both semi-quantitative and quantitative analysis of spinal and vertebral deformities could assist in the diagnostic decision-making process and in guiding therapeutic procedures.
Grading of Grading of Vertebral Fractures Vertebral Fractures (Genant)(Genant) Genant HK et al. Journal of Bone and Mineral Research, 8:1137–1148,
1993.
Manual quantitative morphometric analysis is labor-intensive and subject to inter-observer and intra-observer variability
CAD - PipelineCAD - Pipeline
1212
Image Acquisition
(film digitization)
Vertebral Plateau Segmentation
(Gabor Filters and ANN)
Vertebral Morphometry
(vertebral height measurement)
Analysis of Vertebral Height
(rule-based classification)Genant Grading
Marking Reference PointsMarking Reference Points
Five points, P1–P5, were manually marked near the middle of the intervertebral spaces spanning the range of L1–L4 by using a pointer.
The distances between the points were calculated automatically:
D(1,2), D(2,3), D(3,4), and D(4,5).
Using 75% of each distance measure, the corresponding line joining the manually marked points was shifted in either direction along its perpendicular to create a quadrilateral for each vertebra.
SegmentationSegmentation
Segmentation is based on the detection and characterization of oriented edges using Gabor filters and classification using a neural network.
F. J. Ayres and R. M. Rangayyan. Journal of Electronic Imaging, 16(2):023007:1–12, 2007.
Each image was filtered with a bank of 180 Gabor filters (sinusoidally modulated Gaussian functions) in steps of 1 degree
Width = 4 pixels and elongation factor = 8.
For each pixel, the magnitude response and angle of the Gabor filter providing the highest output were used to compose a Gabor magnitude image and an orientation field.
F S
Result of Gabor Result of Gabor FiltersFilters
original image Gabor magnitude response
coherence image
Manual Delineation of Manual Delineation of Vertebral PlateausVertebral Plateaus
5-pixel thick lines drawn for L1-L4
Pixels in regions corresponding to L1-L4 were obtained from the original image, the Gabor magnitude response, and the coherence image for analysis using a logistic sigmoid neural network.
A leave-one-out training and testing procedure was used.
The output of the neural network for each pixel was used to label the pixel as belonging to a vertebral plateau or not.
Detection of Vertebral Plateaus
with a Neural Network
Detection of Vertebral Detection of Vertebral Plateaus Plateaus with a Neural Networkwith a Neural Network
Original image Output of neural networkManual annotation
Vertebral Vertebral MorphometryMorphometry
1919
skeletonconvex
hull
skeleton
remove
spurs
apply
skeleton
to plateaus
Measurement of Measurement of Vertebral HeightVertebral Height
Measures of height obtained for a normal
vertebral body
Measures of height obtained for an abnormal vertebral
body
Initial Results of CADInitial Results of CAD
Results of computer-aided grading of vertebral fracture using the method proposed by Genant.
Values along the main diagonal correspond to correct classification by the CAD (86%).
Content-Based Image RetrievalContent-Based Image Retrieval
CBIRCBIRDefinition:
Content-based image retrieval (CBIR), also known as query by image content (QBIC) is the application of computer vision techniques to the image retrieval problem, that is, the problem of searching for similar images in large databases.
Content-based means that the search will analyze the actual contents of the image rather than the metadata such as keywords, tags, and/or descriptions associated with the image.
Muller H. et al. International Journal of Medical Informatics (2004) 73, 1—23
CBIR FrameworkCBIR Framework
Features Features
ExtractionExtraction
Query
by Similarity
Module
SimilarSimilar
ImagesImages
Query ImageQuery Image
Features of the Features of the query objectquery object
Similar Features + Similar Features + distances + Images distances + Images
IDID
ID of retrieved ID of retrieved featuresfeatures
...
Color
texture
shape
Extracted Features
feedbackfeedbackIndexingIndexing
structurestructure
query object
query object
Image Processing TechniquesImage Processing Techniques– Feature ExtractionFeature Extraction
Feature Vector (based on shape, Feature Vector (based on shape, texture, color or others techniques) texture, color or others techniques)
X1X2...
XNOriginal Image
Feature Extraction
Feature Vector
Computer VisionComputer Vision
Similarity Searches Similarity Searches Data DomainsData Domains
– MAM-Metric Access MAM-Metric Access MethodsMethods Multi-dimensional DomainsMulti-dimensional Domains Adimensional DomainsAdimensional Domains
– Fingerprints, words and so on.Fingerprints, words and so on. ExampleExample
– mvp-treemvp-tree, , vp-treevp-tree, , M-treeM-tree, , Slim-Slim-TreeTree
SLIM-TREE armazenando 17 objetosSLIMSLIM--TREE armazenando 17 objetosTREE armazenando 17 objetos
(query by example)(query by example)
Metric SpaceMetric Space is a pair: is a pair: M=(M=(DD,d),d) where: where:– DD is the characteristic domain of objects is the characteristic domain of objects – dd is a metric distance function. is a metric distance function.
Properties of the distance function Properties of the distance function d()d()::– symmetry: symmetry:
d(x,y) = d(y,x)d(x,y) = d(y,x)– non-negativity: non-negativity:
0 < d(x,y) < 0 < d(x,y) < , , x x y y e e d(x,x) = 0d(x,x) = 0– triangle inequality: triangle inequality:
d(x,y) d(x,y) d(x,z) + d(z,y) d(x,z) + d(z,y) WhereWhere x, y x, y ee z z are objects of are objects of DD
Similarity Searches Similarity Searches MetricMetric SpaceSpace
Minkowski Minkowski FunctionFunction
Range QueryRange Query::““Find all the images that are Find all the images that are
within 10 units of distance within 10 units of distance from image1.from image1.””
Similarity Searches Similarity Searches Query Definitions Query Definitions
Nearest Neighbor Query (k-Nearest Neighbor Query (k-NNNN):):
"Find the 5 nearest images to image1"Find the 5 nearest images to image1””
CBIRCBIRPERFORMANCE PERFORMANCE
MEASUREMENTSMEASUREMENTS
Precision X Recall curvesPrecision X Recall curves
0.75
0.8
0.85
0.9
0.95
1
0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
f) 0.5% of database
Recall
MRHead500
0.75
0.8
0.85
0.9
0.95
1
0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
e) 1% of database
Recall
MRHead500
0.8
0.82
0.84
0.86
0.88
0.9
0.92
0.94
0.96
0.98
1
0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
d) 2% of database
Recall
MRHead500
0.82
0.84
0.86
0.88
0.9
0.92
0.94
0.96
0.98
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
c) 5% of database
Recall
MRHead500
0.84
0.86
0.88
0.9
0.92
0.94
0.96
0.98
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
b) 10% of database
Recall
MRHead500
0.88
0.9
0.92
0.94
0.96
0.98
1
0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1
a) 15% of database
MRHead500
Recall
CBIR- EXAMPLECBIR- EXAMPLE
Content-based retrieval of color images of Content-based retrieval of color images of dermatological ulcers.dermatological ulcers.
Silvio Moreto Pereira, Marco Andrey C. Frade, Silvio Moreto Pereira, Marco Andrey C. Frade,
Rangaraj M. Rangayyan, Rangaraj M. Rangayyan, Paulo M. Azevedo-MarquesPaulo M. Azevedo-Marques
School of Medicine of Ribeirão Preto, University of São Paulo, School of Medicine of Ribeirão Preto, University of São Paulo,
Ribeirão Preto, São Paulo, BrazilRibeirão Preto, São Paulo, Brazil
Department of Electrical & Computer Engineering, University of Calgary, Department of Electrical & Computer Engineering, University of Calgary,
Calgary, Alberta, CanadaCalgary, Alberta, Canada
Dermatological Ulcers
Ulcers may appear on the legs due to chronic diseases such as diabetes and venous insufficiency.
Visual assessment of pathological regions and evaluation of macroscopic features are used for the diagnosis of skin lesions in clinical practice.
The appearance of a lesion provides important clues regarding the diagnosis, severity, and prognosis.
The red-yellow-black-white (RYKW) model of tissue composition is useful as a descriptive tool.
Ulcer Tissue Types
Granulation (red) Fibrin (yellow)
Scar or necrosis (black) Mixed
Imaging of Ulcers
3232
Representation of
Color ImagesEach color image was represented using the standard representations as
• [red, green, blue] or RGB,
• [hue, saturation, intensity] or HSI, and
• L*u*v* .
Segmentation of
Ulcer Images
Black regions Ulcer regions
Red regions Yellow regions
Original image Hue-saturation
histogram
S>0.4 and
H 300º to 0 to 30º
S>0.2 and
H 30º to 90º
S<0.2 and
I<0.25*max
Features ExtractionMultispectral cooccurrence matrices (CCMs) obtained from the
RGB, HS, u*v*, and a*b* components.
a total of 111 statistical features were extracted from the R, G, B, H, S, u*, v*, a*, and b* components to characterize each color image
KNN Based Retrieval using Cosine Distance
High-Speed
Network
HIS/MIS
Archive
Imaging
Modality
Web-based RIS/PACS/EMR
RIS
Visualization
Workstation
HL-7
DICOM
Firewall
PACS
DB
DICOM
RAID
Speech Recognition
PACS – Picture Archiving and Communication System
CAD-CBIR/PACS INTEGRATIONCAD-CBIR/PACS INTEGRATION
PACS AND IMAGING INFORMATICS: Basic Principles and
Applications - H.K. Huang, New Jersey - USA, 2004
Example of CAD/PACS integration framework:
– Communication services (DICOM functionalities)
– Image-processing pipeline (CAD-CBIR server)
Azevedo Marques PM et. al. International Journal of Computer Assisted Radiology and Surgery. 2009, v. 4. p. S-180-S-181.
Example of CAD-PACS integration cores.put("normal", Color.WHITE);
cores.put("ground-glass", Color.BLUE);
cores.put("reticular-linear", Color.GREEN);
cores.put("micronodules", Color.RED);
cores.put("honeycombing", Color.YELLOW);
cores.put("emphysematous", Color.MAGENTA);
cores.put("consolidation", Color.CYAN);
Azevedo Marques PM, et. al. International Journal of Computer Assisted Radiology and Surgery. 2009, v. 4. p. S-180-S-181.
CAD scheme using CBIR approach.CAD scheme using CBIR approach.
Example of applying a CAD scheme using CBIR approach to detect and classify a suspicious breast mass region. A suspicious mass is automatically detected by CAD scheme and queried by the observer (pointed by the arrow). In CAD workstation, the mass region segmentation (boundary contour), 12 CBIR-selected similar ROIs, and both detection and classification scores are displayed. Among the 12 similar ROIs, 8 depict malignant masses (marked by Red frame), 2 depict benign masses (marked by Green frame), and 2 depict CAD-cued false-positive regions (marked by Blue frame).
Bin Zheng. Computer-Aided Diagnosis in Mammography Using Content based Image Retrieval Approaches: Current Status and Future Perspectives.
Algorithms. 2009 June 1; 2(2): 828–849.
Example of CAD/CBIR-PACS integration
CONCLUSIONCONCLUSION
Computer-aided diagnosis has become a part of clinical work in the Computer-aided diagnosis has become a part of clinical work in the detection of breast cancer by use of mammograms, but is still in detection of breast cancer by use of mammograms, but is still in the infancy of its full potential for applications to many different the infancy of its full potential for applications to many different types of lesions obtained with various modalities.types of lesions obtained with various modalities.
Content-based image retrieval is an alternative and complementary Content-based image retrieval is an alternative and complementary approach for image retrieval based on key-words and metadata. approach for image retrieval based on key-words and metadata. Initial results are very promising about using CBIR as a Initial results are very promising about using CBIR as a diagnostic support tooldiagnostic support tool
In the future, it is likely that CAD and CBIR schemes will be In the future, it is likely that CAD and CBIR schemes will be incorporated into PACS incorporated into PACS
CAD and CBIR will be employed as useful tools for diagnostic CAD and CBIR will be employed as useful tools for diagnostic examinations in daily clinical work.examinations in daily clinical work.
