Upload
oswald-lambert
View
228
Download
5
Tags:
Embed Size (px)
Citation preview
A Lightweight A Lightweight Image Retrieval Image Retrieval
System for System for PaintingsPaintings
T. Lombardi, S. Cha, and C. T. Lombardi, S. Cha, and C. Tappert Tappert
January 19th, 2005January 19th, 2005
Electronic Imaging 2005
IntroductionIntroductionStudents of art history learnStudents of art history learn
three primary skills:three primary skills:
Formal analysisFormal analysis ComparisonComparison ClassificationClassification
How can computer science How can computer science
contribute to the contribute to the developmentdevelopment
of these skills?of these skills? Figure 1: Girl with a Pearl Earring, Jan Vermeer, 1665
Electronic Imaging 2005
Working HypothesisWorking Hypothesis
An Interactive Indexing and Image An Interactive Indexing and Image Retrieval System (IIR) for fine-art Retrieval System (IIR) for fine-art paintings can aid students in these paintings can aid students in these endeavors by providing:endeavors by providing: a mathematical summarization of an imagea mathematical summarization of an image a measurable basis for comparing two a measurable basis for comparing two
imagesimages an elementary way to classify an image an elementary way to classify an image
relative to those in a database relative to those in a database
Electronic Imaging 2005
Previous WorkPrevious WorkWe synthesize the goals of two research areas:We synthesize the goals of two research areas:
Classification of paintings:Classification of paintings: R. Sablatnig, P. Kammerer, and E. Zolda, “Hierarchical Classification of
Paintings Using Face- and Brush Stroke Models”, in Proc. of the 14th International Conference on Pattern Recognition (1998).
D. Keren, “Painter Identification Using Local Features and Naïve Bayes”, in Proc. of the 16th International Conference on Pattern Recognition (2002).
Image retrieval which aims to bridge the semantic gap:Image retrieval which aims to bridge the semantic gap: J. Corridoni, A. Del Bimbo, and P. Pala, “Retrieval of Paintings using Effects
Induced by Color Features”, in Proc. of the International Workshop on Content-Based Access of Image and Video Databases (1998).
Can we construct a feature set that satisfies the objectives of both areas while providing analytically relevant data to students?
Electronic Imaging 2005
System OverviewSystem Overview
The system consists of two major The system consists of two major components:components:
Image Database Image Database stores images, thumbnail images, and stores images, thumbnail images, and
extracted features for later retrieval extracted features for later retrieval and analysis.and analysis.
Graphical User Interface Graphical User Interface provides interactive query capabilities provides interactive query capabilities
to the end userto the end user
Electronic Imaging 2005
Database ConstructionDatabase Construction
An XML index file stores extracted An XML index file stores extracted features and control information.features and control information.
A file system stores images and A file system stores images and thumbnail images.thumbnail images.
The open design of the database The open design of the database contributes to the goals of ease of contributes to the goals of ease of use and exchange of information.use and exchange of information.
Electronic Imaging 2005
Database Construction – Database Construction – Cont.Cont.
Figure 2: XML Index File Figure 3: File System
Electronic Imaging 2005
Global Feature Global Feature ExtractionExtraction
Two different kinds of features are extracted:Two different kinds of features are extracted: Palette features Palette features
concern the set of colors in an image (color concern the set of colors in an image (color map)map)
examples: palette scopeexamples: palette scope Canvas features Canvas features
concern the spatial and frequency distribution concern the spatial and frequency distribution of colors in an image (image index)of colors in an image (image index)
examples: max, min, median, mean (for each examples: max, min, median, mean (for each color channel)color channel)
Electronic Imaging 2005
Sample Feature Set Sample Feature Set
Feature NameFeature Name Description and NotesDescription and Notes
MaxMax Max value of H, S, and V channelsMax value of H, S, and V channels
MinMin Min value of H, S, and V channelsMin value of H, S, and V channels
MeanMean Mean of H, S, and V channelsMean of H, S, and V channels
MedianMedian Median of H, S, and V channelsMedian of H, S, and V channels
Standard Dev.Standard Dev. Std of H, S, and V channelsStd of H, S, and V channels
Color EntropyColor Entropy Measures the frequency distribution of colorMeasures the frequency distribution of color
Line CountLine Count Normalized number of detected edges – Sobel Normalized number of detected edges – Sobel edge detectoredge detector
Intensity MeanIntensity Mean Arithmetic mean of values in a grayscale imageArithmetic mean of values in a grayscale image
Table 1: Sample Features used for Web Museum Interactive Test
Electronic Imaging 2005
Example: Palette ScopeExample: Palette Scope
Palette ScopePalette Scope -- the total number of unique colors -- the total number of unique colors used in an image.used in an image.
We expect Dali’s piece to have a higher palette We expect Dali’s piece to have a higher palette depth than Mondrian’s work.depth than Mondrian’s work.
Figure 4: Hallucinogenic ToreadorSalvador Dali, 1970
Figure 5: Composition with Large Blue Plane,Red, Black, Yellow, and GrayPiet Mondrian, 1921
Electronic Imaging 2005
Example: Palette Scope – Example: Palette Scope – Cont.Cont.
Formal definition of Palette Scope (U):Formal definition of Palette Scope (U):
U = C/PU = C/P
WhereWhereC=Total # of unique colors measured C=Total # of unique colors measured
in RGB or HSV triples.in RGB or HSV triples.P= Total # of pixels in an image.P= Total # of pixels in an image.
Electronic Imaging 2005
Example: Palette Scope – Example: Palette Scope – Cont.Cont.
ArtistArtist Total Pixels Total Pixels (P)(P)
Total Colors Total Colors (C)(C)
Palette Depth Palette Depth (U)(U)
MondrianMondrian 359700359700 22422242 0.006230.00623
DaliDali 165775165775 38993899 0.023510.02351
Table 2: Palette Scope statistics.
We see that Dali uses more of the color spectrum than Mondrian.
Palette depth is an important feature for artist and period style identification because many styles are defined by color, i.e. Picasso’s Blue Period and fauvism.
Electronic Imaging 2005
Graphical User InterfaceGraphical User Interface
The GUI consists of three primary The GUI consists of three primary windows for:windows for: AnalysisAnalysis ComparisonComparison ClassificationClassification
Electronic Imaging 2005
Analysis WindowAnalysis Window
Figure 6: The Analysis Window
Electronic Imaging 2005
Comparison WindowComparison Window
Figure 7: The Comparison Window
Electronic Imaging 2005
Classification WindowClassification Window
Figure 8: The Classification Window
Electronic Imaging 2005
Test ResultsTest Results
Two types of tests were conducted:Two types of tests were conducted:
Feature testsFeature tests Feature tests focus on the accuracy of Feature tests focus on the accuracy of
specific collections of features.specific collections of features. Interactive testsInteractive tests
Interactive tests assess the accuracy Interactive tests assess the accuracy of the system as a whole.of the system as a whole.
Electronic Imaging 2005
Feature TestFeature Test
Training SetTraining Set Test SetTest Set Percent CorrectPercent Correct
3636 3636 9494
200200 200200 8888
200200 200200 8383
Figure 9: Les Demoiselles d’Avignon,Pablo Picasso, 1907.
Figure 10: Road with Cypress and Star,Vincent Van Gogh, 1890.
Table 3: Feature test to distinguish the work of Picasso and Van Gogh.
Electronic Imaging 2005
Initial Interactive TestInitial Interactive Test
Database of 10 works of each of the following Database of 10 works of each of the following ten artists: ten artists:
Braque, Cezanne, De Chirico, El Greco, Braque, Cezanne, De Chirico, El Greco, Gauguin, Gauguin,
Modigliani, Mondrian, Picasso, Rembrandt, Modigliani, Mondrian, Picasso, Rembrandt, and Van and Van
Gogh.Gogh.
Training SetTraining Set Testing SetTesting Set Percent Percent CorrectCorrect
100100 9090 8181
Table 4: Initial Interactive Test
Electronic Imaging 2005
Interactive Test: Web Museum
ArtistArtist Training Training SetSet
QueriesQueries SuccessSuccess PercentPercent
AertsenAertsen 99 99 55 55.655.6
El GrecoEl Greco 1010 77 44 57.157.1
HopperHopper 1010 77 33 42.942.9
MalevichMalevich 1010 1111 88 72.772.7
MonetMonet 1010 1010 66 60.060.0
MorisotMorisot 1010 1111 77 63.663.6
RembrandtRembrandt 1010 3333 2525 75.875.8
RenoirRenoir 1010 3838 1414 36.836.8
TurnerTurner 1010 1010 44 40.040.0
VelazquezVelazquez 1010 88 88 100.0100.0
OverallOverall 500500 293293 165165 56.356.3
Table 5: Results from Web Museum Interactive Test
Electronic Imaging 2005
EvaluationEvaluation ofWeb Museum Test ResultsTest Results
Overall result: 56.3% accuracy Overall result: 56.3% accuracy 36.3% better than blind guessing (10 36.3% better than blind guessing (10
guesses/50 artists = 20%)guesses/50 artists = 20%) Dissecting the classification Dissecting the classification
mistakes reveals some intelligent mistakes reveals some intelligent mistakesmistakes Rembrandt is most often confused with Rembrandt is most often confused with
Caravaggio, Ast, and VermeerCaravaggio, Ast, and Vermeer
Electronic Imaging 2005
ConclusionsConclusions Simple palette and canvas features are Simple palette and canvas features are
sufficient for an interactive classification sufficient for an interactive classification systemsystem
A single feature set can serve for A single feature set can serve for classification and image retrieval classification and image retrieval applicationsapplications
A general feature set can adequately serve A general feature set can adequately serve for educational applicationsfor educational applications
Although showing promise, we currently Although showing promise, we currently have a low confidence system have a low confidence system
Electronic Imaging 2005
Future WorkFuture Work
Add texture featuresAdd texture features Improved color features: hue Improved color features: hue
histogramshistograms Improved distance metrics: modulo Improved distance metrics: modulo
comparison of hue histogramscomparison of hue histograms Test against larger datasetsTest against larger datasets