Biased Discriminant Subspace Learning for Content Based ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/... · Two-class SVM for CBIR RF One Class for CBIR RF Active SVM for CBIR

Biased Discriminant Subspace Learning

for Content Based Image Retrieval

School of Electrical & Electronic Engineering

Supervisor : Assoc Prof. Wang Lipo School of Electrical &Electronic

Engineering

Co-Supervisor: Assoc Prof. Lin Weisi School of Computer Engineering

PhD Student: Zhang Lining

Outline

• Background

• Research Object

• Existing Work

• Proposed Method

• Conclusion and Future work

Background

• Rapid growth of the number of images records and

explosive increase of on-line images

How to retrieval the

image you want?!

Background

• Text Based Image Retrieval– Google Search, Bing Search, Baidu Search

• However, text based image retrieval is enough?– Hard to describe the image using words

– Exactly search the query image in database

• Actually, a picture is worth thousands of words!!!

Background

• However, the severe challenge in image retrieval is the “semantic

gap” issue

• Dynamic Interpretation

Similar

Features

Different

Concept

Different

Features

Similar

Concept

Skee Sunset

Research Object

• Extend existing techniques

• Design new technique and try to narrow the “semantic gap” in CBIR

• Devise novel and efficient features and try to narrow the “semantic gap” in CBIR

• Devise a more reasonable and efficient framework for CBIR

Research Problem

• What is query by example?

User

Query Image

Image Database

Final Results

Key techniques in CBIR

• Image feature extraction

– Global features: color, texture ,shape…

– Local features: bag of features, SIFT, Gabor Functions…

• High dimensional indexing

– Dimension reduction: PCA, LDA, Manifold learning…

– Similarity measure: Euclidean, manifold, EMD

• Machine learning techniques

– Supervised learning: classification

– Unsupervised learning: clustering

– Semi-supervised learning:

Existing State of Art Methods for CBIR

• Support vector machines for CBIR

– Two-class SVM

– One-class SVM

– Some other machines for classification

• Feature weighting methods for CBIR

– Feature weighting

– Query movement

• Subspace learning for CBIR

– Fisher’s criterion based for discriminant subspace learning

– Manifold learning for preserving the local geometry

Support Vector Machines for CBIR

Two-class SVM for CBIR RF One Class for CBIR RF Active SVM for CBIR RF

Existing Methods

• Support Vector Machines for CBIR• P. Hong, Q. Tian, and T.S. Huang, “Incorporate Support Vector Machines to Content-Based

Image Retrieval with Relevant Feedback,” Proc. IEEE Int’l Conf. Image Processing, pp. 750-753, 2000.

• Y. Chen, X.-S. Zhou, and T.-S. Huang, “One-class SVM for learning in image retrieval,” in Proc. IEEE Int. Conf. Image Processing, 2001

• G. Guo, A.K. Jain, W. Ma, and H. Zhang, “Learning Similarity Measure for Natural Image Retrieval with Relevance Feedback,”IEEE Trans. Neural Networks, vol. 12, no. 4, pp. 811-820, 2002.

• C.-H. Hoi, C.-H. Chan, K. Huang, M. R. Lyu, and I. King, “Biased support vector machine for relevance feedback in image retrieval,” presented at the Int. Joint Conf. Neural Networks, 2004.

• D. Tao, X. Tang, X. Li, and X. Wu, “Asymmetric bagging and random subspace for support vector machines-based relevance feedback in image retrieval,” IEEE Trans. Pattern Anal. Mach. Intell., vol. 28, no.7, pp. 1088–1099, Jul. 2007

Existing Methods

• Feature reweighting• Y. Rui and T.-S. Huang, “Optimizing learning in image retrieval,” presented at the IEEE

Int. Conf. Computer Vision and Pattern Recognition,2000.

• M. L. Kherfi and D. Ziou, “Relevance feedback for CBIR: A new approach based onprobabilistic feature weighting with positive and negative examples,” IEEE Trans.Image Process., vol. 15, no. 4, pp. 1017–1030, Apr. 2006.

Feature Weighting Query Moving

Existing Methods

• Subspace Learning for CBIR• X. Zhou and T. Huang, “Small sample learning during multimedia retrieval using biasmap,” in Proc. IEEE

Int. Conf. Computer Vision and Pattern Recognition (CVPR), 2001, vol. 1, pp. 11–17.

• D. Tao, X. Tang, X. Li, and Y. Rui, “Kernel direct biased discriminant analysis: A new content-based image retrieval relevance feedback algorithm,” IEEE Trans. Multimedia, vol. 8, pp. 716–727, 2005.

• D. Xu, S. Yan, D. Tao, S. Lin, and H.-J. Zhang, “Marginal fisher analysis and its variants for human gait recognition and content-based image retrieval,” IEEE Trans. Image Process., vol. 16, no. 11, pp.2811–2821, Nov. 2007.

Biased Discriminant Analysis

• Reasonable assumption:

– “All the positive samples are alike, but each negative sample is negative in each way.”

1

( )( )xN

T

x i i

i

S x x x x

1

( )( )yN

T

y i i

i

S y x y x

As large as possible

As small as possible

|| ||arg max

|| ||

T

y

T

x

S

S

The drawbacks of the BDA

• For the intra-class compactness

• For the inter-class discriminate

• “Small Sample Size” Problem

Number of Samples << Dimensionality of Features

1

( )( )xN

T

x i i

i

S x x x x

1

( )( )yN

T

y i i

i

S y x y x

Not Reasonable

Not Reasonable

Singular MatrixT

xS XX

How to tackle the drawbacks?

• “Small Sample Size” Problem- Improved Fisher Discriminant Criterion?

• Regularization method-ill posed problem

• Null space method-discard the principle space of the Sw

• Direct method-discard the null space of the Sb

• Gaussian distribution- kernel method?

– How to tune the kernel parameters on line?

• Our Method– Instead of using Fisher Discriminant Criterion, we adopt another effective

discriminant criterion.”Differential Scatter Discriminant Criterion”

– Use the most discriminative samples to describe the inter-class dispersion

– By introducing the manifold regularization, a local smooth and consistentoutput can be learned for CBIR.

Discriminant Criterion

• Fisher Linear Discriminant Criterion

• Differential Scatter Discriminant Criterion

( )arg max

( )

T

b

T

w

tr S

tr S

* arg max ( ) ( )T

T T

b wI

tr S tr S

-K.Fukunnaga -1991

-Tao. -2007

Differential Scatter Discriminant Criterion

• Maximum Scatter Difference Discriminant Criterion– Fengxi Song, David Zhang, Dayong Mei, Zhong Wei Guo, A Multiple Maximum Scatter Difference

Discriminant Criterion For Facial Feature Extraction, IEEE Trans. SMC, Vol.37,No.6, pp.1599-1606.

December 2007.

• Maximum margin Criterion– H. Li, T. Jiang, and K. Zhang, Efficient and robust feature extraction by maximum margin criterion,

IEEE Trans. Neural Network., Vol. 17, No. 1, pp. 157–165, Jan. 2006.

• General tensor discriminant analysis– Dacheng Tao, Xuelong Li, Xindong Wu, Stephen J. Maybank, General Tensor Discriminant Analysis

and Gabor Features for Gait Recognition, IEEE Trans. Pattern Anal. Mach. Intell., Vol. 29, No. 10,

pp.1700-1714, October. 2007.

• Discriminant Locally Linear Embedding With High-Order Tensor

Data– Xuelong Li, Stemhen Lin, Shuicheng Yan, Dong Xu, Discriminant Locally Linear Embedding

With High-Order Tensor Data. IEEE Trans. SMC, Vol.38,No.2, April 2008.

Our Method: Separation(part 1)

• Inter-Class dispersion

• Intra-Class compactness

• Differential Biased Discriminant Analysis

1 2

2 21 1

1

1 ( ) 1 ( )

T

U

( ) arg max( || || || || )

arg max [ X(D -U)X ]

p nN Np n n p

i i i iN N

i j N i i j N i

T

J y y y y

trace

3

212

1 1

p p

v

( ) arg min || ||

arg min [ X (D -V)X ]

p pN Np p

i iN

i j

T T

J y y

trace

1 1 2( ) ( ) ( )sJ J J

Illustration for Biased Discriminant Analysis

Negative

Postive

The distribution of the positive samples

• What is the distribution of the positive samples in high dimensional space?

• We should integrate the geometry of samples features to the seperation

[Roweis et.al Science 2000] [H. Sebastian et.al Science 2000]

Existing manifold learning techniques

• ISOMAP

• Locally Linear Embedding

• Neighbor Preserving Embedding

• Local Preserving Projection

• Laplacian Eigenmaps

• Marginal Fisher Analysis

Manifold Learning References

• Sam T. Roweis, and Lawrence K. Saul, “Nonlinear Dimensionality Reduction by Locally Linear

Embedding”, Science, vol 290, 22 December 2000.

• J. Tenenbaum, V. Silva, and J. Langford, “A Global Geometric Framework for Nonlinear

Dimensionality Reduction,” Science, vol. 290, no. 22, pp. 2319-2323, Dec. 2000.

• H. Sebastian Seung and Daniel D. Lee, “The Manifold Ways of Perception”, Science, vol 290, 22

December 2000.

• D.L. Donoho and C. Grimes, “Hessian Eigenmaps: New Locally Linear Embedding Techniques for

High-dimensional Data,” Proc. Nat’l Academy of Sciences USA, vol. 100, no. 10, pp. 5591-5596,

2003.

• X. He and P. Niyogi, “Locality Preserving Projections,” Advances in Neural Information Processing

System, vol. 16, pp. 153-160, MIT Press, 2004.

• E. Kokiopoulou and Y. Saad, “Orthogonal Neighborhood Preserving Projections: A Projection-Based

Dimensionality ReductionTechnique,” IEEE Trans. Pattern Analysis and Machine Intelligence,vol. 29, no.

12, pp. 2143-2156, Dec. 2007.

• S. Yan, D. Xu, B. Zhang, H. Zhang, Q. Yang, and S. Lin, “Graph Embedding and Extensions: A

General Framework for Dimensionality Reduction,” IEEE Trans. Pattern Analysis and Machine

Intelligence, vol. 29, no. 1, pp. 40-51, Jan. 2007.

Our Method: Manifold Regularization(Part 2)

• Manifold Regularization

Notes: The definition is similar to Laplacian Eigenmaps or Local

Preserving Projection in preserving the local geometry in high dimensional

space.

Minimizing the objective will encourage the consistent output for the

positive samples

• Integrate the two parts together, we have the GBDA

3

21

3

1 ( )

( ) arg min( || || )

pNp p

i j ijN

i j S i

J y y w

1 1 2 2 3( ) ( ) ( ) ( )J J J J

The advantage of the GBDA

• Avoid the “Small Sample Size” Problem

– Transform the trace ratio problem to a trace difference problem

• Integrate the local geometry information of the positive class

– Manifold regularization

• Share the well-known assumption in CBIR

– All the positive samples are alike, each negative sample is

negative in its own way

The proposed framework for CBIR

Feedback

Query Image

Final Results

Visual Features Relevance Feedback Model

Refine

Retrieval

Image Database

Feature Description

• Color: hue, saturation value color histograms and color moments to

form 256+9 dimensional features

• Local Image Descriptor: Weber Local Descriptor(WLD)-(Jie Chen

et.al TPAMI 2009), 240 dimensional features

• Shape: Edge Direction Histogram(EDH) 5 dimensional features

Image Color Histogram Local Descriptor Edge Direction 510-D feature

Performance Evaluation

• Real World Database Comprising 10768 Images with 80 concepts

Corel Image ExamplesLion Fish

Castle

Bus

CarMountain

Experiments

• Comparing Algorithms

– Fisher Criterion

• BDA ( with regularization method)

• MBA ( with regularization method)

• DBDA (with direct method)

– Support Vector Machine based methods

• SVM

• CSVM

Experiments

• Large numbers of experiments based on Corel Image Database.

– 300 random independent experiments based on 10768 Corel

images with 80 concepts.

• Experiments Parameters

– top 20 images in the retrieval results are be considered

• the first 5 relevant images are marked as positive samples

• the first 5 irrelevant images are marked as negative samples

• tuning parameter 1 1

2 100 (base a series of experiments)

Sensitivity of k inter-class samples

• The relationship of different k values and average precision

Top10 Top30Top 50

3 4 5 6 7 8 9 10 11 12 130

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

K-Nearest Neighbor

Avera

ge P

recis

ion

Average Precision in Top 10 Results

the 3rd feedback

the 5th feedback

the 7th feedback

the 9th feedback

3 4 5 6 7 8 9 10 11 12 130

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

K-Nearest Neighbor

Avera

ge P

recis

ion


the 3rd feedback

the 5th feedback

the 7th feedback

the 9th feedback

3 4 5 6 7 8 9 10 11 12 130

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

K-Nearest Neighbor

Avera

ge P

recis

ion


the 3rd feedback

the 5th feedback

the 7th feedback

the 9th feedback

Performance Evaluation

• Average Precision(300 independent experiments)Top10

Top60

0 1 2 3 4 5 6 7 8 9

0.4

0.5

0.6

0.7

0.8

0.9

1

Number of Iterations

Avera

ge P

recis

ion

Average Precision in Top10 Results

GBDA

BDA

MBA

DBDA

SVM

CSVM0 1 2 3 4 5 6 7 8 9

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7

0.75


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65

0.7


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55

0.6

0.65


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.2

0.25

0.3

0.35

0.4

0.45

0.5

0.55


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 9

0.2

0.25

0.3

0.35

0.4

0.45

0.5


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 9

0.2

0.25

0.3

0.35

0.4

0.45

0.5


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 9

0.2

0.25

0.3

0.35

0.4

0.45

0.5


Avera

ge P

recis

ion


GBDA

BDA

MBA

DBDA

SVM

CSVM

Top 20 Top 30

Top 40Top 50 Top 60

Top 70 Top 80 Top 90

Performance Evaluation• Standard deviation of 300 independent experiments

Top 10 Top 20 Top 30

0 1 2 3 4 5 6 7 8 90.16

0.18

0.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34


Sta

ndard

Devia

tion

Standard Deviation in Top10 Results

GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.24

0.26

0.28

0.3

0.32

0.34

0.36


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.22

0.24

0.26

0.28

0.3

0.32

0.34

0.36


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.2

0.22

0.24

0.26

0.28

0.3

0.32

0.34


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.18

0.2

0.22

0.24

0.26

0.28

0.3

0.32


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.18

0.2

0.22

0.24

0.26

0.28

0.3


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 9

0.16

0.18

0.2

0.22

0.24

0.26

0.28

0.3


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 90.16

0.18

0.2

0.22

0.24

0.26

0.28

0.3


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

0 1 2 3 4 5 6 7 8 9

0.16

0.18

0.2

0.22

0.24

0.26


Sta

ndard

Devia

tion


GBDA

BDA

MBA

DBDA

SVM

CSVM

Top 40 Top50Top 60

Top70 Top 80 Top 90

Conclusion

• Based on the Differential Scatter Discriminant Criterion, we

introduce a differential biased discriminant algorithm. By integrating

the manifold regularization, a generalized biased discriminant

analysis is proposed for CBIR.

– Advantage:

• Never meet the small sample size problem

• Avoid the gaussian distribution for the positive class

• Smooth transform and consistent output

– Disadvantage:

• How to tune the parameters 1 2,

Future work

• How to narrow the semantic gap?

– image representation

• traditional low-level features (not sound)

• visual attention features (human perception)

• how to select the most discriminative features

– training samples

• active subspace learning using the most

informative data samples

• How to deal with large size of image database?

– Large scale size image database(sparse solution)

– To use all the labeled image and unlabeled image samples

Visual attention model

Saliency Map [Itti et.al 1998]

Documents

Biased Discriminant Subspace Learning for Content Based ...imi.ntu.edu.sg/NewsEvents/Events/PastSeminars/... · Two-class SVM for CBIR RF One Class for CBIR RF Active SVM for CBIR