Dictionary Learning for Sparse Learning

based Image Classification

Dr. Meng Yang

Smart LV Lab, Computer Vision Institute, Shenzhen University

OUTLINE

Introduction of sparse learning

Discriminative dictionary learning

Outlook

OUTLINE

Introduction of sparse learning

Discriminative dictionary learning

Outlook

DATA REPRESENTATION

Massive High-Dimensional Data

0

0

Low-dimensional structures

SPARSE TRANSFORMATION

x is a sparse vector.

Most energy concentrated in a small number of features

SPARSE SIGNAL PROCESSING

Signal Processing

Compressive

sensing

Sparse Learning

N

M M

y

=

N f x

Measurement matrix

SPARSE NEURAL CODES

Population sparseness Lifetime sparseness

B. Willmore and D. J. Tolhurst. Characterizing the sparseness of neural codes. Network, 12:255–270, 2001

W. E. Vinje and J. L. Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. SCIENCE, 287(5456):1273–1276, 2000.

Visual Cortex

Neural

codes

Sparse Learning

SRC

0 20 40 60 80 100-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Atom index

Co

din

g c

oe

ffici

en

t va

lue

Test image Coefficient Training set corruption/occlusion

8

Coding coefficient x and residual e are sparse!

J. Wright, Y. Ma, et al. Robust face recognition via sparse representation, 2009.

1 2 3 … N

subject i subject 1… subject n subject i

1 2 3 … N

Classification criterion: Identity = argmini{ri}.

DICTIONARY LEARNING FOR CLASSIFIER

2000

Sparse HVS[1][2]

2006 2007 2013 2014 2009 2011 2012 2010

DKSVD[15] LC-KSVD[16] Task DL[17]

Pair DL[8]

Metaface[18] DLSI[19]

CS-DL[20] FDDL[21]

SVDL[22]

LatentDL[23]

KSVD[5] Ana & Syn [6] Ana-KSVD[7] A-S DL[26]

LASSO[3] CS[4]

Hybrid [27][28]

REFERENCE

1. B. Willmore and D. J. Tolhurst. Characterizing the sparseness of neural codes. Network, 12:255–270, 2001

2. W. E. Vinje and J. L. Gallant. Sparse coding and decorrelation in primary visual cortex during natural vision. SCIENCE, 287(5456):1273–1276, 2000.

3. M. Osborne, B. Presnell, and B. Turlach, “A new approach to variable selection in least squares problems,” IMA Journal of Numerical Analysis, vol. 20, 2000

4. Emmanuel Candès, Compressive Sampling. ((Int. Congress of Mathematics, 3, pp. 1433-1452, Madrid, Spain, 2006

5. M. Aharon, M. Elad, and A.M. Bruckstein, "The K-SVD: An Algorithm for Designing of Overcomplete Dictionaries for Sparse Representation", the IEEE Trans. On Signal

Processing, Vol. 54, no. 11, pp. 4311-4322, November 2006.

6. M. Elad, P. Milanfar, and R. Rubinstein, “Analysis versus synthesis in signal priors,” Inverse Problems, vol. 23, no. 3, pp. 947–968, June 2007.

7. R. Rubinstein, T. Peleg and M. Elad, Analysis K-SVD: A Dictionary-Learning Algorithm for the Analysis Sparse Model, IEEE TSP, March 2013.

8. S. Gu, L. Zhang, W. Zuo, and X. Feng, “Projective Dictionary Pair Learning for Pattern Classification,” In NIPS 2014.

9. J. Wright, A. Y. Yang, A. Ganesh, S. S. Sastry, and Y. Ma. Robust face recognition via sparse representation. IEEE PAMI, 31(2):210–227, 2009.

10. M. Yang, L. Zhang, J. Yang, and D. Zhang, “Robust sparse coding for face recognition,” in Proc. IEEE Conf. CVPR, Jun. 2011, pp. 625–632.

11. R. He, W. S. Zheng, and B. G. Hu, “Maximum correntropy criterion for robust face recognition,” IEEE Trans. PAMI, vol. 33, no. 8, pp. 1561–1576,2011

12. J. Yang, et all, Nuclear norm based matrix regression with application to face recognition with occlusion and illumination changes, 2014

13. Jianchao Yang, Kai Yu, Yihong Gong, and Thomas Huang. Linear spatial pyramid matching uisng sparse coding for image classification. CVPR, 2009

14. S.H, Gao, et al. Laplacian Sparse Coding, Hypergraph Laplacian Sparse Coding, and Applications, PAMI, 2012

15. Zhang, Q., & Li, B. X. (2010). Discriminative K-SVD for dictionary learning in face recognition. In:Proceedings of the IEEE CVPR

16. Jiang, Z. L., Lin, Z., & Davis, L. S. (2013). abel consistent K-SVD: Learning a discriminative dictionary for recognition. IEEE TPAMI,34, 533.

17. Mairal, J., Bach, F., & Ponce, J. (2012). Task-driven dictionary learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 34(4), 791–804.

18. Yang, M., Zhang, L., Yang, J., & Zhang, D. (2010c). Metaface learning for sparse representation based face recognition. In:Proceedings of the ICIP

19. Ramirez, I., Sprechmann, P., & Sapiro, G. (2010). Classification and clustering via dictionary learning with structured incoherence and shared features. In:Proceedings of

the IEEE Conference on Computer Vision and Pattern Recognition

20. Wang, H. R., Yuan, C. F., Hu, W. M., & Sun, C. Y. (2012). Supervised class-specific dictionary learning for sparse modeling in action recognition. PR,45(11), 3902–3911

21. Yang, M., Zhang, L., Feng, X. C., & Zhang, D. (2011b). Fisher discrimination dictionary learning for sparse representatio. In:Proceedings of ICCV

22. Yang, M., et al. Sparse Variation Dictionary Learning for Face Recognition with A Single Training Sample Per Person, ICCV 2013

23. Yang, M. , et al. Latent Dictionary Learning for Sparse Representation based Classification, CVPR 2014

24. Zhang, L., Yang, M. et al. Sparse Representation orCollaborative Representation: Which Helps Face Recognition? ICCV 2011

25. Yang. Meng, et al. Relaxed Collaborative Representation for Pattern Classification, CVPR 2012

26. Ron Rubinstein, Member, IEEE, and Michael Elad, Dictionary Learning for Analysis-Synthesis Thresholding, TSP, 2014

27. Zhou, N., & Fan, J. P. (2012). Learning inter-related visual dictionary for object recognition. In:Proceedings of the IEEE CVPR

28. Kong, S., & Wang, D. H. (2012).A dictionary learning approach for classification: Separating the particularity and the commonality.In: ECCV

OUTLINE

Introduction of sparse learning

Discriminative dictionary learning

Outlook

DISCRIMINATIVE DICTIONARY LEARNING

Class-specific DL

Hybrid DL

Latent DL

min y2

2 1

“The choice of the dictionary that sparsifies the signals is crucia

l for the success of this model.” M. Elad et al. Proc. IEEE 10.

?

CLASS-SPECIFIC DL

Metaface[18], DLSI[19],CS-DL[21], FDDL[20]…

FDDL

Predefined bases (e.g., wavelet, DCT) too general

Training data matrix may have a big size (e.g., SRC)

Discriminative Dictionary Learning

Discriminative sparse coding coefficients

Discriminative class-specific sub-dictionary

0 20 40 60 80 100-0.1

0

0.1

0.2

0.3

0.4

0.5

0.6

Atom index

Co

din

g c

oe

ffic

ien

t va

lue

MAIN IDEA

Sparse

Coefficient

Discrimination of representation residual and coding coefficient.

GOOD FOR

BAD FOR Fisher

criterion

SMALL within- class scatter

BIG between- class scatter

D=[D1,D2,…,Dc]

Di Xi

Xj

FDDL MODEL

A=[A1, A2, …, AK], Ai :training samples from class i.

D=[D1,D2,…,DK], Di :sub-dictionary of the ith class.

X=[X1, X2, …, XK], Xi : coding coefficients of Ai over D.

1 211,min , ,

K

i iir f

D XA D X X X

Discriminative data fidelity term

Discriminative coefficient term

2

W B Ff tr X S X S X X

2 22

1, ,Ki jji i i i i i i j iF F Fj i

r

A D X A DX A D X D X

SIMPLIFIED-FDDL MODEL

A=[A1, A2, …, AK], Ai :training samples from class i.

D=[D1,D2,…,DK], Di :sub-dictionary of the ith class.

X=[X1, X2, …, XK], Xi : coding coefficients of Ai over D.

1 21 1,min , , s.t .

K i j

i i i i iir f

0

D XA D X X X X

Discriminative data fidelity term

Discriminative coefficient term

2

i i Ff X X M

2

, , i

i i i i i Fr A D X A D X

CLASSIFICATION MODEL

Global classifier

Local classifier for i-th class

DIGIT RECOGNITION (USPS)

Algorithms FDDL SRSC REC-L REC-BL SDL-G

Error rate (%) 2.89 6.05 6.83 4.38 6.67

TDDL SDL-D DLSI KNN SVM COPAR

2.84 3.54 3.98 5.2 4.2 3.61

Learned dictionary atoms

LATENT DL

Imagenet Vehicle

general

specific

ADAPTIVE HIERARCHY DL TO LATENT DL

. . . . . .

. . . . . . . . .

LATENT DICTIONARY LEARNING

Latent dictionary

atom dm

Training

data

?

Class

1

Class

2

Class

3

Class

4

? ? ? ?

Latent vector for dm

Latent Dictionary Learning (LDL)

w1,m

w2,m

w3,m

w4,m

wj,m indicates the relationship between atom dm and jth class label.

LATENT DICTIONARY LEARNING MODEL

2 2

1 21, ,1

2

3 , , ,1 1

,

min diag

s.t. 0 , ;

, ;

C

j j j j j j FFj

C N T

j m m n l n j mj l j n m n

j mm

w w w j m

w m

D X WA D w X X X M

d d

Latent sparse representation Discriminative coefficient

Latent dictionary incoherence

The latent vector could be efficiently solved by an iterative

procedure, and there is an analytical solution in each

iteration.

LATENT CLASSIFICATION MODEL

Global classifier

Local classifier for j-th class

ACT RECOGNITION (UCF SPORTS ACTION)

OUTLINE

Introduction of sparse learning

Discriminative dictionary learning

Outlook

DICTIONARY LEARING FOR BIG DATA

Beyond small data

DICTIONARY LEARNING

Beyond shallow dictionary learning

非常感谢各位！

Question?