EE462 MLCV

1

Lecture 3-4Clustering (1hr)Gaussian Mixture and EM (1hr)

Tae-Kyun Kim

EE462 MLCV

2

2D data vectors (green) are grouped to two homogenous clusters (blue and red).Clustering is achieved by an iterative algorithm (left to right). The cluster centers are marked x.

Vector Clustering

EE462 MLCV

3

``

RGB

Pixel Clustering (Image Quantisation)Image pixels are represented by 3D vectors of R,G,B values.The vectors are grouped to K=10,3,2 clusters, and represented by the mean values of the respective clusters.

EE462 MLCV

4

dim

ensi

on

D………

……

or raw pixels

…

K codewords

Patch Clustering (BoW in Lecture 9-10) Image patches are harvested around feature points in a large number of images.They are represented by finite dimensional vectors, and clustered to form a visual dictionary.

SIFT

20

20D=400

EE462 MLCV

5……

Image ClusteringWhole images are represented as finite dimensional vectors.Homogenous vectors are grouped together in Euclidean space.

EE462 MLCV

6

K-means vs GMM

Hard clustering: a data point is assigned only one cluster.

Soft clustering: a data point is assigned multiple Gaussians probabilistically.

Two representative techniques are k-means and Gaussian Mixture Model (GMM).K-means assigns data points to the nearest clusters, while GMM assigns data to the Gaussian densities that best represent the data.

EE462 MLCV

7

Matrix and Vector Derivatives

EE462 MLCV

8

EE462 MLCV

9

K-means Clustering

EE462 MLCV

10

EE462 MLCV

11

till converge

EE462 MLCV

12

K=2

μ1

μ2

rnk

EE462 MLCV

13

Convergence proof (yes)Global minimum (no)

EE462 MLCV

14

V=V=

EE462 MLCV

15

Statistical Pattern Recognition Toolbox for Matlab

http://cmp.felk.cvut.cz/cmp/software/stprtool/

…\stprtool\probab\cmeans.m…\stprtool\probab\cmeans_tk.m

EE462 MLCV

16

Mixture of Gaussians

EE462 MLCV

17

EE462 MLCV

18

EE462 MLCV

19

EE462 MLCV

20

Maximum Likelihood

s.t.

EE462 MLCV

21

EE462 MLCV

22

objective ftn. f(x)constraints g(x)

max f(x) s.t. g(x)=0 max f(x) + g(x)

http://en.wikipedia.org/wiki/Lagrange_multiplier

EE462 MLCV

23

EE462 MLCV

24

till converge

EE462 MLCV

25

EE462 MLCV

26

EE462 MLCV

27

Statistical Pattern Recognition Toolbox for Matlab

http://cmp.felk.cvut.cz/cmp/software/stprtool/

…\stprtool\visual\pgmm.m…\stprtool\demos\demo_emgmm.m

EE462 MLCV

28

Supplementary Material

EE462 MLCV

29

Information Theory (for Lecture 7-8)

EE462 MLCV

30

EE462 MLCV

31

Advanced topic (optional)

http://www.iis.ee.ic.ac.uk/~tkkim/mlcv/lecture_clustering_em.pdf

EE462 MLCV

32

EM Algorithm in General

EE462 MLCV

33

EE462 MLCV

34