Simplifying Gaussian Mixture Models Via Entropic Quantization (EUSIPCO 2009)

Simplifying Gaussian Mixture ModelsVia Entropic Quantization

Frank Nielsen1 2, Vincent Garcia1, and Richard Nock3

1 Ecole Polytechnique (Paris, France)2 Sony Computer Science Laboratories (Tokyo, Japan)

3 Universite des Antilles et de la Guyane (Guadeloupe, France)

28th august 2009

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 1 / 23

Introduction

Plan

1 IntroductionMixture modelsProblemMixture model simplification

2 Mixture model simplificationKLD and Bregman divergenceSided BKMCSymmetric BKMCjMEF

3 ExperimentsQuality measure and initializationSided BKMCBKMC vs UTAC

4 Conclusion

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 2 / 23

Introduction Mixture models

Mixture models

Mixture model is a powerful framework to estimate PDF

Mixture model f

f (x) =n∑

i=1

αi fi (x)

where αi ≥ 0 denotes a weight with∑n

i=1 αi = 1

If f is a Gaussian mixture model (GMM),

fi (x) =1

(2π)d/2|Σi |1/2exp

(−

(x − µi )T Σ−1

i (x − µi )

2

)

with µi mean and Σi covariance matrix

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 3 / 23

Introduction Problem

Problem

−0.5 0 0.5 1 1.50

0.5

1

1.5

2

2.5

Density estimation using kernel-based Parzen estimator

Mixture models usually contain a lot of components

Estimation of statistical measures is computationally expensive

Need to reduce the number of componentsRe-lear a simpler mixture model from datasetSimplify the mixture model f

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 4 / 23

Introduction Mixture model simplification

Mixture model simplification

Given a mixture model f of n components

f (x) =n∑

i=1

αi fi (x)

Compute a mixture model g of m components

g(x) =m∑

j=1

α′jgj(x)

such as g is the best approximation of f

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 5 / 23

Mixture model simplification

Plan

1 IntroductionMixture modelsProblemMixture model simplification

2 Mixture model simplificationKLD and Bregman divergenceSided BKMCSymmetric BKMCjMEF

3 ExperimentsQuality measure and initializationSided BKMCBKMC vs UTAC

4 Conclusion

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 6 / 23

Mixture model simplification KLD and Bregman divergence

Relative entropy and Bregman divergence

The fundamental measure between statistical distributions is therelative entropy, also called the Kullback-Leibler divergence

Given fi and fj two distributions, the KLD is given by

KLD(fi ||fj) =

∫fi (x) log

fi (x)

fj(x)dx

In the case of normal distriubtions

KLD(fi ||fj) =1

2log

(det Σj

det Σi

)+

1

2tr(

Σ−1j Σi

)+

1

2(µj − µi )

T Σ−1j (µj − µi )−

d

2

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 7 / 23

Mixture model simplification KLD and Bregman divergence

Relative entropy and Bregman divergence

Nomral distributions belong to the class of exponential families

Canonical form of exponential families

f (x) = exp{〈Θ, t(x)〉 − F (Θ) + C (x)

}Estimation of the KLD by computing the Bregman divergence definedfor the log normalizer F

KLD(fi ||fj) = DF (Θj ||Θi )

where

DF (Θj ||Θi ) = F (Θj)− F (Θi )− 〈Θj − Θi ,∇F (Θi )〉

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 8 / 23

Mixture model simplification KLD and Bregman divergence

Relative entropy and Bregman divergence

For multivariate normal distributions

Sufficient statistics

t(x) = (x ,−1

2xxT )

Natural parameters

Θ = (θ,Θ) = (Σ−1µ,1

2Σ−1)

Log normalizer

F (Θ) =1

4tr(Θ−1θθT )− 1

2log det Θ +

d

2log π

∇F (Θ) =

(1

2Θ−1θ , −1

2Θ−1 − 1

4(Θ−1θ)(Θ−1θ)T

)

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 9 / 23

Mixture model simplification Sided BKMC

Bregman k-means clustering

K-means clustering

Set of points

Initialize k centroids = k classes

Repetition until convergence

Repartition step (distance)Computation of centroids (centers of mass)

Bregman K-means clustering

Set of distributions

Initialize k centroids (α′i , gi ) = GMM with k components

Repetition until convergence

Repartition step (sided Bregman divergence)Computation of centroids (sided centroids)

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 10 / 23

Mixture model simplification Sided BKMC

Sided centroids

5 multivariate Gaussians

Right-centroid

Left-centroid

http://www.sonycsl.co.jp/person/nielsen/BNCj/

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 11 / 23

Mixture model simplification Sided BKMC

Right-sided BKMC algorithm

1: Initialize the GMM g2: repeat3: Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if

DF (Θi‖Θ′j) < DF (Θi‖Θ′l), ∀l ∈ [1,m] \ {j}

4: Compute the centroids: the weight and the natural parameters of the j-thcentroid (i.e. Gaussian gj) are given by:

α′j =∑

i

αi , θ′j =

∑i αiθi∑i αi

, Θ′j =

∑i αiΘi∑

i αi

The sum∑

i is performed on i ∈ [1,m] such as fi ∈ Cj

5: until the cluster does not change between two iterations

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 12 / 23

Mixture model simplification Sided BKMC

Left-sided BKMC algorithm

1: Initialize the GMM g2: repeat3: Compute the cluster C : the Gaussian fi belongs to cluster Cj if and only if

DF (Θ′j‖Θi ) < DF (Θ′l‖Θi ), ∀l ∈ [1,m] \ {j}

4: Compute the centroids: the weight and the natural parameters of the j-thcentroid (i.e. Gaussian gj) are given by:

α′j =∑

i

αi , Θ′j = ∇F−1

(∑i

αi

α′j∇F

(Θi

))

where

∇F−1(Θ) =

(−(Θ + θθT

)−1θ , −1

2

(Θ + θθT

)−1)

5: until the cluster does not change between two iterations

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 13 / 23

Mixture model simplification Symmetric BKMC

Symmetric BKMC algorithm

Symmetric similarity measure can be required (e.g. CBIR)

Repartition step: Symmetric Bregman divergence

SDF (Θp, Θq) =DF (Θq||Θp) + DF (Θp||Θq)

2

Computation of symmetric centroid:

Compute right and left centroids (cr and cl)The symmetric centroid cs belongs to the geodesic link joining cr and cl

cλ = ∇F−1 (λ∇F (cr ) + (1− λ)∇F (cl))

The symmetric centroid cs = cλ verifies

SDF (cλ, cr ) = SDF (cλ, cl).

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 14 / 23

Mixture model simplification jMEF

jMEF

jMEF : Java library for Mixture of Exponential Families

Create and manage MEF

Simplify MEF using BKMC

Available on line at www.lix.polytechnique.fr/∼nielsen/MEF

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 15 / 23

Experiments

Plan

1 IntroductionMixture modelsProblemMixture model simplification

2 Mixture model simplificationKLD and Bregman divergenceSided BKMCSymmetric BKMCjMEF

3 ExperimentsQuality measure and initializationSided BKMCBKMC vs UTAC

4 Conclusion

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 16 / 23

Experiments Quality measure and initialization

Quality measure and initialization

Simplification quality measure

KLD(f ‖g) (right-sided)

No closed-form expression

Draw 10,000 points to estimate this KLD (Monte-Carlo)

Initial GMM f

Learnt from an image

K-means on RGB pixels ⇒ 32 classes

EM algorithm ⇒ fi

Weights αi : proportion of pixels in each class

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 17 / 23

Experiments Sided BKMC

Sided BKMC

Evolution of KLD(f ‖g) as a function of m

The simplification quality increases with m

Left-sided BKMC provides the best results

Right-sided BKMC provides the worst results

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 18 / 23

Experiments BKMC vs UTAC

BKMC vs UTAC

UTAC algorithm based on sigma points + EM algorithm

BKMC provides better results than UTAC

BKMC is faster than UTAC: 20ms vs 100ms

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 19 / 23

Experiments BKMC vs UTAC

Clustering-based image segmentation

Image f UTAC BKMC

KLD=0.23 KLD=0.11

KLD=0.16 KLD=0.13

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 20 / 23

Experiments BKMC vs UTAC

Clustering-based image segmentation

Image f UTAC BKMC

KLD=0.69 KLD=0.53

KLD=0.36 KLD=0.18

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 21 / 23

Conclusion

Plan

1 IntroductionMixture modelsProblemMixture model simplification

2 Mixture model simplificationKLD and Bregman divergenceSided BKMCSymmetric BKMCjMEF

3 ExperimentsQuality measure and initializationSided BKMCBKMC vs UTAC

4 Conclusion

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 22 / 23

Conclusion

Conclusion

GMM simplification algorithm based on k-means and Bregmandivergence

BKMC is faster and provides better results than UTAC algorithm

BKMC extends to mixtures of exponential families

jMEF available on line at www.lix.polytechnique.fr/∼nielsen/MEF

Included features:

Create/manage mixtures of exponential familiesBKMC algorithmHierarchical GMM (ACCV 2009)

V. Garcia (X, Paris, France) Simplifying GMMs 28th august 2009 23 / 23