Clustering with multiple distance metrics - multi-level ...jornsten/FMSJornsten09.pdf · distance metrics Outline Cluster Analysis Within-cluster parameteriza-tions Between-cluster

Jornsten.Clustering with

multipledistance metrics

Outline

ClusterAnalysis

Within-clusterparameteriza-tions

Between-clusterparameteriza-tions:Multi-levelmixturemodeling

Results

Conclusion andFuture work

Clustering with multiple distance metrics

- multi-level mixture models with profile

transformations.

Rebecka Jornsten

Department of Mathematical StatisticsChalmers and Gothenburg University

[email protected], http://www.stat.rutgers.edu/∼rebecka

September 16, 2009



Outline

ClusterAnalysis



Results


1 Cluster Analysis

2 Within-cluster parameterizations

3 Between-cluster parameterizations:Multi-level mixture modeling

4 Results

5 Conclusion and Future work



Outline

ClusterAnalysis



Results


Cluster analysis

General problem

Popular approach for dimension reduction

Wide range of applications: engineering, geologicaldata, social networks, high-throughput biology



Outline

ClusterAnalysis



Results


Thinking about the problems in termsof Model Selection

Clustering1 How many clusters?

2 Can we find an efficient within-cluster parameterization?(E.g., are cluster profiles flat, or increasing?)

3 Can we find an efficient between-clusterparameterization? (Do cluster profiles share a similarshape, or coincide for a subset of data dimensions?)



Outline

ClusterAnalysis



Results


Clustering gene expression data

Tasks

To group genes by similarity of expression profiles acrossexperimental conditions

Assign functions to genes - ’guilt by association’

Goals

We need an objective interpretation of cluster profiles -gene function inferences

Avoid ”Waste of parameters”



Outline

ClusterAnalysis



Results


Within-cluster parameterizations

Mathematical Formulation

For each gene g we observe a feature vector xg :

xg | Rg = k ∼ MVN(µk ,Σk),

where Rg is the cluster membership indicator.

We seek efficient/interpretable parameterizations of thecluster profiles; µk = Wθk

A sparse representation of µk is obtained if we set someof θk to 0.

The design matrix W is chosen to represent a scientificquestion of interest.

Fitting the model: EM with a modified M-step incorporatingthe θk constraints (see Jornsten and Keles, Biostatistics,2008).



Outline

ClusterAnalysis



Results


Within-cluster parameterizations

Model Selection

Searching for an optimal sparse within-cluster representation:

Classification EM (CEM) approach, see Jornsten andKeles, 2008.

Simultaneous selection via rate-distortion theory, seeJornsten, JCGS, 2009.



Outline

ClusterAnalysis



Results


Multi-level Mixture models:Between-cluster parameterizations

Cell-line data

There seem to be moreneuron-specific clusters thanglia specific

MIXL: Mixture models formulti-factor experiments

——————————————

mRNA expression time courseafter spinal cord injury

Clusters seem to share a similarshape, and differ in terms ofoffset and scale.

MIXT : Mixture models withprofile transformations.

Example 1:

11

1

−3−2

−10

12

3

Time

Log

expr

essio

n 1 11

2 22

2 2 2

3

3

3 3 33

4

4

4

4 4

4

5

55

5

5 5

6 6

6

6

6

67

77

7 7 7

8

8

8

8 88

1 2 3 1 2 3

Example 2:

1

1

1

1

1

1 2 3 4 5

−0.5

0.0

0.5

1.0

1.5

Time

Log

expr

essio

n

2

2

2

22

3

33 3

3

4

4

4 4 4

5

5

5

5

5



Outline

ClusterAnalysis



Results


MIXT - Model formulation

Multi-level structure

For each gene g we observe a feature vector xg .

Indicators Rg = k,Ug = l : cluster k and sub-clusterl ∈ {1, · · · , Lk}.

xg | Rg = k,Ug = l ∼ MVN(Aklµk + bkl ,AklΣkA′kl)

Within-cluster parameterization: µk = Wθk

Between-cluster parameterization:affine transformation (Akl , bkl)→ still Gaussian mixture.

Clustering with Multiple distance metrics

Here, consider sign-flip and/or scale transformations:µkl = Tkl(µk) = βkl(µk + αkl), Σkl = β2klΣk

Equivalent to clustering with three distance metricssimultaneously: 1− |correlation|, 1− correlation, andeuclidean distance!



Outline

ClusterAnalysis



Results


Model fitting

Fitting the model: EM with a profile M-stepincorporating the constraints.

Separate the tasks into

1 The top-level estimation problem: (µk = Wkθk ,Σk)

2 The transformation parameter estimation problem:(αkl , βkl) (where βk1 = 1, αk1 = 0,∀k).

How can we do this? Well, given (αkl , βkl), maximizingthe conditional likelihood translates to an application ofthe inverse profile transformation, and(

xgβkl− αkl

)|Rg = k ,Ug = l ∼ MVN(µk ,Σk).

Solve the aggregate problem first.



Outline

ClusterAnalysis



Results


The aggregate problem

For each cluster k , and its sub-clusters l = 1, · · · , Lk ,compute

T−1kl (xg ) =xgβkl− αkl

1 Update

Σk =

∑g ,l ηgkl(T−1kl (xg )− µk)(T−1kl (xg )− µk)′∑

g ,l ηgkl

2 Condition on Σk , solve for the constrained θk :

θk =

∑g ,l ηgkl(W ′

kΣ−1k Wk)−1W ′kΣ−1k

(T−1kl (xg )

)∑g ,l ηgkl

Note that xg contributes to the top-level parameters inmultiple ways, moderated by the sub-clustermembership probabilities.



Outline

ClusterAnalysis



Results


Estimating the transform parameters

Condition on (µk ,Σk), and consider

maxαkl ,βkl

∑g

ηgkl−1

2

(xg − βklµk − α′kl1J)′Σ−1k (xg − βklµk − α′kl1J)

β2kl

+

−J

2log(β2

kl) + c , where α′kl = βklαkl , c is a constant.

Solutions:

1 α′kl =∑

g ηgkl1′JΣ−1

k xg∑g ηgkl1

′Σ−1k 1J

− βkl1′JΣkµk

1′JΣ−1k 1J

.

2 β2kl − Aβkl − B = 0, where

A =1

J∑

g ηgkl

[1′JΣ−1

k µk

∑g ηgkl1

′JΣ−1

k xg

1′JΣ−1k 1J

−∑g

ηgklx′gΣ−1

k µk

]

B =1

J∑

g ηgkl

[∑g

ηgklx′gΣ−1

k xg −(∑

g ηgkl1′JΣ−1

k xg )2∑

g ηgkl1′JΣ−1

k 1J

]



Outline

ClusterAnalysis



Results


Making it work!

Setting up the model is the easy bit...

Starting values: The search space(K , {Lk , k = 1, · · · ,K}) is huge ...

We parameterize this as (M,K ), where M =∑

k Lk isthe total number of clusters.

1 Initialize with M kmeans clusters

2 Use PAM (robust kmeans) with 1− correlation or1− |correlation| to group the M centroids into Ktop-level clusters.

3 Add an additional random element by initializingΣk = |z | × Cov(x|Rg = k), where z ∼ N(r , s).

Search forward for M, backwards for K ∈ {M, · · · , 1}to minimize BIC.

More details in the papers (regularization, parametervalue initialization,...).



Outline

ClusterAnalysis



Results


Clustering results

BIC as a function of the number of clusters,for the standard method (K), the sign-flip (F),

and sign-flip+scale (B) models.

K

K K KK

K

K

K

K

K

K

2 4 6 8 10 12

8900

9000

9100

9200

9300

9400

9500

Number of Clusters

BIC

FF F F F F F

F F

BB

B B B B BB B

B

We select 9 clusters with an efficient between-clusterparameterization, compared to 5 using the standard mixturemodel.



Outline

ClusterAnalysis



Results


Clustering results

1

1

1

1

1

1 2 3 4 5

−0.

50.

00.

51.

01.

5

Time

Log

expr

essi

on

2

2

2

22

3

33 3

3

4

4

4 4 4

5

5

5

5

51

1

1

1

1

1 2 3 4 5

−0.

50.

00.

51.

01.

5

Time

Log

expr

essi

on

2

2

22

2

3

3

3

3

3

4

4

44

4

5

5

55

5

The MIXT with M = 5 has K = 2 profile clusters, withsub-clusters (1,2,3), where cluster 3 is a sign-flip cluster, andsub-clusters (4,5), both with positive βkl .



Outline

ClusterAnalysis



Results


Clustering resultsThe 9 cluster profiles

k = 1: (1,3,4) sign-flip of (2,5,6,7), k = 2: (8,9)

1

1

1

1

1

1 2 3 4 5

−0.5

0.0

0.5

1.0

1.5

2.0

Time

Log

expr

essi

on

2

2

22

2

3

33

33

4

4

44

4

5

5

55

5

6

6

6

6

6

7

77

77

8

8

8

8

8

99

99

9

Clusters 7,2,5,6: increased level of response - distinctfunctional gene classes in each cluster.Clusters 1,3,4: wave of early immune responseClusters 8,9: under-expression of neuron-specific genes,marker of neuron death after injury.



Outline

ClusterAnalysis



Results


Clustering results

MIXT allows for clustering via multiple distancemetrics simultaneously

Sub-clusters: Mahalanobis distanceTop-level: 1− cor or 1− |cor |.

Compare cluster profiles:

PAM(1− cor) (dash)PAM(1− |cor |) (dash-dot)MIXT top-level profilesµk , k = 1, 2 (solid).

1

1

1

1

1

1 2 3 4 5

−1.5

−1.0

−0.5

0.0

0.5

1.0

1.5

Time

log

expr

essio

n

2

2

2

2

2

2

2

2 2

2

1

1

1

1

1

1

1

1

1

1

2

2

2

2

2

With MIXT we can choose to investigate clusters ateither level of the modeling hierarchy.



Outline

ClusterAnalysis



Results


Multi-factor MIXT

xg and yg the time-course expression of mild (x) andmoderate (y) injury levels. We assume that

(xg , yg )|Rg = k ,Ug = l ∼

∼ N(

(βkl(µXk + αkl1J), δkl(µ

Yk + γkl1J)),Σkl

),

where

Σkl =

[β2klΣ

Xk βklδklΣ

XYk

βklδklΣYXk δ2klΣ

Yk

].

µYk = µX

k + ∆k = W Xk θX

k + ∆k , where ∆k is a vector ofcluster contrast parameters between the two levels of injury.This parameterization provides a sparse representation of µk

if the cluster profiles µXk ' µY

k .



Outline

ClusterAnalysis



Results


Multi-factor MIXT

The joint maximization with respect to scaletransformation parameters (αkl , βkl , γkl , δkl), given theprofile shape parameters (µk ,Σk), is complex.

We approximate this optimization by de-coupling theproblem in terms of the components (x, y), essentiallyassuming that Σk is block-diagonal.



Outline

ClusterAnalysis



Results


Clustering results

Cluster mean profiles. The left sub-panel: mild injury data.The right figure sub-panel: the moderate injury data.

1

1

1

1

1 2 3 4 5 6 7 8

−0.5

0.0

0.5

1.0

Time

Log

expr

essio

n

1

1

1

12

2

2

2

2

2

2

2

3

33 3

3

3

33

4

4 4 4

4

4

44

5

5 55

5

5

5

5

6

6 6

6

6

6

6

6

7 77

7

7

7

7 7

The MIXT model with M = 7 clusters has K = 4 profileclusters. No sign-flip clusters are detected.



Outline

ClusterAnalysis



Results


Clustering results

We detect several scale-transformation clusters ((1,2), (3,4),and (5,6)).

1

1

1

1

1 2 3 4 5 6 7 8

−0.5

0.0

0.5

1.0

Time

Log

expr

essio

n

1

1

1

12

2

2

2

2

2

2

2

3

33 3

3

3

33

4

4 4 4

4

4

44

5

5 55

5

5

5

5

6

6 6

6

6

6

6

6

7 77

7

7

7

7 7

The sub-cluster separation for the mild injury level data,|βk1 − βk2|, is smaller than the sub-cluster separation for themoderate injury level data, |δk1 − δk2|.



Outline

ClusterAnalysis



Results


Conclusions

Multi-level mixture models

Between-cluster parameterizations provide an objectiveinference tool for comparing clusters...

... and constitutes a substantial savings in terms of thenumber of model parameters.

We detect clusters that a standard model-basedclustering method may miss.

Multi-level mixture models allow for clustering usingmultiple distance metrics simultaneously.

———————————————————————–

Papers are available athttp://www.stat.rutgers.edu/∼rebecka



Outline

ClusterAnalysis



Results


Future Work

More complex experiments - need to automate subsetselection as much as possible (e.g. consider differentparameterizations simultaneously).

Time course experiments: no multi-level structure isknown a-priori. Define clusters by a unique paththrough a set of nodes. This allows clusters to coincidefor a subset of data dimensions.

d=1 d=2 d=3

•Number of nodes K(d)={2,3,2}

•Number of unique paths = 12

µ2(d=1)

µ1(d=1)µ2(d=2)

µ1(d=3)

Path Pl (bold arrows) = {k(d),d=1,…,D}={2,2,1} This path connects the shaded nodes.

µ1(d=2)

µ2(d=3)µ3(d=2)



Outline

ClusterAnalysis



Results


Acknowledgements

Prof. Ron Hart and Loyal Goff, W.M. Keck center forCollaborative Neuroscience, Rutgers

Sunduz Keles, U. Wisconsin, Madison

NSF, EPA



Outline

ClusterAnalysis



Results


Documents

Clustering with multiple distance metrics - multi-level ...jornsten/FMSJornsten09.pdf · distance metrics Outline Cluster Analysis Within-cluster parameteriza-tions Between-cluster