Multiple kernel Self-Organizing Maps

Madalina Olteanu(2), Nathalie Villa-Vialaneix(1,2), ChristineCierco-Ayrolles(1)

http://www.nathalievilla.org

nathalie.villa@univ-paris1.fr, madalina.olteanu@univ-paris1.fr,

christine.cierco@toulouse.inra.fr

Groupe de travail SAMM-Graph (25/01/2013)

(1) (2)

MK-SOM (SAMM-Graph) Nathalie Villa-Vialaneix Toulouse, 25 Janvier 2013 1 / 25

Introduction

Outline

1 Introduction

2 MK-SOM

3 Applications

Introduction

Data, notations and objectives

Data: D datasets (xdi )i=1,...,n,d=1,...,D all measured on the same individuals

or on the same objects, {i, . . . , n}, each taking values in an arbitrary spaceGd .

Examples:• (xd

i )i is p numeric variables;

• (xdi )i is n nodes of a graph;

• (xdi )i is p factors;

• (xdi )i is a text...

Purpose: Combine all datasets to obtain a map of individuals/objects:self-organizing maps for clustering {i, . . . , n} using all datasets

Introduction

or on the same objects, {i, . . . , n}, each taking values in an arbitrary spaceGd .Examples:• (xd

Introduction

or on the same objects, {i, . . . , n}, each taking values in an arbitrary spaceGd .Examples:• (xd

Introduction

Particular case [Villa-Vialaneix et al., 2013]Data: A weighted undirected network represented by a graph G with nnodes x1, . . . , xn with weight matrix W : Wij = Wji and Wii = 0.

For every node, additional information is provided: either , ,

Examples: Weight of people in a social network, Number of visits of aweb page in WWW...

Introduction

Particular case [Villa-Vialaneix et al., 2013]Data: A weighted undirected network represented by a graph G with nnodes x1, . . . , xn with weight matrix W : Wij = Wji and Wii = 0.For every node, additional information is provided: either numericalvariables, factors, textual information... or a combination of all of them.

Introduction

Particular case [Villa-Vialaneix et al., 2013]Data: A weighted undirected network represented by a graph G with nnodes x1, . . . , xn with weight matrix W : Wij = Wji and Wii = 0.For every node, additional information is provided: either numericalvariables, factors, textual information... or a combination of all of them.

Examples: Gender in a social network, Functional group of a gene in agene interaction network...

Introduction

Particular case [Villa-Vialaneix et al., 2013]Data: A weighted undirected network represented by a graph G with nnodes x1, . . . , xn with weight matrix W : Wij = Wji and Wii = 0.For every node, additional information is provided: either numericalinformation, factors, textual information... or a combination of all of them.

Introduction

Purpose

Node clustering: find communities, i.e., groups of “close” nodes in thegraph; close meaning:

• densely connected and sharing (comparatively) a few links with theother groups (“communities”);

• but also having similar labels.

Here: self-organizing map approach to produce a map of the nodes(clutering+visualization).

Multiple sources of information are handled bycombining kernels.

Introduction

Purpose

Introduction

Purpose

Introduction

Purpose

Here: self-organizing map approach to produce a map of the nodes(clutering+visualization). Multiple sources of information are handled bycombining kernels.

MK-SOM

Outline

1 Introduction

2 MK-SOM

3 Applications

MK-SOM

Basics on SOM

Project the data on a squared grid (each square of the grid is a cluster)

Project the data on a squared grid (each square of the grid is a cluster)such that:• the nodes in a same cluster are highly connected• the nodes in two close clusters are also (less) connected• the nodes in two distant clusters are (almost) not connected

MK-SOM

Basics on SOM

Project the data on a squared grid (each square of the grid is a cluster)such that:• the nodes in a same cluster are highly connected• the nodes in two close clusters are also (less) connected• the nodes in two distant clusters are (almost) not connected

MK-SOM

Basics on SOM• the map is made of neurons (visually symbolized by, e.g.,

rectangles), 1...M, each associated to a prototype pi (a prototype is a“representer” of the neuron in the original dataset);

• the map is equipped with a neighborhood relationship, i.e., a“distance” (actually a dissimilarity) between neurons, δ;

• goal: find the best mapping f(xi) ∈ {1, . . . ,M} of the observations xi

on the map minimizing the energy

E =n∑

M∑j=1

h(δ(f(xi), j))‖xi − pi‖2.

i.e., each data is assigned to a neuron so that xi is:• “close” to the prototype of f(xi);• “close” (to a lesser extent) to the prototypes of the neighboring neurons

of f(xi);• “distant” to the prototypes of the neurons that are distant of f(xi).

(topology preservation)

MK-SOM

E =n∑

M∑j=1

MK-SOM

E =n∑

M∑j=1

MK-SOM

on-line SOMData: x1, . . . , xn ∈ R

1: Initialization: randomly set p01 ,...,p0

M in Rd

2: for t = 1→ T do3: Randomly choose i ∈ {1, . . . , n}4: Assignment

f t (xi)← arg minj=1,...,M

‖xi − pt−1j ‖Rd

5: for all j = 1→ M do Representation6: pt

j ← pt−1j + µtht (δ(f t (xi), j))

(1i − pt−1

)7: end for8: end for

where ht is a decreasing function which reduces the neighborhood when tincreases and µt is generally of order 1/t .

Problem with non numerical data: definitions of ‖.‖ and pj???

MK-SOM

on-line SOMData: x1, . . . , xn ∈ R

1: Initialization: randomly set p01 ,...,p0

M in Rd

‖xi − pt−1j ‖Rd

5: for all j = 1→ M do Representation6: pt

j ← pt−1j + µtht (δ(f t (xi), j))

(1i − pt−1

where ht is a decreasing function which reduces the neighborhood when tincreases and µt is generally of order 1/t .Problem with non numerical data: definitions of ‖.‖ and pj???

MK-SOM

Kernels/Multiple kernel

What is a kernel?

(xi) ∈ G, (K(xi , xj))ij st: K(xi , xj) = K(xj , xi) and∀ (αi)i ,

∑ij αiαjK(xi , xj) ≥ 0. In this case [Aronszajn, 1950],

∃ (H , 〈., .〉) , Φ : G → H st : K(xi , xj) = 〈Φ(xi),Φ(xj)〉

Examples:• nodes of a graph: Heat kernel K = e−βL or K = L+ where L is the

Laplacian [Kondor and Lafferty, 2002, Smola and Kondor, 2003,Fouss et al., 2007]

• numerical variables: Gaussian kernel K(xi , xj) = e−β‖xi−xj‖2;

• text: [Watkins, 2000]...

MK-SOM

Kernels/Multiple kernel

What is a kernel?

(xi) ∈ G, (K(xi , xj))ij st: K(xi , xj) = K(xj , xi) and∀ (αi)i ,

∑ij αiαjK(xi , xj) ≥ 0. In this case [Aronszajn, 1950],

∃ (H , 〈., .〉) , Φ : G → H st : K(xi , xj) = 〈Φ(xi),Φ(xj)〉

Examples:• nodes of a graph: Heat kernel K = e−βL or K = L+ where L is the

Laplacian [Kondor and Lafferty, 2002, Smola and Kondor, 2003,Fouss et al., 2007]

• numerical variables: Gaussian kernel K(xi , xj) = e−β‖xi−xj‖2;

• text: [Watkins, 2000]...

MK-SOM

Kernel SOM[Mac Donald and Fyfe, 2000, Andras, 2002, Villa and Rossi, 2007](xi)i ⊂ G described by a kernel (K(xi , xi′))ii′ . Prototypes are defined in thefeature space: pj =

∑i γjiφ(xi); energy calculated calculated in H :

1: Prototypes initialization: randomly set p0j =

∑ni=1 γ

0ij Φ(xi) st γ0

ij ≥ 0and

∑i γ

0ij = 1

‖xi − pt−1j ‖H

5: for all j = 1→ M do Representation6: γt

j ← γt−1j + µtht (δ(f t (xi), j))

(1i − γ

t−1j

Usually, µt ∼ 1/t .MK-SOM (SAMM-Graph) Nathalie Villa-Vialaneix Toulouse, 25 Janvier 2013 11 / 25

MK-SOM

Kernel SOM[Mac Donald and Fyfe, 2000, Andras, 2002, Villa and Rossi, 2007](xi)i ⊂ G described by a kernel (K(xi , xi′))ii′ . Prototypes are defined in thefeature space: pj =

∑i γjiφ(xi); energy calculated calculated in H :

∑ni=1 γ

0ij Φ(xi) st γ0

ij ≥ 0and

∑i γ

0ij = 1

∑ll′γt−1

jl γt−1jl′ K t (xl , xl′) − 2

γt−1jl K t (xl , xi)

(1i − γ

t−1j

Usually, µt ∼ 1/t .MK-SOM (SAMM-Graph) Nathalie Villa-Vialaneix Toulouse, 25 Janvier 2013 11 / 25

MK-SOM

Combining kernels

Suppose: each (xdi )i is described by a kernel Kd , kernels can be

combined (e.g., [Rakotomamonjy et al., 2008] for SVM):

K =D∑

αdKd ,

with αd ≥ 0 and∑

d αd = 1.

Remark: Also useful to integrate different types of information comingfrom different kernels on the same dataset.

MK-SOM

Combining kernels

Suppose: each (xdi )i is described by a kernel Kd , kernels can be

combined (e.g., [Rakotomamonjy et al., 2008] for SVM):

K =D∑

αdKd ,

with αd ≥ 0 and∑

d αd = 1.Remark: Also useful to integrate different types of information comingfrom different kernels on the same dataset.

MK-SOM

multiple kernel SOM

∑ni=1 γ

0ji Φ(xi) st γ0

ji ≥ 0 and∑i γ

0ji = 1

2: Kernel initialization: set (α0d) st α0

d ≥ 0 and∑

d αd = 1 (e.g., α0d = 1/D);

K0 ←∑

d α0dKd

‖xi − pt−1j ‖

2H(K t )

(1i − γ

t−1j

)8: end for9: empty state ;-)

10: end for

MK-SOM

multiple kernel SOM

∑ni=1 γ

0ji Φ(xi) st γ0

ji ≥ 0 and∑i γ

0ji = 1

d ≥ 0 and∑

d αd = 1 (e.g., α0d = 1/D);

K0 ←∑

d α0dKd

∑ll′γt−1

jl γt−1jl′ K t (xl , xl′) − 2

γt−1jl K t (xl , xi)

(1i − γ

t−1j

)8: end for9: empty state ;-)

10: end for

MK-SOM

Tuning (αd)d on-linePurpose: minimize over (γji)ji and (αd)d the energy

E((γji)ji , (αd)d) =n∑

M∑j=1

h (f(xi), j)∥∥∥∥φα(xi) − pαj (γj)

∥∥∥∥2

KSOM picks up an observation xi whose contribution to the energy is:

E|xi =M∑

∥∥∥∥2

Idea: Add a gradient descent step based on the derivative of (1):

∂E|xi

∂αd=

M∑j=1

h(f(xi), j)

Kd(xdi , x

di ) − 2

n∑l=1

γjlKd(xdi , x

n∑l,l′=1

γjlγjl′Kd(xdl , x

dl′ )

h(f(xi), j)‖φ(xi) − pdj (γj)‖

MK-SOM

M∑j=1

∥∥∥∥2

E|xi =M∑

∥∥∥∥2

∂E|xi

∂αd=

M∑j=1

h(f(xi), j)

Kd(xdi , x

di ) − 2

n∑l=1

γjlKd(xdi , x

n∑l,l′=1

dl′ )

MK-SOM

M∑j=1

∥∥∥∥2

E|xi =M∑

∥∥∥∥2

∂E|xi

∂αd=

M∑j=1

h(f(xi), j)

Kd(xdi , x

di ) − 2

n∑l=1

γjlKd(xdi , x

n∑l,l′=1

dl′ )

MK-SOM

adaptive multiple kernel SOM

∑ni=1 γ

0ji Φ(xi) st γ0

ji ≥ 0 and∑i γ

0ji = 1

d ≥ 0 and∑

d αd = 1 (e.g., α0d = 1/D);

K0 ←∑

d α0dKd

‖xi − pt−1j ‖

2H(K t )

(1i − γ

t−1j

)8: end for9:

Kernel update αtd ← αt−1

d + νt∂E|xi∂αd

and K t ←∑

d αtdKd

10: end for

MK-SOM

adaptive multiple kernel SOM

∑ni=1 γ

0ji Φ(xi) st γ0

ji ≥ 0 and∑i γ

0ji = 1

d ≥ 0 and∑

d αd = 1 (e.g., α0d = 1/D);

K0 ←∑

d α0dKd

‖xi − pt−1j ‖

2H(K t )

(1i − γ

t−1j

)8: end for9: Kernel update αt

d ← αt−1d + νt

∂E|xi∂αd

and K t ←∑

d αtdKd

10: end for

Applications

Outline

1 Introduction

2 MK-SOM

3 Applications

Applications

Example1: simulated data

Graph with 200 nodes classified in 8 groups:

• graph: Erdös Reyni models: groups 1 to 4 and groups 5 to 8 withintra-group edge probability 0.3 and inter-group edge probability 0.01;

• numerical data: nodes labelled with 2-dimensional Gaussian

vectors: odd groups N((

(0.3 00 0.3

• factor with two levels: groups 1, 2, 5 and 7: first level; other groups:second level.

Only the knownledgeon the three datasetscan discriminate all 8groups.

Applications

Experiment

Kernels: graph: L+, numerical data: Gaussian kernel; factor: anotherGaussian kernel on the disjunctive recoding

Comparison: on 100 randomly generated datasets as previouslydescribed:

• multiple kernel SOM with all three data;

• kernel SOM with a single dataset;

• kernel SOM with two datasets or all three datasets in a single(Gaussian) kernel.

Applications

Experiment

Kernels: graph: L+, numerical data: Gaussian kernel; factor: anotherGaussian kernel on the disjunctive recodingComparison: on 100 randomly generated datasets as previouslydescribed:

• multiple kernel SOM with all three data;

• kernel SOM with a single dataset;

• kernel SOM with two datasets or all three datasets in a single(Gaussian) kernel.

Applications

An exampleMK-SOM All in one kernel

Graph only Numerical variables only

Applications

Numerical comparison(over 100 simulations) with mutual information∑

|Ci ∩ C̃j |

200log|Ci ∩ C̃j |

|Ci | × |C̃j |

(adjusted version, equal to 1 if partitions are identical[Danon et al., 2005]).

Applications

Numerical comparison(over 100 simulations) with mutual information∑

|Ci ∩ C̃j |

200log|Ci ∩ C̃j |

|Ci | × |C̃j |

(adjusted version, equal to 1 if partitions are identical[Danon et al., 2005]).

Applications

Example 2: Data coming from a medieval corpusExample from [Boulet et al., 2008]http://graphcomp.univ-tlse2.fr/ In the “Archive départementalesdu Lot” (Cahors, France), big corpus of 5000 transactions (mostly landcharters)

• coming from 4 “seigneuries” (about 25 little villages) in South West ofFrance;

• being established between 1240 and 1520 (just before and after thehundred years’ war).

Applications

Graph descriptionGraph:

• nodes: 1446 individuals directly involved in the transactions (1 446individuals);

• edges: the two individuals are involved in the same transaction (3 192edges).

Labels on nodes:

• numerical: average date of activity;

• text: family name of the individual.

Kernels:

• L+ for the graph;

• linear kernel for the dates;

• spectral string kernel for the family names[Karatzoglou and Feinerer, 2010] (uses common strings having alength larger than 4).

Applications

Labels on nodes:

Kernels:

Applications

Labels on nodes:

Kernels:

Applications

Date and graph maps

Applications

Name map

Applications

Conclusion

Summary• finding communities in graph, while taking into account labels;

• uses multiple kernel and automatically tunes the combination;

• the method gives relevant communities according to all sources ofinformation and a well-organized map.

Possible developments• currently studying a similar approach for dissimilarity/relational SOM;

• Main issue: computational time; currently studying a sparse version.

Thank you for your attention...

Applications

Conclusion

Applications

Conclusion

Applications

ReferencesAndras, P. (2002).Kernel-Kohonen networks.International Journal of Neural Systems, 12:117–135.

Aronszajn, N. (1950).Theory of reproducing kernels.Transactions of the American Mathematical Society, 68(3):337–404.

Boulet, R., Jouve, B., Rossi, F., and Villa, N. (2008).Batch kernel SOM and related laplacian methods for social network analysis.Neurocomputing, 71(7-9):1257–1273.

Danon, L., Diaz-Guilera, A., Duch, J., and Arenas, A. (2005).Comparing community structure identification.Journal of Statistical Mechanics, page P09008.

Fouss, F., Pirotte, A., Renders, J., and Saerens, M. (2007).Random-walk computation of similarities between nodes of a graph, with application to collaborative recommendation.IEEE Trans Knowl Data En, 19(3):355–369.

Karatzoglou, A. and Feinerer, I. (2010).Kernel-based machine learning for fast text mining in R.Comput Statist Data Anal, 54:290–297.

Kondor, R. and Lafferty, J. (2002).Diffusion kernels on graphs and other discrete structures.In Proceedings of the 19th International Conference on Machine Learning, pages 315–322.

Mac Donald, D. and Fyfe, C. (2000).The kernel self organising map.In Proceedings of 4th International Conference on knowledge-based intelligence engineering systems and applied technologies,pages 317–320.

Applications

Rakotomamonjy, A., Bach, F., Canu, S., and Grandvalet, Y. (2008).SimpleMKL.J Mach Learn Res, 9:2491–2521.

Smola, A. and Kondor, R. (2003).Kernels and regularization on graphs.In Warmuth, M. and Schölkopf, B., editors, Proceedings of the Conference on Learning Theory (COLT) and Kernel Workshop,Lecture Notes in Computer Science, pages 144–158.

Villa, N. and Rossi, F. (2007).A comparison between dissimilarity SOM and kernel SOM for clustering the vertices of a graph.In 6th International Workshop on Self-Organizing Maps (WSOM), Bielefield, Germany. Neuroinformatics Group, BielefieldUniversity.

Villa-Vialaneix, N., Olteanu, M., and Cierco-Ayrolles, C. (2013).Carte auto-organisatrice pour graphes étiquetés.In Actes des Ateliers FGG (Fouille de Grands Graphes), colloque EGC (Extraction et Gestion de Connaissances), Toulouse,France.

Watkins, C. (2000).Dynamic alignment kernels.In Smola, A., Bartlett, P., Schölkopf, B., and Schuurmans, D., editors, Advances in Large Margin Classifiers, pages 39–50,Cambridge, MA, USA. MIT P.

Multiple kernel Self-Organizing Maps

Science

Self Organizing Maps

Kohonen self organizing maps

A self-organizing maps classifier structure for brain ... · A self-organizing maps classifier structure for brain computer interfaces ... Keywords: Self-organizing map, Brain-Computer

Neural Networks: Self-Organizing Maps (SOM)

Heterogeneous data fusion with multiple kernel growing self organizing maps

Self-Organizing Maps 101 (Dhiana Deva)

Continual Learning with Self-Organizing Maps · combines standard supervised neural networks with self-organizing maps to solve the continual learning problem. The role of the self-organizing

ADAPTIVE KERNEL SELF-ORGANIZING MAPS USING …ufdcimages.uflib.ufl.edu/UF/E0/04/17/07/00001/chalasani_r.pdf · ADAPTIVE KERNEL SELF-ORGANIZING MAPS USING INFORMATION THEORETIC LEARNING

Cartogram Data Projection for Self-Organizing Maps

Boosting (Part II) and Self-Organizing Maps

Limitations of Self-organizing Maps for Vector Quantization and Multidimension…papers.nips.cc/paper/1295-limitations-of-self-organizing-maps-for... · Limitations of Self-organizing

Self Organizing Maps (SOM) Unsupervised Learning

Kohonen Self Organizing Maps Shyam Guthikonda 2

Self-organizing maps and associative memorySelf-organizing networks - introduction Most popular self-organizing network: Kohonen maps (by Teuvo Kohonen) ÆTopology preserving maps

Principal Component Analysis and Self-organizing Maps

November 24, 2009Introduction to Cognitive Science Lecture 21: Self-Organizing Maps 1 Self-Organizing Maps (Kohonen Maps) In the BPN, we used supervised

Creating Product Maps with Self-Organizing Maps - ibai Publishing

Kohonen Self-Organizing Maps - Shyam Guthikonda

Self-Organizing Maps (SOMs): Proposed RMIP Diagnosis

Self-organizing maps for virtual sensors, fault detection ...20098/FULLTEXT01.pdf · Self-organizing maps for virtual sensors, fault detection and fault ... Self-organizing maps for