1

Heat Diffusion Classifier on a Graph

Haixuan Yang, Irwin King, Michael R. Lyu The Chinese University of Hong Kong

Group Meeting2006

2

Introduction

Heat Diffusion Model on a Graph

Three Graph Inputs

Connections with Other Models

Experiments

Conclusions and Future Work

Outline

3

IntroductionKondor & Lafferty (NIPS2002)

Construct a diffusion kernel on a graphApply to a large margin classifier

Lafferty & Kondor (JMLR2005)Construct a diffusion kernel on a special manifoldApply to SVM

Belkin & Niyogi (Neural Computation 2003)Reduce dimension by heat kernel and local distance

Tenenbaum et al (Science 2000) Reduce dimension by local distance

4

Introduction

The ideas we inherit Local information

relatively accurate in a nonlinear manifold. Heat diffusion on a manifold

a generalization of the Gaussian density from Euclidean space to manifold.

heat diffuses in the same way as Gaussian density in the ideal case when the manifold is the Euclidean space.

The ideas we think differently Establish the heat diffusion equation directly on a graph

three proposed candidate graphs. Construct a classifier by the solution directly.

5

Heat Diffusion Model on a Graph

Notations

. of period a during neighbor its from receives

, at timeheat that ofamount :)(

. at time nodeat heat the:)(

matrix. weight the:))((

}. to from edgean is thereif :){(

}{

wheregraph, weighteddirectedgiven a :)(

Δtji

ti,j,t,Δ,M

tii,tf

i,jwW=

jii,jE=

.1,2,...,nV

V,E,WG

6

Heat Diffusion Model on a Graph

Assumptions

. differenceheat the toalproportion is ) ( ,t)f(j,t)-f(iΔti,j,t,M

before. described as spaceEuclidean in the

doesit as way same in the to from flowsheat The ij

. period time the toalproportion is ) ( ΔtΔti,j,t,M

Δtt(t)-ff ααΔti,j,t,M ijβ

w-

2ij

))(() (

7

Heat Diffusion Model on a Graph

Solution

Eijij

β

w-

ii

ii

Δtt(t)-ffαetfΔttf

tfΔttf2ij

),(

))(()()(

:as expressed becan )( and )( differenceheat The

)()()(

:formmatrix a as expressed becan It

tαHfΔt

t-fΔttf

)0()0()()()(

:becomesequation above the,0Let

fefetftαHftfdt

d

Δt

HtH

8

Heat Diffusion Model on a Graph

Three candidate graphs KNN Graph

Connect points j and i from j to i if j is one of the K nearest neighbors of i, measured by the Euclidean distance.

SKNN-Graph Choose the smallest K*n/2 undirected edges,

which amounts to K*n directed edges. Minimum Spanning Tree

Choose the subgraph such that It is a tree connecting all vertices; the sum of

weights is minimum among all such trees.

9

Heat Diffusion Model on a Graph Illustration

Manifold KNN Graph SKNN-Graph Minimum Spanning Tree

10

Heat Diffusion Model on a Graph Advantages and disadvantages

KNN Graph Democratic to each node Resulting classifier is a generalization of KNN May not be connected Long edges may exit while short edges are removed

SKNN-Graph Not democratic May not be connected Short edges are more important than long edges

Minimum Spanning Tree Not democratic Long edges may exit while short edges are removed Connection is guaranteed Less parameter Faster in training and testing

11

Heat Diffusion Classifier (HDC)

Choose a graph Compute the heat kernel Compute the heat distribution for

each class according to the initial heat distribution

Classify according to the heat distribution

12

Connections with other models The Parzen window approach (when the window

function takes the normal form) is a special case of the HDC for the KNN and SKNN graphs (whenγis small, K=n-1).

KNN is a special case of the HDC for the KNN graph (whenγis small, 1/β=0).

In Euclidean space, the proposed heat diffusion model for the KNN graph (when K is set to be 2m, 1/β=0) is a generalization of the solution deduced by Finite Difference Method.

Hopefield Model (PNAS, 1982) is the original one which determines class by looking at immediate neighbors. (Thanks to the anonymous reviewer)

13

Experiments Experimental Setup

Experimental Environments Hardware: Nix Dual Intel Xeon

2.2GHz OS: Linux Kernel 2.4.18-27smp

(RedHat 7.3) Developing tool: C

Data Description 3 artificial Data sets and 6

datasets from UCI Comparison

Algorithms: Parzen window

KNNSVM KNN-HSKNN-HMST-H

Results: average of the ten-fold cross validation

DatasetCase

sClasses Variable

Syn-1 100 2 2

Syn-2 100 2 3

Syn-3 200 2 3

Breast-w 683 2 9

Glass 214 6 9

Iono 351 2 34

Iris 150 3 4

Sonar 208 2 60

Vehicle 846 4 18

14

Experiments

Results Dataset SVM KNN PWA KNN-H MST-H SKNN-H

Syn-1 66.0 67.0 80.0 93.0 95.0 95.0

Syn-2 34.0 67.0 83.0 94.0 94.0 89.0

Syn-3 54.0 79.5 92.0 91.0 90.0 92.0

Breast-w 96.8 94.1 96.6 96.9 95.9 99.4

Glass 68.1 61.2 63.5 68.1 68.7 70.5

Iono 93.7 83.2 89.2 96.3 96.3 96.3

Iris 96 97.3 95.3 98.0 92.0 94.7

Sonar 88.5 80.3 53.9 90.9 91.8 94.7

Vehicle 84.8 63.0 66.0 65.5 83.5 66.6

15

Conclusions and Future Work

KNN-H, SKNN-H and MST-H Candidates for the Heat Diffusion Classifier on a

Graph.

Future Work Apply the asymmetric exp{γH} to SVM. Extend the current heat diffusion model further

(from inside). DiffusionRank is a generalization of PageRank