1 Efficient Algorithms for Non-parametric Clustering With Clutter Weng-Keen Wong Andrew Moore (In...

1

Efficient Algorithms for Non-parametric Clustering With Clutter

Weng-Keen Wong Andrew Moore

(In partial fulfillment of the speaking requirement)

2

Problems From the Physical Sciences

Minefield detection

(Dasgupta and Raftery 1998)

Earthquake faults

(Byers and Raftery 1998)

3

Problems From the Physical Sciences

(Pereira 2002) (Sloan Digital Sky Survey 2000)

4

A Simplified Example

5

Clustering with Single Linkage Clustering

ClustersSingle Linkage Clustering MST

6

Clustering with Mixture ModelsResulting ClustersMixture of Gaussians with a

Uniform Background Component

7

Clustering with CFFCuevas-Febrero-Fraiman Original Dataset

8

Related Work(Dasgupta and Raftery 98) Mixture model approach – mixture of

Gaussians for features, Poisson process for clutter

(Byers and Raftery 98) K-nearest neighbour distances for all points

modeled as a mixture of two gamma distributions, one for clutter and one for the features

Classify each data point based on which component it was most likely generated from

9

Outline

1. Introduction: Clustering and Clutter

2. The Cuevas-Febreiro-Fraiman Algorithm

3. Optimizing Step One of CFF4. Optimizing Step Two of CFF5. Results

10

The CFF Algorithm Step One

Find the highdensity datapoints

11

The CFF Algorithm Step Two Cluster the

high density points using Single Linkage Clustering

Stop when link length >

12

The CFF Algorithm

Originally intended to estimate the number of clusters

Can also be used to find clusters against a noisy background

13

Step One: Density Estimators

Finding high density points requires a density estimator

Want to make as few assumptions about underlying density as possible

Use a non-parametric density estimator

14

A Simple Non-Parametric Density Estimator

A datapoint is a highdensity datapoint if:The number of datapoints within ahypersphere of

radiush is > threshold c

15

Speeding up the Non-Parametric Density Estimator

Addressed in a separate paper (Gray and Moore 2001)

Two basic ideas:1. Use a dual tree algorithm (Gray and

Moore 2000)2. Cut search off early without computing

exact densities (Moore 2000)

16

Step Two: Euclidean Minimum Spanning Trees (EMSTs)

Traditional MST algorithms assume you are given all the distances

Implies O(N2) memory usage Want to use a Euclidean Minimum

Spanning Tree algorithm

17

Optimizing Clustering Step

Exploit recent results in computational geometry for efficient EMSTs

Involves modification to GeoMST2 algorithm by (Narasimhan et al 2000)

GeoMST2 is based on Well-Separated Pairwise Decompositions (WSPDs) (Callahan 1995)

Our optimizations gain an order of magnitude speedup, especially in higher dimensions

18

Outline for Optimizing Step Two

1. High level overview of GeoMST22. Properties of a WSPD3. How to create a WSPD4. More detailed description of GeoMST25. Our optimizations

19

Intuition behind GeoMST2

20

Intuition behind GeoMST2

21

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

Well-Separated Pairwise

Decomposition

22

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

Well-Separated Pairwise

Decomposition

Each Pair (Ai,Bi) represents a possible edge in the MST

23

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

1. Create the Well-Separated Pairwise Decomposition

2. Take the pair (Ai,Bi) that corresponds to the shortest edge

3. If the vertices of that edge are not in the same connected component, add the edge to the MST. Repeat Step 2.

24

A Well-Separated Pair (Callahan 1995)

Let A and B be point sets in d Let RA and RB be their respective bounding hyper-rectangles Define MargDistance(A,B) to be the minimum distance

between RA and RB

25

A Well-Separated Pair (Cont)The point sets A and B are considered to be well-separated if: MargDistance(A,B) max{Diam(RA),Diam(RB)}

26

Interaction ProductThe interaction product between two point

sets A and B is defined as:

A B = {{p,p’} | p A, p’ B, p p’}

27

Interaction ProductThe interaction product between two point

sets A and B is defined as:

A B = {{p,p’} | p A, p’ B, p p’}

This is the set of all distinct pairs with one element

in the pair from A and the other element from B

28

Interaction Product DefinitionThe interaction product between two point

sets A and B is defined as:

A B = {{p,p’} | p A, p’ B, p p’}

For Example:

A = {1,2,3}B = {4,5}

A B = {{1,4}, {1,5}, {2,4}, {2,5}, {3,4}, {3,5}}

29

Interaction Product

A B = {{0,1}, {0,2}, {0,3},{0,4},

{1,2}, {1,3}, {1,4},

{2,3}, {2,4},

{3,4}}

Now let A and B be the same point set ie.

A = {0,1,2,3,4} B = {0,1,2,3,4}

30

Interaction Product

A B = {{0,1}, {0,2}, {0,3}, {0,4},

{1,2}, {1,3}, {1,4},

{2,3}, {2,4},

{3,4}}

Now let A and B be the same point set ie.

A = {0,1,2,3,4} B = {0,1,2,3,4}

Think of this as all possible edges in a complete, undirected graph with {0,1,2,3,4} as the

vertices

31

A Well-Separated Pairwise Decomposition

Pair #1:

([0],[1])

Pair #2:

([0,1], [2])

Pair #3:

([0,1,2],[3,4])

Pair #4:

([3], [4])

Claim:

The set of pairs {([0],[1]), ([0,1], [2]), ([0,1,2],[3,4]),

([3], [4])} form a Well-Separated Decomposition.

32

Interaction Product Properties If P is a point set in d then a WSPD of P is a set of pairs (Ai,Bi),…,(Ak,Bk) with the following properties:

1. Ai P and Bi P for all i = 1,…,k

2. Ai Bi = for all i = 1, …, k

A = {0,1,2,3,4} B = {0,1,2,3,4}{([0],[1]), ([0,1], [2]), ([0,1,2],[3,4]), ([3], [4])} clearly satisfies Properties 1 and 2

33

Interaction Product Property 33. (Ai Bi) (Aj Bj) = for all i,j such that i j

From {([0],[1]), ([0,1], [2]), ([0,1,2],[3,4]), ([3], [4])}

we get the following interaction products:

A1 B1 = {{0,1}}

A2 B2 = {{0,2},{1,2}}

A3 B3 = {{0,3},{1,3},{2,3},{0,4},{1,4},{2,4}}

A4 B4 = {{3,4}}

These Interaction Products are all disjoint

34

Interaction Product Property 44. k

i ii BAPP1

P P = {{0,1}, {0,2}, {0,3}, {0,4}, {1,2}, {1,3}, {1,4},

{2,3}, {2,4}, {3,4}}

A1 B1 = {{0,1}}

A2 B2 = {{0,2},{1,2}}

A3 B3 = {{0,3},{1,3},{2,3},{0,4},{1,4},{2,4}}

A4 B4 = {{3,4}}

The Union of the above Interaction Products gives back

P P

35

Interaction Product Property 55. Ai and Bi are well-separated for all i=1,…,k

36

Two Points to Note about WSPDs

Two distinct points are considered to be well-separated

For any data set of size n, there is a trivial WSPD of size (n choose 2)

37

A Well-Separated Pairwise Decomposition (Continued)

If there are n points in P, a WSPD of P can be constructed in O(nlogn) time with O(n) elements using a fair split tree (Callahan 1995)

38

A Fair Split Tree

39

Creating a WSPD

Are the nodes outlined in yellow well-separated? No.

40

Creating a WSPD

Recurse on children of node with widest dimension

41

Creating a WSPD

Recurse on children of node with widest dimension

42

Creating a WSPD

Recurse on children of node with widest dimension

43

Creating a WSPD

And so on…

44

Base Case

Eventually you will find a well-separated pair of nodes.Add this pair to the WSPD.

45

Another Example of the Base Case

46

Creating a WSPDFindWSPD(W,NodeA,NodeB)

if( IsWellSeparated(NodeA,NodeB))AddPair(W,NodeA,NodeB)

elseif( MaxHrectDimLength(NodeA) <

MaxHrectDimLength(NodeB) )Swap(NodeA,NodeB)

FindWSPD(W,NodeA->Left,NodeB)FindWSPD(W,NodeA->Right,NodeB)

47

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

1. Create the Well-Separated Pairwise Decomposition

2. Take the pair (Ai,Bi) that corresponds to the shortest edge

3. If the vertices of that edge are not in the same connected component, add the edge to the MST. Repeat Step 2

48

Bichromatic Closest Pair Distance

Given two sets (Ai,Bi), the Bichromatic

Closest Pair Distance is the closest distancefrom a point in Ai to a point in Bi

49

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

1. Create the Well-Separated Pairwise Decomposition

2. Take the pair (Ai,Bi) with the shortest BCP distance

3. If Ai and Bi are not already connected, add the edge to the MST. Repeat Step 2.

50

GeoMST2 Example Start

Current MST

51

GeoMST2 Example Iteration 1

Current MST

52

GeoMST2 Example Iteration 2

Current MST

53

GeoMST2 Example Iteration 3

Current MST

54

GeoMST2 Example Iteration 4

Current MST

55

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

1. Create the Well-Separated Pairwise Decomposition

2. Take the pair (Ai,Bi) with the shortest BCP distance

3. If Ai and Bi are not already connected, add the edge to the MST. Repeat Step 2.

Modification for CFF:

If BCP distance > , terminate

56

Optimizations We don’t need the EMST We just need to cluster all points

that are within distance or less from each other

Allows two optimizations to GeoMST2 code

57

High Level Overview of GeoMST2

(A1,B1)

(A2,B2)

. . .(Am,Bm)

1. Create the Well-Separated Pairwise Decomposition

2. Take the pair (Ai,Bi) with the shortest BCP distance

3. If Ai and Bi are not already connected, add the edge to the MST. Repeat Step 2.

Optimizations take place in Step 1

58

Recall: How to Create the WSPD

59

Optimization 1 Illustration

60

Optimization 1

Ignore all links that are > Every pair (Ai, Bi) in the WSPD

becomes an edge unless it joins two already connected components

If MargDistance(Ai,Bi) > , then an edge of length cannot exist between a point in Ai and Bi

Don’t include such a pair in the WSPD

61

Optimization 2 Illustration

62

Optimization 2

Join all elements that are within distance of each other

If the max distance separating the bounding hyper-rectangles of Ai and Bi is , then join all the points in Ai and Bi if they are not already connected

Do not add such a pair (Ai,Bi) to the WSPD

63

Implications of the optimizations

Reduce the amount of time spent in creating the WSPD

Reduce the number of WSPDs, thereby speeding up the GeoMST2 algorithm by reducing the size of the priority queue

64

Results

Ran step two algorithms on subsets of the Sloan Digital Sky Survey

7 attributes – 4 colors, 2 sky coordinates, 1 redshift value

Compared Kruskal, GeoMST2, and -clustering

65

Results (GeoMST2 vs -Clustering vs Kruskal in 4D)

66

Results (GeoMST2 vs -Clustering in 3D)

67

Results (GeoMST2 vs -Clustering in 4D)

68

Results (Change in Time as changes for 4D data)

69

Results (Increasing Dimensions vs Time

70

Future Work More accurate, faster non-

parametric density estimator Use ball trees instead of fair split

tree Optimize algorithm if we keep h

constant but vary c and

71

Conclusions -clustering outperforms GeoMST2

by nearly an order of magnitude in higher dimensions

Combining the optimizations in both steps will yield an efficient algorithm for clustering against clutter on massive data sets