An Efficient Greedy Method for Unsupervised Feature Selection

Ahmed Farahat

Joint work with

Ali Ghodsi, and

Mohamed Kamel{afarahat, aghodsib, mkamel}@uwaterloo.ca

ICDM 2011

Outline

• Introduction– Dimension Reduction & Feature Selection– Previous Work

• Proposed Work– Feature Selection Criterion– Recursive Formula– Greedy Feature Selection

• Experiments and Results• Conclusion

2

Dimension Reduction

• In data mining applications, data instances are typically described by a huge number of features.– Images (>2 megapixels)– Documents (>10K words)

• Most of these features are irrelevant or redundant.

• Goal: Reduce the dimensionality of the data:– allow a better understanding of data– improve the performance of other learning tasks

3

Feature Selection vs. Extraction

• Feature Selection (a.k.a variable selection)

searches for a relevant subset of existing features

(−) a combinatorial optimization problem

(+) features are easy to interpret

• Feature Extraction (a.k.a feature transformation)

learns a new set of features

(+) unique solutions in polynomial time

(−) features are difficult to interpret

4

Feature Selection

• Wrapper vs. filter methods: – Wrapper methods search for features which enhance the

performance of the learning task (+) more accurate, (−) more complex

– Filter methods analyze the intrinsic properties of the data, and select highly-ranked features according to some criterion.

(+) less complex, (−) less accurate

• Supervised vs. unsupervised methods

• This work: filter and unsupervised methods

5

Previous Work

• PCA-basedcalculate PCA, associate features with principal components based on their coefficients, select features associated with the first principal components (Jolliffe, 2002)

• Sparse PCA-based calculate sparse PCA (Zou et al. 2006), select for each principal component the subset of features with non-zero coefficients

• Convex Principal Feature Selection (CPFS) (Masaeli et al SDM’10)

formulates a continuous optimization problem which minimizes the reconstruction error of the data matrix with sparsity constraints

6

Previous Work (Cont.)

• Feature Selection using Feature Similarity (FSFS) (Mitra et al. TPAMI’02)

groups features into clusters and then selects a representative feature for each cluster

• Laplacian Score (LS) (He et al. NIPS’06)

selects features that preserve similarities between data instances

• Multi-Cluster Feature Selection (MCFS) (Cai et al. KDD’10)

selects features that preserve the multi-cluster structure of the data

7

This Work

• A criterion for unsupervised feature selection– minimizes the reconstruction error of the data matrix based on

the selected subset of features

• A recursive formula for calculating the criterion

• An effective greedy algorithm for unsupervised feature selection

8

P

S

Feature Select Criterion9

Data matrix

n

mLeast squares

Minimize lossfeatures

instances

Reconstructed matrix

Problem 1: (Unsupervised Feature Selection) Find a subset of features such that

where

and

This is an NP-hard combinatorial optimization problem.

Feature Select Criterion (Cont.)

10

Theorem 1: Given a set of features . For any ,

where

Recursive Selection Criterion11

P

S

Lemma 1: Given a set of features . For any ,

where

Recursive Selection Criterion (Cont.)

12

Proof of Lemma 113

Proof of Lemma 1 (Cont.)

• Let be the Schur complement of in .

• Use block-wise inversion formula of :

14

Recursive Selection Criterion (Cont.)

• Corollary 1: Given a set of features . For any ,

• Proof:

– Using Lemma 1,

15

0

Theorem 1: Given a set of features . For any ,

where

Recursive Selection Criterion16

Proof of Theorem 117

Greedy Selection Criterion

• Problem 2: (Greedy Feature Selection) At iteration t, find feature l such that,

• Using Theorem 1:

where

• Problem 2 is equivalent to:

18

Greedy Selection Criterion (Cont.)

19

Greedy Selection Criterion (Cont.)

• At iteration t:

• Problems:– Memory inefficient: – Computationally complex: per iteration

20

Greedy Selection Criterion (Cont.)

• At iteration t, define:

• Calculate E and G recursively as:

• ,

• Define ,

21

Update formulas for f and g

Memory-Efficient Selection22

Partition-based Selection

• Greedy selection criterion: + per iteration

• At each iteration, n candidate features x n projections• Solution:

– Partition features into c << n random groups– Select the feature which best represents the centroids of these

groups – Similar update formulas can be developed for f and g– Complexity: + per iteration

23

24

Experiments

Seven methods were compared• PCA-LRG: is a PCA-based method that selects features associated

with the first k principal components (Masaeli et al 2010)

• FSFS: is the Feature Selection using Feature Similarity (Mitra et al. 2006)

• LS: is the Laplacian Score (LS) method (He et al. 2006)

• SPEC: is the spectral feature selection method (Zhao et al. 2007)

• MCFS: is the Multi-Cluster Feature Selection method (Cai et al. 2010)

• GreedyFS: is the basic greedy algorithm (using recursive update formulas for f and g but without random partitioning)

• PartGreedyFS: is the partition-based greedy algorithm

25

Data Sets26

• These data sets were recently used by Cai et al. (2010) to evaluate different feature selection methods in comparison to the Multi-Cluster Feature Selection (MCFS) method.

Results – k-means27

Results – Affinity Propagation28

Results – Run Times29

Results – Run Times30

Conclusion

• This work presents a novel greedy algorithm for unsupervised feature selection.– a feature selection criterion which measures the reconstruction

error of the data matrix based on the subset of selected features– a recursive formula for calculating the feature selection criterion– an efficient greedy algorithm for feature selection, and two

memory and time efficient variants

• It has been empirically shown that the proposed algorithm – achieves better clustering performance– is less computationally demanding than methods that give

comparable clustering performance

31

32

Thank you!

References

• I. Jolliffe, Principal Component Analysis, 2nd ed. Springer, 2002• H. Zou, T. Hastie, and R. Tibshirani, “Sparse principal component analysis,” J.

Comput. Graph. Stat., 2006• M. Masaeli, Y. Yan, Y. Cui, G. Fung, and J. Dy, “Convex principal feature

selection,” SIAM SDM 2010• X. He, D. Cai, and P. Niyogi, “Laplacian score for feature selection,” NIPS 2006• Y. Cui and J. Dy, “Orthogonal principal feature selection,” in the Sparse

Optimization and Variable Selection Workshop, ICML 2008• Z. Zhao and H. Liu, “Spectral feature selection for supervised and unsupervised

learning,” ICML 2007• D. Cai, C. Zhang, and X. He, “Unsupervised feature selection for multi-cluster

data,” KDD 2010• P. Mitra, C. Murthy, and S. Pal, “Unsupervised feature selection using feature

similarity,” IEEE Trans. Pattern Anal. Mach. Intell., 2002.

33