Editor: Ruili Wang 39

Computational Methods of Feature SelectionHUAN L IU AND HIROSHI MOTODA

REVIEWED BY LONGBING CAOAND DAVID TANIAR

Feature selection selects a subset ofrelevant features, and also removes ir-relevant and redundant features from thedata to build robust learning models.Feature selection is very important, notonly because of the curse of dimension-ality, but also due to emerging data com-plexities and quantities faced by multipledisciplines, such as machine learning,data mining, pattern recognition, statis-tics, bioinformatics, and text mining.

In recent years, we have seen exten-sive and productive efforts on featureselection. The research has been ex-panding from simple to complex featuretypes, from supervised to unsupervisedand semi-supervised feature selection,and from simple to more advanced tech-niques, both in depth and in breadth.

"Computational Methods of FeatureSelection", edited by H. Liu and H.Motoda, two leading experts in the field,collects recent research works from var-ious disciplines on computational meth-ods in feature selection and extrac-tion. The collection reflects the advance-ments in recent years, following the ed-

itors’ pioneer book on feature selectionpublished in 1998. Consequently, thesepublications provide a comprehensiveroadmap of feature selection research,and this is certainly very helpful to avery wide audience from beginners toprofessionals, and from practitioners toresearchers.

The collection features a state-of-the-art survey, technique advancement, prac-tical guides, promising directions, andcase studies. It ranges from presentingbackground and fundamentals relevantto feature selection, to recent results inextending feature selection, weightingand local methods for feature selection,as well as feature selection progressin text mining and bioinformatics, or-ganized in five independent parts. Thecontent, carefully selected from invitedcontributors, relevant workshops and tu-torials, covers areas such as text classifi-cation, web mining, bioinformatics andhigh-dimensional data.

The book starts with an introductionand background material, which consistsof four chapters. A very insightful andenjoyable overview on background andthe basics of feature selection are pre-sented in Chapter 1. An evolutionarypicture is drawn on feature selectiondevelopment from supervised to unsu-pervised and semi-supervised learning inorder to handle the increasing mixture oflabeled, unlabeled and partially labeleddata. An overview of unsupervised fea-ture selection is presented in Chapter 2,which highlights the identification of thesmallest feature subset that best uncov-ers interesting and natural clusters basedon certain criteria. Filter and wrappermethods are introduced to select featuresin unlabeled data, while subspace clus-tering and co-clustering/bi-clustering arediscussed from the local unsupervisedfeature selection perspective.

Randomization is widely used in fea-ture selection when appropriate choicescan be managed. A survey on random-ized feature selection is presented in

Chapter 3. Two types of randomization,namely Las Vegas and Monte Carlo al-gorithms, are introduced, followed byan overview of three complexity classesdefining the probabilistic requirementsin analyzing randomized algorithms. Thework features six illustrations of ran-domized features and prototype algo-rithms for feature selection problems.Another factor that may facilitate featureselection is causal relationship discov-ery for cutting down dimensionality anddeep understanding of the underlyingmechanism. Chapter 4 addresses non-causal and causal feature selection. Witha definition of probabilistic causality,the causal Bayesian network is used toanalyze feature relevance and further todesign a causal discovery algorithm byfinding the Markov blanket.

Recent advancements in feature se-lection are highlighted in Part II in anumber of strategies. Firstly, Chapter 5describes how an active feature value isacquired to estimate feature relevance indomains where feature values are ex-pensive to measure. A sampling bene-fit function is derived from a statisticalformulation of the problem, followed byan active sampling algorithm, which isshown to outperform random samplingby a mixture model for the joint class-feature distribution to reduce the numberof feature samples. Secondly, from thefeature extraction perspective, Chapter6 presents the notion of the decisionborder, in which a labeled vector quan-tizer, that can efficiently be trained bythe Bayes risk weighted vector quanti-zation (BVQ) algorithm, is devised toextract the best linear approximation. Itis shown that the approach gives compa-rable results to the SVM-based decisionboundary and performs better than themulti-layer perceptron-based method.

Chapter 7 further explains how in-dependent probe variables are used inthe same distribution in generating aprobe. Feature relevance is comparedwith the relevance of its randomly per-

IEEE Intelligent Informatics Bulletin November 2008 Vol.9 No.1

40 Book Review: Computational Methods of Feature Selection

muted probes for classification usingrandom forests. The approach is promis-ing in terms of data types and quantity,performance, and computational com-plexity. Finally, in Chapter 8, an incre-mental ranked usefulness is used to de-cide whether or not a feature is relevantin massive data, and then to select thebest non-consecutive features from theranking. The approach chooses a smallsubset of features with similar predictiveperformance to others in dealing withhigh-dimensional data.

Strategies and methods related toweighting and local methods are ad-dressed in Part III. Firstly, the Relieffamily algorithms are described in Chap-ter 9. Relief is extended to a more real-istic variant ReliefF to deal with incom-plete data for classification, and is fur-ther extended to the Regressional Reli-efF for regression problems. The varietyof the Relief family shows its generalapplicability as a non-myopic featurequality measure. Feature selection in K-means is usually not automated. Chapter10 proposes techniques to automaticallydetermine the important features in K-means clustering. This is done throughcalculating the sum of the within-clusterdispersions of the feature, and renewingthe weights in an iterative process.

In contrast to maximum benefit-basedactive feature sampling, Chapter 11 fo-cuses on local feature relevance andweighting by designing adaptive metricsor parameter estimates that are local inan input space. Chapter 12 presents amathematical interpretation of the Reliefalgorithms. It is proven to be equivalentto solving an online convex optimizationproblem with a margin-based objectivefunction. New feature weighting algo-rithms are then proposed to find thenearest neighbor classifier.

In Part IV, text feature selection isaddressed by a survey, a new feature se-lection score, and constraint-guided andaggressive feature selection approaches.Firstly, Chapter 13 presents a compre-hensive overview of feature selectionfor text classification, including featuregeneration, representation, and selec-

tion, with illustrative examples, from apragmatic viewpoint. Text feature gen-erators, such as word merging, wordphrases, character N-grams, and multi-field records are introduced. An intro-duction to classification feature filter-ing is also provided. Secondly, Chap-ter 14 introduces a new feature selec-tion score, namely posterior inclusionprobability under Bernoulli and Poissondistributions. The score is defined asthe posterior probability of including agiven feature over all possible models,in which each model corresponds to adifferent set of features that includes thegiven feature. The advantage of the scoreis that the selected features are easy tointerpret while maintaining comparableperformance to other typical score met-rics, such as information gain.

Two different pairwise constraint-guided dimensionality reduction ap-proaches, through projecting data intoa lower space and co-clustering of fea-tures and data, are introduced in Chap-ter 15. Investigations are also conductedon improving semi-supervised clusteringperformance in high-dimensional data.In Chapter 16, an aggressive feature se-lection method is proposed, which canfilter more than 95% features for textmining. To handle feature redundancy,information gain-based ranking for textclassification is also proposed using amutual information measure and inclu-sion index.

The last section covers feature selec-tion in bioinformatic data, which maynot be effectively handled by generalfeature selection approaches. This partconsists of four chapters. Chapter 17introduces the challenges of micro-arraydata analysis and presents a redundancy-based feature selection algorithm. AMarkov blanket based filter method isproposed to approximate the selection ofdiscriminative and non-redundant genes.In Chapter 18, a scalable method basedon sequence components and domainknowledge is developed to generate au-tomatic features on biological sequencedata. The algorithm can construct fea-

tures, explore the space of possiblefeatures, and identify the most usefulones. Chapter 19 proposes an ensemble-based method to find robust featuresfor biomarker discovery. Ensembles areobtained by choosing different alter-natives at each stage of data miningfrom normalization to binning, featureselection, and classification. Finally, apenalty-based feature selection methodis proposed in Chapter 20 to produce asparse model by utilizing the groupingeffect. As a generalization of a penal-ized least squares method, lasso, theproposed approach is promising in han-dling high-dimensional data for variouspurposes, such as regression and classi-fication problems.

Overall, we enjoyed reading this book.It presents state-of-the-art guidanceand tutorials on methodologies andalgorithms in computational methodsin feature selection. Enhanced by theeditors insights, and based on previouswork by these leading experts in thefield, the book forms another milestoneof relevant research and development infeature selection. The selected chaptersalso present interesting open issuesand promising directions for furtherexploration of feature selection in thenext decade. With such a researchroadmap, it is highly exciting to foreseethe next generation of feature selectionmethodologies and techniques inspiredby this collection.

About the reviewers:

LONGBING CAO: Data Sciences andKnowledge Discovery Laboratory, Cen-tre for Quantum Computation and In-telligent Systems, University of Tech-nology Sydney, Australia. Contact himat lbcao@it.uts.edu.au, and http://www-staff.it.uts.edu.au/∼lbcao/ and datamin-ing.it.uts.edu.au for more information.

DAVID TANIAR: Clayton Schoolof Information Technology, MonashUniversity, Australia. Contact him atDavid.Taniar@infotech.monash.edu.au,and http://users.monash.edu.au/∼dtaniar/for more information.

November 2008 Vol.9 No. 1 IEEE Intelligent Informatics Bulletin