Upload
ramya
View
38
Download
3
Tags:
Embed Size (px)
DESCRIPTION
Special topics on text mining [ Part I: text classification ]. Hugo Jair Escalante , Aurelio Lopez, Manuel Montes and Luis Villaseñor. Multi label text classification. Hugo Jair Escalante , Aurelio Lopez, Manuel Montes and Luis Villaseñor. - PowerPoint PPT Presentation
Citation preview
Special topics on text mining[Part I: text classification]
Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor
Multi label text classification
Hugo Jair Escalante, Aurelio Lopez, Manuel Montes and Luis Villaseñor
Most of this material was taken from: G. Tsoumakas, I. Katakis and I. Vlahavas. Mining multi-label data. Data Mining and Knowledge Discovery Handbook, Part 6, O. Maimon, L. Rokach (Ed.), Springer, 2nd edition, pp. 667-685, 2010.
Machine learning approach to TC
• Develop automated methods able to classify documents with a certain degree of success
Training documents(Labeled)
Learning machine(an algorithm)
Trained machine
Unseen (test, query) document
Labeled document
What is a learning algorithm?
• A function:
• Given:
: df C {1,..., }C K
1,...,{( , )}i i ND y x
;di iy C x
Binary vs multiclass classification
• Binary classification: each document can belong to one of two classes.
• Multiclass classification: each document can belong to one of K classes.
: { 1,1}df
: {1,..., }df K
Classification algorithms
• (Some) classification algorithms for TC :– Naïve Bayes – K-Nearest Neighbors– Centroid-based classification– Decision trees– Support Vector Machines– Linear classifiers (including SVMs)– Boosting, bagging and ensembles in general– Random forest– Neural networks
Some of this methods were designed for binary classification problems
Linear models• Classification of DNA micro-arrays
?
x1
x2
No Cancer
Cancer
( )f b x w x1 2,x x x
0b w x
0b w x
0b w x
?
Main approaches to multiclass classification
• Single machine: Learning algorithms able to deal with multiple classes (e.g., KNN, Naïve Bayes)
• Combining the outputs of several binary classifiers:– One-vs-all: one classifier per-class– All-vs-all: one classifier per pair of classes
Multilabel classification
• To what category belong these documents:
Multilabel classification
• A function:
• Given:
: df Z {1,..., }Z L K
1,...,{( , )}i i ND Z x
;di iZ L x
Conventions
X={xij}
n
mxi
y ={yj}
wSlide taken from I. Guyon. Feature and Model Selection. Machine Learning Summer School, Ile de Re, France, 2008.
Conventions
X={xij}
n
mxi
Z ={Zj}
w
|L|
Slide taken from I. Guyon. Feature and Model Selection. Machine Learning Summer School, Ile de Re, France, 2008.
Multi-label classification
• Each instance can be associated to a set of labels instead of a single one
• Specialized multilabel classification algorithms must be developed
• How to deal with the multilabel classification problem?
(Text categorization is perhaps the dominant multilabel application)
Multilabel classifiers
• Transformation methods: Transform the multilabel classification task into several single-label problems
• Adaptation approaches: Modify learning algorithms to support multilabel classification problems
Transformation methods
• Copy transformation. Transforms the multilabel instances into several single-label ones
Original ML problem Transformed ML problem (unweighted)
Transformed ML problem (weighted)
Transformation methods
• Select transformation. Replaces the multilabel of each instance by a single one
Original ML problem Transformed ML problem
Max Min Rand
Ignore approach
Transformation methods
• Label power set. Considers each unique set of labels in the ML problem as a single class
Original ML problem Transformed ML problem
Pruning can be applied
Transformation methods
• Binary relevance. Learns a different classifier per each different label. Each classifier i is trained using the whole data set by considering examples of class i as positive and examples of other classes (j≠i) as negative
• How labels are assigned to new instances?
Original ML problem Data sets generated by BR
Transformation methods
• Ranking by pairwise comparison. Learns a different classifier per each pair of different labels.
Original ML problem
Data sets generated by BR
Algorithm adaptation techniques
• Many variants, including – Decision trees – Boosting ensembles – Probabilistic generative models – KNN– Support vector machines
Algorithm adaptation techniques
• MLkNN. For each test instance:– Retrieve the top-k nearest neighbors to each
instance – Compute the frequency of occurrence of each
label – Assign a probability to each label and select the
labels for the test instance
Feature selection in multilabel classification
• An (almost) unstudied topic = opportunities • Wrappers can be applied directly (define an objective
function to optimize based on a multilabel classifier)
ValidationOriginal feature set
Generation EvaluationSubset of feature
Stopping criterion
yesnoSelected subset of feature
process
From M. Dash and H. Liu. http://www.comp.nus.edu.sg/~wongszec/group10.ppt
Feature selection in multilabel classification
• An almost un-studied topic = opportunities
• Existing filter methods transform the multilabel problem and apply standard filters for feature selection
Statistics
• Label cardinality
• Label density1
1( ) | |
m
ii
LC D Lm
1
| |1( )
mi
i
LLC D
m q
Evaluation of multilabel learning
• (New) conventions:
;di iY L x 1,...,{( , )}i i ND Y x
{ : 1,..., }jL j q
Data set
Labels
iZ L Predictions of a ML classifier for
instances in D
Evaluation of multilabel learning
• Hamming loss:
• Classification accuracy:
1
| |1
| |
Ni i
i
Y ZHL
N L
1
1( )
N
i ii
ACC I Z YN
( ) 1; ( ) 0;I true I false
Evaluation of multilabel learning
• Precision:
• Recall:
1
| |1
| |
Ni i
i i
Y ZP
N Y
1
| |1
| |
Ni i
i i i
Y ZR
N Y Z
Evaluation of multilabel learning
• F1-measure
11
2 | |1
| | | |
Ni i
i i i
Y ZF
N Z Y
Suggested readings• G. Tsoumakas, I. Katakis,I. Vlahavas. Mining multi-label data. Data Mining and Knowledge Discovery
Handbook, Part 6, O. Maimon, L. Rokach (Ed.), Springer, 2nd edition, pp. 667-685, 2010.
• G. Tsoumakas, I. Katakis. Multi-label classification: an overview. International Journal of Data Warehousing, 3(3), 1—13, 2007.
• M. Zhang, Z. Zhou. ML-kNN, A lazy learning approach to multi-label learning. Pattern recognition 40:2038—2048, 2007.
• M. Boutell, J. Luo, X. Shen. C. Brown. Learning multi-label scene classification. Pattern recognition 37:1757—1771, 2004.