Upload
friendbce
View
217
Download
0
Embed Size (px)
Citation preview
7/26/2019 Classify Ensemble
1/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 1
Examples of Ensemble Methods
How to generate an enemb!e o" c!ai"ier#$ %agging
$ %ooting
7/26/2019 Classify Ensemble
2/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 2
Bagging
Sam&!ing with re&!acement
%ui!d a c!ai"ier on each boottra& am&!e
'ach e(am&!e ha &robabi!it) 1/* being e!ected
+or an e(am&!e not being e!ected a"ter * time, the
&robabi!it) i 1 $ 1/*-*
when * i !arge, it i c!oe to 1/e- +or an e(am&!e being e!ected a"ter * time, the
&robabi!it) i 1 $ 1/e . 02
boottra& am&!e contain 3 o" the origina! data
Original Data 1 2 4 5 8 6 10
Bagging (Round 1) 5 8 10 8 2 10 10 6
Bagging (Round 2) 1 4 6 1 2 2 5 2
Bagging (Round 3) 1 8 10 6 5
7/26/2019 Classify Ensemble
3/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004
Boosting
n iterati7e &rocedure to ada&ti7e!) changeditribution o" training data b) "ocuing more on
&re7iou!) mic!ai"ied record
$ Initia!!), a!! * record are aigned eua!
weight$ 9n!i:e bagging, weight ma) change at the
end o" booting round
7/26/2019 Classify Ensemble
4/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 4
Boosting
;ecord that are wrong!) c!ai"ied wi!! ha7e theirweight increaed
;ecord that are c!ai"ied correct!) wi!! ha7e
their weight decreaed
Original Data 1 2 4 5 8 6 10
Boosting (Round 1) 5 2 8 5 6 4 10
Boosting (Round 2) 4 6 4 2 1 5 4 2
Boosting (Round 3) 4 4 8 10 4 4 4
7/26/2019 Classify Ensemble
5/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004
Example: AdaBoost
%ae c!ai"ier= >1, >2, ?, >T
'rror rate=
Im&ortance o" a c!ai"ier=
( )= =N
j
jjiji yxCwN 1 )(
1
=
i
ii
1ln
2
1
7/26/2019 Classify Ensemble
6/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004
Example: AdaBoost
@eight u&date=
I" an) intermediate round &roduce error rate
higher than 03, the weight are re7erted bac:
to 1/n and the ream&!ing &rocedure i re&eated >!ai"ication=
factorionnormalizattheiswhere
)(ifexp
)(ifexp)()1(
j
iij
iij
j
j
ij
i
Z
yxC
yxC
Z
ww
j
j
=
=
+
( )=
==T
j
jjy
yxCxC1
)(maxarg)(*
7/26/2019 Classify Ensemble
7/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 5
Illustrating AdaBoost
Data &oint"or trainingInitia! weight "or each data &oint
7/26/2019 Classify Ensemble
8/8
Tan,Steinbach, Kumar Introduction to Data Mining 4/18/2004 8
Illustrating AdaBoost