Evaluation Measures for Models Assessment over Imbalanced Data Sets.pdf

8/14/2019 Evaluation Measures for Models Assessment over Imbalanced Data Sets.pdf

1/13

Journal of Information Engineering and Applications www.iiste.org

ISSN 2224-5782 print! ISSN 2225-"5"# online!

$ol.%& No.'"& 2"'%

27

Evaluation Measures for Models Assessment over Imbalanced

Data Sets

(o)amed *e++ar'& ,r.assia /)eliouane ,0emaa2& ,r.1a+lit A+rouf Alitouc)e''. ENSSEA& National Sc)ool of Statistics and Applied Economics& Algiers& Algeria&

2. EE& Ecole des autes Etudes ommerciales& Algiers& Algeria&3 E-mail of t)e corresponding aut)or mo).e++argmail.com

AbstractImalanced data learning is one of t)e c)allenging prolems in data mining6 among t)is matter& founding t)erig)t model assessment measures is almost a primar researc) issue. S+ewed class distriution causes amisreading of common ealuation measures as well it lead a iased classification. 1)is article presents a set ofalternatie for imalanced data learning assessment& using a comined measures 9-means& li+eli)ood ratios&,iscriminant power& :-(easure *alanced Accurac& ;ouden inde cure& Area ?nder ure& @artial A?& eig)ted A?& umulatie

9ains ure and lift c)art& Area ?nder Bift A?B!& t)at aim to proide a more credile ealuation. e analCet)e applications of t)ese measures in c)urn prediction models ealuation& a well +nown application ofimalanced dataKeywords: imalanced data& (odel assessment& accurac & 9-means& li+eli)ood ratios& :-(easure& ;oudeninde& A?& @-A?&-A?& Bift& A?B

1. Introduction:

1)e prolem of mining imalanced data sets receie muc) interest in recent earsD'# D2' D%'D%4D45D576considered as one of t)e top '" c)allenges for data mining D#4& t)e imalanced data is encounter in seeral realworld application suc) as social sciences& credit card fraud detection& customer retention& c)urn prediction&segmentation. Een& in medical diagnostic and fraud detection t)e imalanced data sets are t)e norms and note


2/13



$ol.%& No.'"& 2"'%

28

1ale 2 presents t)e most well +nown fundamental ealuation metrics1ale 2. :undamental ealuation metrics ased on confusion matri< analsis

(easure :ormula interpretation

Accurac + + + + Error rate '-Accurac +

+ + +

Sensitiit or =ecall! +

Accurac of positie e


3/13



$ol.%& No.'"& 2"'%

2G

1ale %. 1)res)olds for positie li+eli)ood ratio interpretationB alue (odel contriution

' Negligile'-5 @oor

5-'" :airQ '" 9ood

1)e li+eli)ood ratios are used particularl in medical diagnosis prediction D'8 D4"& learning fromp)armaceutical data sets D23.3 Discriminant power:

,iscriminant power ,@! is a measure t)at summariCes sensitiit and speciMcit& calculated as per formula D2%

=,-

. (0 + 02)

)ere R sensitiit '- sensitiit! & and ; speciMcit '- speciMcit!,@ ealuates )ow well an algorit)m distinguis)es etween positie and negatie en t)e ot)er )and& t)e :-(easure is deried from a more general relations)ip called T aried :-(easure& t)eformula is written as

8=(1 + 9:) 4 566 4 7

9:4 566 4 7

)ere T is a coefficient to ad0ust t)e relatie importance of precision ersus recall& decreasing T leads areduction of precision importance6 in t)e general case T is considered eKual to '.)awla et al suggest to e


4/13



$ol.%& No.'"& 2"'%

%"

prediction on t)e ma0orit class& t)en t)e alanced accurac will drop D''. 1)e alanced accurac )as een usedin seeral pulications as on statistical patterns of epitasis learning D2"& 1e


5/13



$ol.%& No.'"& 2"'%

%'

Anot)er innoatie approac) was proposed D'% in deeloping *-=> as an alternatie to t)e traditional=>& t)at generate a new tpe of cure most suitale to t)e comparison incase of intersection as s)own in figure

elow

:igure 2. *-=> cures illustration1)e => cures are a good wa to compare models or sets of models& )oweer& as stated ,rummond and olteD22 t)e )ae t)ree ma0or disadantages for ealuating imalanced data models

osts and t)e prior distriution of t)e classes are not ta+en into account6 1)e decision t)res)olds are not e cure6 1)e difference etween t)e two models is difficult to Kuantif.

4.'.2 Area ?nder ure& A? 1)e area under t)e => cure Area under cure& A?! is a summar indicator of => cure performance t)atcan summariCe t)e performance of a classifier into a single metric. ?nli+e difficulties encountered in t)ecomparison of different => cure especiall in intersection case& t)e A? can sort models oerall

performance& as a result& t)e A? is more considered in models assessment D7.1)e A? is estimated t)roug) arious tec)niKues& t)e most used is t)e trapeCoidal met)od& w)ic) is ageometrical met)od ased on linear interpolation etween eac) point on t)e => cure Een simpler& someaut)ors D5"D4 propose to ma+e t)e appro6? @A7 =1 3B ( + )=

B +

B

1)e A? )as an important statistical propert t)e A? of a classiMer is eKuialent to t)e proailit t)at t)eclassiMer will ran+ a randoml c)osen positie instance )ig)er t)an a randoml c)osen negatie instance D27.In practice& t)e alue of A? aries etween ".5 and '6 Allaire D% suggest t)e following scale for t)einterpretation of A? alue

1ale 5. A? $alue interpretationA? $alue (odel performance

".5 - ".# @oor".# W ".7 :air".7 W ".8 9ood".8 W ".G $er 9ood".G W '." Et)eralternaties were proposed in t)e literature to ac)iee a more clear assessments& we can mention4.'.% @artial A?@artial A? @A?! originall proposed (clis) D42& is getting increasingl used in recent pulicationsD#2. 1)e logic e)ind t)e @A? as descried ,odd and @epeD'G is to estimate t)e A? on a specific areaof t)e decision t)res)old6 t)us t)e @A? can compare different models for t)e same enc)mar+ of decisiont)res)old& t)e aut)ors illustrate t)e @A? t)roug) t)e following grap)


6/13



$ol.%& No.'"& 2"'%

%2

:igure %. 1)e @artial A? illustration4.'.4 eig)ted A?eig)ted A? is a ariant of A? t)at fits etter t)e imalanced data learning case D'5. 1)e rationale e)indA? is +nowing t)at classifier w)ic) perform well in t)e )ig)er 1@ region is preferred oer ones t)at does not6so instead of summing up area under cure wit) eKual weig)ts& we want to gie more importance to t)e area nearto t)e top of grap)& so we create a s+ew weig)t ector distriuting more weig)ts towards t)e top of t)e =>cure as s)own in t)e following figure

.:igure 4. eig)ted A? estimation approac)

1)e A? was considered as alternatie to asic A? in t)e pulication of 9o)ore *i D2G& and ;uanfang andal D5G.4.2 +umulatie Gains +ure and li)t chart:

1)e cumulatie gains cure represents t)e percentage of positie relatie to t)e percentage of targeted population

according to score deciles. t)e lift c)art is deried from cumulatie gains cure w)ere on eac) point t)e liftrepresents t)e ratio etween t)e percentage positie to t)e percentage of targeted population6 so it tells )owmuc) etter a classiMer predicts compared to a random selectionwit)in t)e same grap) setting t)e F of t)e target population to 2"F & t)e point * on t)e lift cure 57F

positie! is aoe t)e random position point A wit) 2"F positie! and a lower t)at t)e ideal situation wit) 8"Fpositie!


7/13



$ol.%& No.'"& 2"'%

%%

:igure 5. umulatie 9ains ure and lift )artBift is nearl related to accurac& and )as t)e adantage of eing well used in mar+eting practice D%G6 as on

imalanced data learning assessment D5%.4.2.' Area ?nder Bift& A?BSimilarl to t)e A? associated to => ure& we define t)e A?B Area under lift!6 1uffer D#% demonstratet)at t)e A?B can e estimated t)roug) t)e following eKuation

@D =3

+ (1 )4@D;

)ere @ @rior proailit of positie oseration on t)e populatione can conclude t)at t)e A? is alwas greater t)an A?B& proided t)at A? Q ".5& in ot)er words t)at =>cure is aoe t)e diagonal. 1)is formula also s)ows t)at

If A? ' i.e score separated perfectl! t)en A?B p2 ' W p! ' W p2 If A?".5 i.e =andom prediction model! t)en A?B p2 X '-@! "&5 If t)e proailit @ is er small& t)e areas under t)e two cures are er close

In all cases& to deduce t)at a model is superior to anot)er& it is eKuialent to measuring t)e area under t)e cure

lift A?B or t)e area under t)e => cure A? & ie if A?'Q A?2 So A?B'Q A?B2

&. A$$lication casesand discussion:!.1 #pplications +ases area:

>ne of t)e most common cases of imalanced data set learning is t)e c)urn prediction. onsidered in ariousindustries suc) as telecom D'# D%2 D4G an+ing and finance D54 D#" retail D4% Y insuranceD'7. 1)e c)urn

prediction )as een also a fruitful field to deelops and assess a new approac)es for )andling imalanced data D8D44 D4G or comine t)e +nown data mining tec)niKues and met)ods D%2 D%".:ollowing t)e same trend& we perform applications cases of c)urn prediction ased data sets issued from twowireless telecom operators. it)in t)is e


8/13



$ol.%& No.'"& 2"'%

%4

1ale #. =esults of c)urn prediction (odels performance on 'stdata Set

,ata Set '(odel 'A5 *asic

(odel '* 5cost sensitie

(odel ' 5E


9/13



$ol.%& No.'"& 2"'%

%5

1)e => and lift cures confirms t)e initial results& t)e models '* and ', perform etter t)an ' speciall ont)e upper limits of percentile 4"Z'""!. :ollowing t)is assessment we can affirm t)at model '* is t)e est inclass in t)is case in spite of t)e lowest Accurac t)at it )as!.3 Discussion o) 2

ndData et results:

1)e applications on 2nd

data set proide t)e results detailed on t)e ne and lift ures for models generated on 2nddata Set.+,! curve -ift !urve


10/13



$ol.%& No.'"& 2"'%

%#

In t)is case t)at appear more comple< for decision ma+ing t)an t)e preious one& t)e final decision s)ouldconsider a comination of different measures instead of reling on one measure. :ollowing an approac) ofmodels ran+ing we can osere t)at (odel 2E )as one of )ig)est accurac rate& coupled wit) e and Bift.

&. !onclusion:

1)e model assessment measure is a +e factor in data mining process success6 wit)in imalanced data conte& Jos)i A0a& 2""8!&\Automaticall counteringimalance and its empirical relations)ip to cost]& prin5er cience ?"usiness Media& BB.D'5 )eng 9& @oon J&2""8!&\A New Ealuation (easure for Imalanced ,atasets]& con)erence (th

#ustralasian Data Minin5 +on)erence 8#usDM 9*9lenelg& Australia.D'# lement /irui& Bi ong& Edgar /irui&2"'%!&\andling lass Imalance in (oile 1elecoms ustomer)urn @rediction]&7nternational ournal o) +omputer #pplications722%!7-'%.D'7 ,anso Samuel >dei& 2""#!& \An E


11/13



$ol.%& No.'"& 2"'%

%7

D'G ,odd& B& E.& @epe& (& S.& 2""%!& \@artial A? Estimation and =egression]& @A "iostatistics Aorkin56aper eries& National Institute of ealt) ?niersit of as)ington6 @aper '8'.D2" ,igna& =.& $eleC& *ill& .& )ite& A& A&& (otsinger& & S.& *us)& (& ,.& =itc)ie& S& (.& illiams& J&.&2""7!&\A alanced accurac function for epitasis modeling in imalanced datasets using multifactor

dimensionalit reduction]& Genetic pidemiolo5$6 $olume %'& Issue 4& pages %"#W%'5.D2' ,ing& .&2"''!& ^,iersified Ensemle lassifier for ig)l imalanced ,ata Bearning and t)eirapplication in *ioinformatics^& @). , t)esis& ollege of Arts and science& ,epartment of omputer Science&9eorgia State ?niersitD22 ,rummond )ris& olte =oert & 2"""!&\E analsis]&6attern ,eco5nition Better278!& 8#'W874.D28 :em & :lac) @& >rallo J& Bac)ice N&2""4!& ^:irst or+s)op on => Analsis in AI^& The uropean+on)erence on #rti)icial 7ntelli5ence*EAI[ 2""4.D2G 9o)ore *i 9oue ,enis&2"'"!&_ aluation et contrle de lUirrbgularitb de la prise mbdicamenteuse @roposition et dbeloppement de stratbgies rationnelles fondbes sur une dbmarc)e de modblisations

p)armacocinbtiKues et p)armacodnamiKues& 1)se de @)ilosop)ih ,octor @).,.! en sciencesp)armaceutiKues option tec)nologie p)armaceutiKue& ?niersitb de (ontrbal.D%" adden Jo)n& 2""8!& \A ustomer @rofiling (et)odolog for )urn @rediction]& 6hD thesis& Sc)ool ofApplied Sciences& ranfield ?niersit.D%' ido S)o)ei & /as)ima isas)i& 1a+a)as)i ;uta+a.2""G!.\=oug)l alanced agging for imalanced data]&tatistical #nal$sis and Data Minin5* $olume 2& Issue 5-#.D%2 wang& .& Jung& 1.& Su)& E.& 2""4!6 \An B1$ (odel and ustomer Segmentation *ased on ustomer

$alue A ase Stud on t)e ireless 1elecommunications Industr]& 'pert s$stems with applications& 2#& '8'-'88.D%% /aragiannopoulos ( 9.& Anfantis , S.& /otsiantis S *.& @intelas @ E.&2""7!&\Bocal cost sensitielearning for )andling imalanced data sets]&Mediterranean +on) on +ontrol ? #utomation& (E, ["7.D%4 /ittisa+& /.& Nittaa& /.&2"''!&\A ,ata (ining Approac) to Automate :ault ,etection (odel,eelopment in t)e Semiconductor (anufacturing @rocess]&7nternational ournal o) Mechanics& 4& $ol 5.D%5 /uat& (.& Y (atwin& S. 'GG7!& ^Addressing t)e curse of imalanced training sets >ne-sided selection]&

7n Dou5las C. Fisher* editor* 7+MB& pages '7GW'8#. (organ /aufmann.D%# Bewis ,& 9ale & 'GG8!&^1raining te cure]&Medical Decision Makin5& 'G"W'G5.D4% (igubis& $.B.& $an den @oel& ,.& aman)o& A&S.& un)a J& :.& 2"'2!&\(odeling partial customer c)urn >nt)e alue of first product-categor purc)ase seKuences]&'pert $stems with #pplications%GD44 @end)ar+ar @& 2""G!&\9enetic algorit)m ased neural networ+ approac)es for predicting c)urn in cellularwireless networ+s serice]&'pert $stems with #pplications& %#.D45 @etr Nle+a& $o0tjc) Ste+& 2"'2!& \Improing Efficienc of 1elemedical @reention @rograms t)roug)

,ata-mining on ,iagnostic ,ata]& 4th 7nternational +on)erence on "ioin)ormatics and "iomedical Technolo5$*76+"ol.2G


12/13



$ol.%& No.'"& 2"'%

%8

D4# @roost :& :awcett 1& 2""'!&\=oust classification for imprecise enironments]& (ac)ine Bearning42%!2"%W 2%'.D47 @roost :& :awcett 1&'GG7!&\Analsis and $isualiCation of lassifier @erformance omparison underImprecise lass and ost ,istriutions* 7n 6roceedin5s o) the 3rd 7nternational +on)erence on 1E - (a0orit eig)ted(inorit >ersampling 1ec)niKue for Imalanced ,ata Set Bearning]&7 Transactions on stract and #pplied #nal$sis6 Article I, 'G#25#.D58 ;ouden & 'G5"!&_ Inde< for rating diagnostic tests 6 +ancer& % %2W%5&

D5G ;uanfang& ,.& Riongfei& B.& Jun& B.& aiing& .&2"'2! &\Analsis on eig)ted A? for Imalanced ,ataBearning 1)roug) Isometrics]&ournal o) +omputational 7n)ormation $stems8 '& %7'W%78D#" ;uan *o& (a Riaoli& 2"'2!& \Sampling =eweig)ting *oosting t)e @erformance of Ada*oost onImalanced ,atasets]& A++7 7 Aorld +on5ress on +omputational 7ntelli5ence& June& '"-'5& *risane&Australia.D#' )ang& ,.& Bee& & S&.2""8!&^Bearning classiMers wit)out negatie e


13/13

This academic article was published by The International Institute for Science,

Technology and Education (IISTE). The IISTE is a pioneer in the Open Access

Publishing service based in the U.S. and Europe. The aim of the institute is

Accelerating Global Knowledge Sharing.

More information about the publisher can be found in the IISTEs homepage:http://www.iiste.org

CALL FOR JOURNAL PAPERS

The IISTE is currently hosting more than 30 peer-reviewed academic journals and

collaborating with academic institutions around the world. Theres no deadline for

submission. Prospective authors of IISTE journals can find the submission

instruction on the following page: http://www.iiste.org/journals/ The IISTE

editorial team promises to the review and publish all the qualified submissions in a

fastmanner. All the journals articles are available online to the readers all over the

world without financial, legal, or technical barriers other than those inseparable from

gaining access to the internet itself. Printed version of the journals is also available

upon request of readers and authors.

MORE RESOURCES

Book publication information:http://www.iiste.org/book/

Recent conferences: http://www.iiste.org/conference/

IISTE Knowledge Sharing Partners

EBSCO, Index Copernicus, Ulrich's Periodicals Directory, JournalTOCS, PKP Open

Archives Harvester, Bielefeld Academic Search Engine, Elektronische

Zeitschriftenbibliothek EZB, Open J-Gate, OCLC WorldCat, Universe Digtial

Library , NewJour, Google Scholar
http://www.iiste.org/http://www.iiste.org/http://www.iiste.org/journals/http://www.iiste.org/journals/http://www.iiste.org/book/http://www.iiste.org/book/http://www.iiste.org/book/http://www.iiste.org/conference/http://www.iiste.org/conference/http://www.iiste.org/conference/http://www.iiste.org/book/http://www.iiste.org/journals/http://www.iiste.org/

Documents

Evaluation Measures for Models Assessment over Imbalanced Data Sets.pdf