• Home

  • Documents

  • CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon...
140
CS 6140: Machine Learning Spring 2017 Instructor: Lu Wang College of Computer and Informa@on Science Northeastern University Webpage: www.ccs.neu.edu/home/luwang Email: [email protected]

CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

  • Upload
    lequynh

  • View
    218

  • Download
    0

Embed Size (px)

Citation preview

Page 1: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

CS6140:MachineLearningSpring2017

Instructor:LuWangCollegeofComputerandInforma@onScience

NortheasternUniversityWebpage:www.ccs.neu.edu/home/luwang

Email:[email protected]

Page 2: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Logis@cs

•  Assignment1isout– Due2/9/2017– Startearly!

Page 3: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatwelearnedlast@me

•  Evalua@onmetrics

•  DecisionTree

•  Genera@veModels

•  Genera@veModelandDiscrimina@veModel

•  Logis@cRegression

Page 4: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 5: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 6: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 7: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

ROCPlot

•  Sensi@vity=a/(a+b)=Recall– Trueposi@verate

•  1-Specificity=1-d/(c+d)=c/(c+d)– Falseposi@verate

Page 8: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 9: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

DecisionTree

•  Playtennis?

•  Eachinternalnode:testonefeatureXi•  Eachbranchfromanode:selectsonevalueforXi•  Eachleafnode:predictY(orP(Y|X∈leaf))

Page 10: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Top-DownInduc@onofDecisionTrees

•  Whicha_ributetouseforsplit?

Page 11: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Top-DownInduc@onofDecisionTrees

•  Whicha_ributetouseforsplit?•  Goodsplitifwearemorecertainaboutclassifica@ona`ersplit– Determinis@cgood(alltrueorallfalse)– Uniformdistribu@onbad– Whataboutdistribu@oninbetween?

Page 12: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Informa@onGain

•  Gain(S,A)=expectedreduc@oninentropyduetosor@ngonA

Page 13: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Informa@onGain

Page 14: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Overficng

Page 15: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 16: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Bayesianmodel

•  H:Hypothesisspaceofpossibleconcepts•  X:nexamplesofaconceptC•  EvaluatehypothesesgivendatausingBayes’rule:

Page 17: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

NaïveBayes

Page 18: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

MaximumLikelihoodEs@ma@on

Page 19: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

MaximumLikelihoodEs@ma@on

Page 20: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

[email protected]@veModel

•  P(Y|X)=p(X,Y)/P(X)

•  Genera@vemodel–  LearnP(X,Y)fromtrainingsample–  P(X,Y)=P(Y)P(X|Y)–  Specifieshowtogeneratetheobservedfeaturesxfory

•  Discrimina@vemodel–  LearnP(Y|X)fromtrainingsample– Directlymodelsthemappingfromfeaturesxtoy

Page 21: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Logis@cRegression

Page 22: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Sigmoidfunc@on

•  Defini@on

Page 23: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Logis@cRegression

Page 24: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Logis@cRegression

Page 25: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

MaximizingLogLikelihood

Page 26: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 27: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 28: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 29: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GradientDescent•  Example

Page 30: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

ChangingStepSize

Page 31: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

AddingPrior

Page 32: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 33: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Today’sOutline

•  Perceptron(andkernels)

•  SupportVectorMachines

Page 34: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Perceptron

[SomeoftheslidesareborrowedfromAlexSmola’stutorial]

Page 35: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

BiologyandLearning

•  Idea1:Goodbehaviorshouldberewarded,badbehaviorpunished(ornotawarded).– Raisingadog.

•  Idea2:Correlatedeventsshouldbecombined.– Babieslearnlanguage.

Page 36: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

BiologyandLearning

•  Idea1:Goodbehaviorshouldberewarded,badbehaviorpunished(ornotawarded).– Raisingadog.

•  Idea2:Correlatedeventsshouldbecombined.– Babieslearnlanguage.

•  TrainingMechanisms– Behavioralmodifica@onofindividuals(learning)

•  Feedingthedog,thenthedoglearnstostandandsit.– Hard-codedbehaviorinthegenes(ins@nct)

•  Thewronglycodedanimaldies.

Page 37: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Neurons

Page 38: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Perceptron

Page 39: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Perceptron

•  Weightedcombina@on– Theoutputoftheneuronisalinearcombina@onoftheinputs

•  DecisionFunc@on– Attheendtheresultsarecombinedinto

Page 40: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Perceptron

•  Anabstractmodelistoassumethat

– Wherewistheweight,xisthefeaturevectors

–  bisthebias,•  BiologicalInterpreta@on–  Theweightswicorrespondtothesynap@cweights,themul@plica@oncorrespondstotheprocessingofinputsviathesynapses,andthesumma@onisthecombina@onofsignalsinthecellbody(soma).

Page 41: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LearningGoal:LinearSepara@on

Page 42: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PerceptronAlgorithm

Page 43: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PerceptronAlgorithm•  Nothinghappensifweclassify(xi,yi)correctly•  Ifweseeincorrectlyclassifiedobserva@onweupdatew

andb•  Posi@vereinforcementofobserva@ons

Page 44: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PerceptronAlgorithm•  Aboutthesolu@on– Weightvectorislinearcombina@onofobserva@onsxi:

Page 45: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PerceptronAlgorithm•  Aboutthesolu@on–  Classifica@oncanbewri_enintermsofdotproducts:

Page 46: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 47: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 48: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Pseudocode

Page 49: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

TheXORProblem

Page 50: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

TheXORProblem

•  Perceptronscannotlearnsuchlinearlyinseparablefunc@ons!

Page 51: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Problem

•  Linearfunc@onsareo`entoosimpletoprovidegoodes@mators.

Page 52: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Problem

•  Linearfunc@onsareo`entoosimpletoprovidegoodes@mators.

•  Idea:– Maptoahigherdimensionalfeaturespacevia

– Replaceevery by intheperceptronalgorithm.

Page 53: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 54: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 55: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PerceptrononFeatures

Page 56: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

ProblemswithConstruc@ngFeatures

Page 57: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

ProblemswithConstruc@ngFeatures

•  Needtobeanexpertinthedomain(e.g.Chinesecharacters).

•  Canbeexpensivetocompute.

Page 58: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PolynomialFeatures•  Dimension=1•  Dimension=2

•  Dimension=d(skipproof)

Page 59: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Kernels

•  Defini@on•  Akernelfunc@on isasymmetricfunc@oninitsargumentsforwhichthefollowingpropertyholds

Page 60: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Somechoicesofkernelfunc@ons

RBFkernel:Radialbasisfunc@onkernel

Page 61: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LinearKernel

Page 62: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LaplacianKernel

Page 63: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianKernel

Page 64: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

KernelPerceptron

Page 65: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LinearSeparators

•  Whichoftheselinearseparatorsisop@mal?

[SomeoftheslidesareborrowedfromDavidSontag’slecture]

Page 66: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 67: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Outline

•  Perceptron(andkernels)

•  SupportVectorMachines

Page 68: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

SupportVectorMachine(SVM)

•  SVMs(Vapnik,1990’s)choosethelinearseparatorwiththelargestmargin.

Hyperplane

Page 69: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

SupportVectorMachine(SVM)

•  Reasons:

–  Intui@on–  Theore@calguarantee(skiphere)–  Inprac@caltasks:SVMbecamefamouswhen,usingimagesasinput,it

gaveaccuracycomparabletoneural-networkwithhand-designedfeaturesinahandwri@ngrecogni@ontask.

Page 70: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

SupportVectorMachine(SVM)

•  Howtofindthehyperplane?

Hyperplane

Page 71: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Planes

•  Aplanecanbespecifiedasthesetofpointsgivenby

Page 72: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Planes

•  Aplanecanbespecifiedasthesetofpointsgivenby

NormalVector:decidethedirec@onoftheplane

Page 73: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Normaltoaplane

Lengthofthevector

Page 74: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Scaleinvariance

Page 75: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model
Page 76: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatisthedistance ?γ

Finalresult:canmaximizemarginbyminimizing

Page 77: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Supportvectormachines

Page 78: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatifthedataisnotlinearlyseparable?

Page 79: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatifthedataisnotlinearlyseparable?

•  Morefeatures

Page 80: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatifthedataisnotlinearlyseparable?

Oldobjec@ve

Newobjec@ve

Page 81: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatifthedataisnotlinearlyseparable?

Oldobjec@ve

Newobjec@ve

Jointlyminimizew.wandnumberoftrainingmistakes!

Page 82: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatifthedataisnotlinearlyseparable?

Page 83: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Allowingforslack:“So`marginSVM”

Page 84: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Allowingforslack:“So`marginSVM”

Page 85: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PopularToolsforSVMs

•  LIBSVM(c++)– h_ps://www.csie.ntu.edu.tw/~cjlin/libsvm/

•  SVMlight(c)– h_p://svmlight.joachims.org/

•  Scikit-learn(python)– h_p://scikit-learn.org/

Page 86: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

PopularToolsforSVMs

•  Torch(LuaJIT)– h_p://torch.ch/

•  Spider(Matlab)

•  Weka(Java)

Page 87: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Howdoweop@mizetheobjec@ve?

•  Quadra@cprogramming

Page 88: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Kernels

•  Defini@on•  Akernelfunc@on isasymmetricfunc@oninitsargumentsforwhichthefollowingpropertyholds

Page 89: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Howdoweop@mizetheobjec@ve?

•  Quadra@cprogramming

Noplacetoapplythekerneltrick

Page 90: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Constrainedop@miza@on

Page 91: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Constrainedop@miza@on

Page 92: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Constrainedop@miza@on

Page 93: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Lagrangemul@pliers–Dualvariables

•  LagrangeMul@pliers

Page 94: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Howdowesolvewithconstraints?

•  LagrangeMul@pliers

Page 95: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Lagrangemul@pliers–Dualvariables

Page 96: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

BacktoSVM(hardmargin)

Page 97: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

DualSVMderiva@on

Page 98: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

DualSVMderiva@on

Slater’scondi@onfromconvexop@miza@onguaranteesthatthesetwoop@miza@onproblemsareequivalent!(skipproof)

Page 99: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

DualSVMderiva@on

Page 100: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

DualSVMderiva@on

Page 101: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Togetwandb

Page 102: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Classifica@onruleusingdualsolu@onUsingdualsolu@ondotproduct

Page 103: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Dualforthenon-separablecase

Page 104: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Dualforthenon-separablecase

Page 105: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Howtointerpretdualform

Page 106: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Backtotheques@on:Whatifthedataisnotlinearlyseparable?

Page 107: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Forexample:Higherorderpolynomials

Page 108: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Dualformula@ononlydependsondot-productsofthefeatures!

Page 109: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Dualformula@ononlydependsondot-productsofthefeatures!

Page 110: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Kernels

•  Defini@on•  Akernelfunc@on isasymmetricfunc@oninitsargumentsforwhichthefollowingpropertyholds

Page 111: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

KernelTrick

Page 112: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

So`marginSVMwithkernel

Page 113: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

CommonkernelsforSVM

AkaGaussianRadialbasisfunc@on(RBF)kernel

Page 114: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Overficng

•  Hugefeaturespacewithkernels:shouldweworryaboutoverficng?– SVMobjec@veseeksasolu@onwithlargemargin– Goodtheore@calguarantee– Buteverythingoverfitssome@mes

Page 115: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Overficng

•  Hugefeaturespacewithkernels:shouldweworryaboutoverficng?– SVMobjec@veseeksasolu@onwithlargemargin– Goodtheore@calguarantee– Buteverythingoverfitssome@mes

•  Cancontrolby:– SecngC– Choosingabe_erkernel– Varyingparametersofthekernels

Page 116: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Dualforthenon-separablecase

Page 117: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LinearSVMC=50

Page 118: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LinearSVMC=50

Page 119: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LinearSVMC=50

Page 120: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

LinearSVMC=50

Page 121: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Insights

•  ChangingC– ForcleandataCdoesn’tma_ermuch.– Fornoisydata,largeCleadstonarrowmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)

Page 122: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Insights

•  ChangingC– ForcleandataCdoesn’tma_ermuch.– Fornoisydata,largeCleadstonarrowmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)

•  Noisydata– Cleandatahasfewsupportvectors– Noisydataleadstodatainthemargins– Moresupportvectorsfornoisydata

Page 123: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=0.1

Page 124: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=0.2

Page 125: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=0.4

Page 126: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=0.8

Page 127: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=1.6

Page 128: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=3.2

Page 129: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=6.4

Page 130: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFKernelwithC=12.8

Page 131: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Insights

•  ChangingC–  ForcleandataCdoesn’tma_ermuch.–  Fornoisydata,largeCleadstomorecomplicatedmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)

– OverfiMngforlargeC•  Noisydata–  Cleandatahasfewsupportvectors– Noisydataleadstodatainthemargins– Moresupportvectorsfornoisydata

Page 132: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

CommonkernelsforSVM

AkaGaussianRadialbasisfunc@on(RBF)kernel

Page 133: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFwithdifferent

Page 134: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFwithdifferent

Page 135: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFwithdifferent

Page 136: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

GaussianRBFwithdifferent

Page 137: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Insights

•  Changing–  Forcleandata, doesn’tma_ermuch.–  Fornoisydata,small leadstomorecomplicatedmargin(SVMtriestodoagoodjobatsepara@ng,eventhoughitisn’tpossible)

–  Lotsofoverficngforsmall•  Noisydata–  Cleandatahasfewsupportvectors– Noisydataleadstodatainthemargins– Moresupportvectorsfornoisydata

Page 138: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Homework(partofassignment2)

•  Studythe“Sequen@alMinimalOp@miza@on”algorithmandimplementanSVMclassifierbyyourself

•  References– h_p://cs229.stanford.edu/materials/smo.pdf– FastTrainingofSupportVectorMachinesusingSequen@alMinimalOp@miza@on

– h_p://research.microso`.com/pubs/68391/smo-book.pdf

Page 139: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Whatwelearnedtoday

•  Perceptron(andkernels)

•  SupportVectorMachines

Page 140: CS 6140: Machine Learning - College of Computer … · What we learned last @me • Evaluaon metrics • Decision Tree • Generave Models • Generave Model and Discriminave Model

Homework

•  ReadMurphyCH14.1-14.2,14.4-14.5.•  Assignment1isout.Dueintwoweeks.


Evaluaon) - Information and Computer Sciencecis519/fall2014/lectures/03_Evaluation.p… · Training)Dataand)TestData • Training)data:)dataused)to)build)the)model) • Testdata:)

Evaluaon) - Information and Computer Sciencecis519/fall2014/lectures/03_Evaluation.p… · Training)Dataand)TestData • Training)data:)dataused)to)build)the)model) • Testdata:)

Documents
Mc Millan Assignment 4 Edtc 6140

Mc Millan Assignment 4 Edtc 6140

Education
Sun storage tek 6140 technical presentation

Sun storage tek 6140 technical presentation

Technology
Storagetek 6140 Array

Storagetek 6140 Array

Documents
6140 final malware (1)

6140 final malware (1)

Documents
6140 Firmware Upgrade

6140 Firmware Upgrade

Documents
An Evaluaon of Real-Time Troposphere Products Based on mul

An Evaluaon of Real-Time Troposphere Products Based on mul

Documents
Preliminary*Evaluaon*of*ABAQUS,* FLUENT,*and*in

Preliminary*Evaluaon*of*ABAQUS,* FLUENT,*and*in

Documents
Battery Model: Part Number: Nominal Voltage: NSN: Description: Battery Model · 2019. 3. 17. · Part Number: 8010-044 Nominal Voltage: 6 volts NSN: 6140 01 475 9414 Description:

Battery Model: Part Number: Nominal Voltage: NSN: Description: Battery Model · 2019. 3. 17. · Part Number: 8010-044 Nominal Voltage: 6 volts NSN: 6140 01 475 9414 Description:

Documents
Whereare! you?!...EvaluatingWebsites… • Notanew%idea • Website%Evaluaon%Checklist:% • Accuracy,%Authority,%Objec4vity,%Currency,%Coverage% • Limitaons%of%website%evaluaon

Whereare! you?!...EvaluatingWebsites… • Notanew%idea • Website%Evaluaon%Checklist:% • Accuracy,%Authority,%Objec4vity,%Currency,%Coverage% • Limitaons%of%website%evaluaon

Documents
Performance*Evaluaon*of* …...Performance*Evaluaon*of* Programming*Models*for*the* Mul/core*Era Nicholas*J.*Wright NERSC/LBNL* njwright@lbl.gov* Programming*weather,*climate,*and*earthFsystem

Performance*Evaluaon*of* …...Performance*Evaluaon*of* Programming*Models*for*the* Mul/core*Era Nicholas*J.*Wright NERSC/LBNL* [email protected]* Programming*weather,*climate,*and*earthFsystem

Documents
Sun StorageTek 6140 Array Getting Started Guide

Sun StorageTek 6140 Array Getting Started Guide

Documents
fi-6140 MS

fi-6140 MS

Documents
Chunking:)An)Empirical)Evaluaon) of)So7ware)Architecture)(?)) · Chunking:)An)Empirical)Evaluaon) of)So7ware)Architecture)(?)) Rachana)Koneru)))David)M.)Weiss)))))IowaState)University)

Chunking:)An)Empirical)Evaluaon) of)So7ware)Architecture)(?)) · Chunking:)An)Empirical)Evaluaon) of)So7ware)Architecture)(?)) Rachana)Koneru)))David)M.)Weiss)))))IowaState)University)

Documents
LDR 6140 Strategic Leadership Reflection Paper

LDR 6140 Strategic Leadership Reflection Paper

Documents
BEYOND’VOLUNTEERING:MEASURING’ THESOCIALIMPACT ... · BEYOND’VOLUNTEERING:MEASURING’ THESOCIALIMPACT ’ CASESTUDYVOLUNTEERINGMATTERS’ March&2018& Thaddaeus&Douglas,&Impactand&Evaluaon&Manager&

BEYOND’VOLUNTEERING:MEASURING’ THESOCIALIMPACT ... · BEYOND’VOLUNTEERING:MEASURING’ THESOCIALIMPACT ’ CASESTUDYVOLUNTEERINGMATTERS’ March&2018& Thaddaeus&Douglas,&Impactand&Evaluaon&Manager&

Documents
Sun StorageTek 6140 Array Administration Guide

Sun StorageTek 6140 Array Administration Guide

Documents
hw aRRAY 6140

hw aRRAY 6140

Documents
Phaser 6140 User Guide

Phaser 6140 User Guide

Documents
The Botanical Arts City / City of St. Albert · ACCT 6140 6140 6140 6140 ACCT 1220 1225 1226 From 40.00 30.00 38.00 38.00 Mleage Claim (From Qut:of-Regipn Mileage Claim (or In-RegigQt

The Botanical Arts City / City of St. Albert · ACCT 6140 6140 6140 6140 ACCT 1220 1225 1226 From 40.00 30.00 38.00 38.00 Mleage Claim (From Qut:of-Regipn Mileage Claim (or In-RegigQt

Documents
ProDAQ Relay Switching Units - Bustecbustec.com/support/datasheets/6140-XX-DS.pdf · Switching Units ProDAQ 6140 LXI Matrix Switching Unit OveRvIew The ProDAQ 6140 provides a relay

ProDAQ Relay Switching Units - Bustecbustec.com/support/datasheets/6140-XX-DS.pdf · Switching Units ProDAQ 6140 LXI Matrix Switching Unit OveRvIew The ProDAQ 6140 provides a relay

Documents
DIY Evaluaon - UNM Evaluation Labevallab.unm.edu/for-organizations/2017-diy-eval-pres-2.pdf · DIY Evaluaon Annual Evaluaon Lab Workshop Melissa Binder, Director, NM Evaluaon Lab

DIY Evaluaon - UNM Evaluation Labevallab.unm.edu/for-organizations/2017-diy-eval-pres-2.pdf · DIY Evaluaon Annual Evaluaon Lab Workshop Melissa Binder, Director, NM Evaluaon Lab

Documents
SR 6140 - Porters

SR 6140 - Porters

Documents
Sun storage tek 6140 customer presentation

Sun storage tek 6140 customer presentation

Technology
Evaluaon) - Penn Engineeringcis519/fall2014/lectures/03_Evaluation… · Training)Dataand)TestData • Training)data:)dataused)to)build)the)model) • Testdata:) new)data,)notused)in)the)training)process)

Evaluaon) - Penn Engineeringcis519/fall2014/lectures/03_Evaluation… · Training)Dataand)TestData • Training)data:)dataused)to)build)the)model) • Testdata:) new)data,)notused)in)the)training)process)

Documents
IIRecogniDon,&Characterizaon,&&&&&& Risk&Evaluaon ,&Risk

IIRecogniDon,&Characterizaon,&&&&&& Risk&Evaluaon ,&Risk

Documents
CHOOSE THE 6-SERIES THATSUITS YOUR FARM DF... · 6140 TTV 136HP 6130 TTV 126HP DEUTZ-FAHR 6140 SUMMIT FROM $123,750+GST FROM $135,000+GST DEUTZ-FAHR 6130/6140 $2530 $2759 PER MONTH

CHOOSE THE 6-SERIES THATSUITS YOUR FARM DF... · 6140 TTV 136HP 6130 TTV 126HP DEUTZ-FAHR 6140 SUMMIT FROM $123,750+GST FROM $135,000+GST DEUTZ-FAHR 6130/6140 $2530 $2759 PER MONTH

Documents
(PLUMBER) Summary and ongoing work · The PALS Land sUrface Model Benchmarking Evaluaon pRoject (PLUMBER) Summary and ongoing work Gab Abramowitz UNSW Sydney & ARCCSS Mar

(PLUMBER) Summary and ongoing work · The PALS Land sUrface Model Benchmarking Evaluaon pRoject (PLUMBER) Summary and ongoing work Gab Abramowitz UNSW Sydney & ARCCSS Mar

Documents
Paper review of ENGG*6140 Optimization Techniques

Paper review of ENGG*6140 Optimization Techniques

Documents
heurisc)evaluaon )clab.iat.sfu.ca/432/wp-content/uploads/files/HeuristicEvaluation.pdf · heurisc)evaluaon interface$inspec@on$method$thatis$used$to$see$ if$an$interface$complies$with$design$principles$

heurisc)evaluaon )clab.iat.sfu.ca/432/wp-content/uploads/files/HeuristicEvaluation.pdf · heurisc)evaluaon interface$inspec@on$method$thatis$used$to$see$ if$an$interface$complies$with$design$principles$

Documents