Embed Size (px)
Citation preview
SVM and Decision Tree
Machine Learning I CSE 6740, Fall 2013
Le Song
Which decision boundary is better?
Suppose the training samples are linearly separable
We can find a decision boundary which gives zero training error
But there are many such decision boundaries
Which one is better?
2
Class 1
Class 2
Compare two decision boundaries
Suppose we perturb the data, which boundary is more susceptible to error?
3
Constraints on data points
Constraints on data points
For all 𝑥 in class 2, 𝑦 = 1 and 𝑤⊤𝑥 + 𝑏 ≥ 𝑐
For all 𝑥 in class 1, 𝑦 = −1 and 𝑤⊤𝑥 + 𝑏 ≤ −𝑐
Or more compactly, (𝑤⊤𝑥 + 𝑏)𝑦 ≥ 𝑐
4
Class 1
Class 2
0 bxwT
c c
𝑤
Classifier margin
Pick two data points 𝑥1 and 𝑥2 which are on each dashed line respectively
The margin is 𝛾 =1
𝑤𝑤⊤ 𝑥1 − 𝑥2 =
2𝑐
| 𝑤 |
5
Class 1
Class 2
0 bxwT
c c
𝑤
𝑥1
𝑥2
Maximum margin classifier
Find decision boundary 𝑤 as far from data point as possible
max𝑤,𝑏 2𝑐
| 𝑤 |
𝑠. 𝑡. 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 ≥ 𝑐, ∀𝑖
6
Class 1
Class 2
0 bxwT
c c
𝑤
𝑥1
𝑥2
Support vector machines with hard margin
min𝑤,𝑏 𝑤
2
𝑠. 𝑡. 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 ≥ 1, ∀𝑖
Convert to standard form
min𝑤,𝑏
1
2𝑤⊤𝑤
𝑠. 𝑡. 1 − 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 ≤ 0, ∀𝑖
The Lagrangian function
𝐿 𝑤, 𝛼, 𝛽 = 1
2𝑤⊤𝑤 + 𝛼𝑖 1 − 𝑦
𝑖 𝑤⊤ 𝑥𝑖 + 𝑏
𝑚
𝑖
7
Deriving the dual problem
𝐿 𝑤, 𝛼, 𝛽 = 1
2𝑤⊤𝑤 + 𝛼𝑖 1 − 𝑦
𝑖 𝑤⊤ 𝑥𝑖 + 𝑏
𝑚
𝑖
Taking derivative and set to zero
𝜕𝐿
𝜕𝑤= 𝑤 − 𝛼𝑖𝑦
𝑖𝑥𝑖 =
𝑚
𝑖
0
𝜕𝐿
𝜕𝑏= 𝛼𝑖𝑦
𝑖 =
𝑚
𝑖
0
8
Plug back relation of w and b
𝐿 𝑤, 𝛼, 𝛽 =
1
2 𝛼𝑖𝑦
𝑖𝑥𝑖𝑚𝑖
⊤ 𝛼𝑗𝑦
𝑗𝑥𝑗𝑚𝑗 +
𝛼𝑖 1 − 𝑦𝑖 𝛼𝑗𝑦
𝑗𝑥𝑗𝑚𝑗
⊤ 𝑥𝑖 + 𝑏𝑚
𝑖
After simplification
𝐿 𝑤, 𝛼, 𝛽 = 𝛼𝑖 −
𝑚
𝑖
1
2 𝛼𝑖𝛼𝑗𝑦
𝑖𝑦𝑗 𝑥𝑖⊤𝑥𝑗
𝑚
𝑖,𝑗
9
The dual problem
max𝛼 𝛼𝑖 −
𝑚
𝑖
1
2 𝛼𝑖𝛼𝑗𝑦
𝑖𝑦𝑗 𝑥𝑖⊤𝑥𝑗
𝑚
𝑖,𝑗
𝑠. 𝑡. 𝛼𝑖 ≥ 0, 𝑖 = 1,… ,𝑚
𝛼𝑖𝑦𝑖 =
𝑚
𝑖
0
This is a constrained quadratic programming
Nice and convex, and global maximum can be found
𝑤 can be found as 𝑤 = 𝛼𝑖𝑦𝑖𝑥𝑖𝑚
𝑖
How about 𝑏?
10
Support vectors
Note that the KKT condition
𝛼𝑖 1 − 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 = 0
For data points with 1 − 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 < 0 , 𝛼𝑖 = 0
For data points with 1 − 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 = 0 , 𝛼𝑖 > 0
11
a6=1.4
Class 1
Class 2
a1=0.8
a2=0
a3=0
a4=0
a5=0
a7=0
a8=0.6
a9=0
a10=0 Call the training data points
whose ai's are nonzero the
support vectors (SV)
Computing b and obtain the classifer
Pick any data point with 𝛼𝑖 > 0, solve for 𝑏 with
1 − 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 = 0
For a new test point z
Compute
𝑤⊤𝑧 + 𝑏 = 𝛼𝑖𝑦𝑖 𝑥𝑖𝑧 + 𝑏
𝑖∈𝑠𝑢𝑝𝑝𝑜𝑟𝑡 𝑣𝑒𝑐𝑡𝑜𝑟𝑠
Classify 𝑧 as class 1 if the result is positive, and class 2 otherwise
12
Interpretation of support vector machines
The optimal w is a linear combination of a small number of data points. This “sparse” representation can be viewed as data compression
To compute the weights 𝛼𝑖, and to use support vector machines we need to specify only the inner products (or
kernel) between the examples 𝑥𝑖⊤𝑥𝑗
We make decisions by comparing each new example z with only the support vectors:
𝑦∗ = 𝑠𝑖𝑔𝑛 𝛼𝑖𝑦𝑖 𝑥𝑖𝑧 + 𝑏
𝑖∈𝑠𝑢𝑝𝑝𝑜𝑟𝑡 𝑣𝑒𝑐𝑡𝑜𝑟𝑠
13
Soft margin constraints
What if the data is not linearly separable?
We will allow points to violate the hard margin constraint (𝑤⊤𝑥 + 𝑏)𝑦 ≥ 1 − 𝜉
14
Class 1
Class 2
0 bxwT
1 1
𝑤
𝜉1
𝜉2 𝜉3
Soft margin SVM
min𝑤,𝑏,𝜉 𝑤
2+ 𝐶 𝜉𝑖
𝑚
𝑖=1
𝑠. 𝑡. 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 ≥ 1 − 𝜉𝑖 , 𝜉𝑖 ≥ 0, ∀𝑖
Convert to standard form
min𝑤,𝑏
1
2𝑤⊤𝑤
𝑠. 𝑡. 1 − 𝑦𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 − 𝜉𝑖 ≤ 0, 𝜉𝑖 ≥ 0, ∀𝑖
The Lagrangian function 𝐿 𝑤, 𝛼, 𝛽
= 1
2𝑤⊤𝑤 + 𝐶𝜉𝑖 + 𝛼𝑖 1 − 𝑦
𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 − 𝜉𝑖𝑚
𝑖
− 𝛽𝑖𝜉𝑖
15
Deriving the dual problem
𝐿 𝑤, 𝛼, 𝛽
= 1
2𝑤⊤𝑤 + 𝐶𝜉𝑖 + 𝛼𝑖 1 − 𝑦
𝑖 𝑤⊤ 𝑥𝑖 + 𝑏 − 𝜉𝑖𝑚
𝑖
− 𝛽𝑖𝜉𝑖
Taking derivative and set to zero
𝜕𝐿
𝜕𝑤= 𝑤 − 𝛼𝑖𝑦
𝑖𝑥𝑖 =
𝑚
𝑖
0
𝜕𝐿
𝜕𝑏= 𝛼𝑖𝑦
𝑖 =
𝑚
𝑖
0
𝜕𝐿
𝜕𝜉𝑖 = 𝐶 − 𝛼𝑖 − 𝛽𝑖 = 0
16
Plug back relation of 𝑤, 𝑏 and 𝜉
𝐿 𝑤, 𝛼, 𝛽 =
1
2 𝛼𝑖𝑦
𝑖𝑥𝑖𝑚𝑖
⊤ 𝛼𝑗𝑦
𝑗𝑥𝑗𝑚𝑗 +
𝛼𝑖 1 − 𝑦𝑖 𝛼𝑗𝑦
𝑗𝑥𝑗𝑚𝑗
⊤ 𝑥𝑖 + 𝑏𝑚
𝑖
After simplification
𝐿 𝑤, 𝛼, 𝛽 = 𝛼𝑖 −
𝑚
𝑖
1
2 𝛼𝑖𝛼𝑗𝑦
𝑖𝑦𝑗 𝑥𝑖⊤𝑥𝑗
𝑚
𝑖,𝑗
17
The dual problem
max𝛼 𝛼𝑖 −
𝑚
𝑖
1
2 𝛼𝑖𝛼𝑗𝑦
𝑖𝑦𝑗 𝑥𝑖⊤𝑥𝑗
𝑚
𝑖,𝑗
𝑠. 𝑡. 𝐶 − 𝛼𝑖 − 𝛽𝑖 = 0, 𝛼𝑖 ≥ 0, 𝛽𝑖 ≥ 0, 𝑖 = 1,… ,𝑚
𝛼𝑖𝑦𝑖 =
𝑚
𝑖
0
The constraint 𝐶 − 𝛼𝑖 − 𝛽𝑖 = 0, 𝛼𝑖 ≥ 0, 𝛽𝑖 ≥ 0 can be simplified to 𝐶 ≥ 𝛼𝑖 ≥ 0
This is a constrained quadratic programming
Nice and convex, and global maximum can be found
18
Learning nonlinear decision boundary
Linearly separable
Nonlinearly separable
19
The XOR gate Speech recognition
A decision tree for Tax Fraud
Input: a vector of attributes 𝑋 = [Refund,MarSt,TaxInc]
Output: 𝑌= Cheating or Not
H as a procedure:
20
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
Each internal node: test one attribute 𝑋𝑖 Each branch from a node: selects one value for 𝑋𝑖 Each leaf node: predict 𝑌
Apply model to test data I
21
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
R e fu n d M a r ita l
S ta tu s
T a x a b le
In c o m e C h e a t
N o M a rr ie d 8 0 K ? 10
Query Data
Start from the root of tree.
Apply model to test data II
22
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
R e fu n d M a r ita l
S ta tu s
T a x a b le
In c o m e C h e a t
N o M a rr ie d 8 0 K ? 10
Query Data
Apply model to test data III
23
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
R e fu n d M a r ita l
S ta tu s
T a x a b le
In c o m e C h e a t
N o M a rr ie d 8 0 K ? 10
Query Data
Apply model to test data IV
24
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
R e fu n d M a r ita l
S ta tu s
T a x a b le
In c o m e C h e a t
N o M a rr ie d 8 0 K ? 10
Query Data
Apply model to test data V
25
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
R e fu n d M a r ita l
S ta tu s
T a x a b le
In c o m e C h e a t
N o M a rr ie d 8 0 K ? 10
Query Data
Assign Cheat to “No”
Expressiveness of decision tree
Decision trees can express any function of the input attributes.
E.g., for Boolean functions, truth table row → path to leaf:
Trivially, there is a consistent decision tree for any training set with one path to leaf for each example.
Prefer to find more compact decision trees
26
Hypothesis spaces (model space
How many distinct decision trees with n Boolean attributes?
= number of Boolean functions
= number of distinct truth tables with 2n rows = 22n
E.g., with 6 Boolean attributes, there are 18,446,744,073,709,551,616 trees
How many purely conjunctive hypotheses (e.g., Hungry Rain)?
Each attribute can be in (positive), in (negative), or out 3n distinct conjunctive hypotheses
More expressive hypothesis space increases chance that target function can be expressed
increases number of hypotheses consistent with training set
may get worse predictions
27
Decision tree learning
28
Apply
Model
Induction
Deduction
Learn
Model
Model
Tid Attrib1 Attrib2 Attrib3 Class
1 Yes Large 125K No
2 No Medium 100K No
3 No Small 70K No
4 Yes Medium 120K No
5 No Large 95K Yes
6 No Medium 60K No
7 Yes Large 220K No
8 No Small 85K Yes
9 No Medium 75K No
10 No Small 90K Yes 10
Tid Attrib1 Attrib2 Attrib3 Class
11 No Small 55K ?
12 Yes Medium 80K ?
13 Yes Large 110K ?
14 No Small 95K ?
15 No Large 67K ? 10
Test Set
Tree
Induction
algorithm
Training Set
Decision
Tree
Example of a decision tree
29
T id R e fu n d M a rita l
S ta tu s
T a x a b le
In c o m e C h e a t
1 Y e s S in g le 1 2 5 K N o
2 N o M a rr ie d 1 0 0 K N o
3 N o S in g le 7 0 K N o
4 Y e s M a rr ie d 1 2 0 K N o
5 N o D iv o rc e d 9 5 K Y e s
6 N o M a rr ie d 6 0 K N o
7 Y e s D iv o rc e d 2 2 0 K N o
8 N o S in g le 8 5 K Y e s
9 N o M a rr ie d 7 5 K N o
1 0 N o S in g le 9 0 K Y e s10
Refund
MarSt
TaxInc
YES NO
NO
NO
Yes No
Married Single, Divorced
< 80K > 80K
Splitting Attributes
Training Data Model: Decision Tree
Another example of a decision tree
30
MarSt
Refund
TaxInc
YES NO
NO
NO
Yes No
Married Single,
Divorced
< 80K > 80K
There could be more than one tree that
fits the same data!
T id R e fu n d M a rita l
S ta tu s
T a x a b le
In c o m e C h e a t
1 Y e s S in g le 1 2 5 K N o
2 N o M a rr ie d 1 0 0 K N o
3 N o S in g le 7 0 K N o
4 Y e s M a rr ie d 1 2 0 K N o
5 N o D iv o rc e d 9 5 K Y e s
6 N o M a rr ie d 6 0 K N o
7 Y e s D iv o rc e d 2 2 0 K N o
8 N o S in g le 8 5 K Y e s
9 N o M a rr ie d 7 5 K N o
1 0 N o S in g le 9 0 K Y e s10
Training Data
Top-Down Induction of Decision tree
Main loop:
𝐴 ← the “best” decision attribute for next node
Assign A as the decision attribute for node
For each value of A, create new descendant of node
Sort training examples to leaf nodes
If training examples perfectly classified, then STOP;
ELSE iterate over new leaf nodes
31
Tree Induction
Greedy strategy.
Split the records based on an attribute test that optimizes certain criterion.
Issues
Determine how to split the records
How to specify the attribute test condition?
How to determine the best split?
Determine when to stop splitting
32
Multi-way split: Use as many partitions as distinct values.
Binary split: Divides values into two subsets. Need to find optimal partitioning.
CarType Family
Sports
Luxury
CarType {Family,
Luxury} {Sports}
CarType {Sports,
Luxury} {Family} OR
Splitting Based on Nominal Attributes
Multi-way split: Use as many partitions as distinct values.
Binary split: Divides values into two subsets. Need to find optimal partitioning.
Size Small
Medium
Large
Size {Medium,
Large} {Small}
Size {Small,
Medium} {Large} OR
Splitting Based on Ordinal Attributes
Different ways of handling Discretization to form an ordinal categorical attribute
Static – discretize once at the beginning
Dynamic – ranges can be found by equal interval bucketing, equal frequency bucketing (percentiles), or clustering.
Binary Decision: (𝐴 < 𝑡) or (𝐴 𝑡)
consider all possible splits and finds the best cut
can be more compute intensive
Splitting Based on Continuous Attributes
Taxable
Income
> 80K?
Yes No
Taxable
Income?
(i) Binary split (ii) Multi-way split
< 10K
[10K,25K) [25K,50K) [50K,80K)
> 80K
How to determine the Best Split
Idea: a good attribute splits the examples into subsets that are (ideally) "all positive" or "all negative"
Greedy approach: Nodes with homogeneous class distribution are preferred
Need a measure of node impurity
Non-homogeneous,
High degree of impurity
Homogeneous,
Low degree of impurity
How to compare attribute?
Entropy Entropy H(X) of a random variable X
H(X) is the expected number of bits needed to encode a randomly drawn value of X (under most efficient code)
Information theory:
Most efficient code assigns -log2P(X=i) bits to encode the message X=I, So, expected number of bits to code one random X is:
Sample Entropy
S is a sample of training examples
p+ is the proportion of positive examples in S
p- is the proportion of negative examples in S
Entropy measure the impurity of S
Examples for computing Entropy
C 1 0
C 2 6
C 1 2
C 2 4
C 1 1
C 2 5
P(C1) = 0/6 = 0 P(C2) = 6/6 = 1
Entropy = – 0 log 0 – 1 log 1 = – 0 – 0 = 0
P(C1) = 1/6 P(C2) = 5/6
Entropy = – (1/6) log2 (1/6) – (5/6) log2 (1/6) = 0.65
P(C1) = 2/6 P(C2) = 4/6
Entropy = – (2/6) log2 (2/6) – (4/6) log2 (4/6) = 0.92
How to compare attribute?
Conditional Entropy of variable 𝑋 given variable 𝑌 Given specific Y=v entropy H(X|Y=v) of X:
Conditional entropy H(X|Y) of X: average of H(X|Y=v)
Mutual information (aka information gain) of X given Y :
Information Gain
Information gain (after split a node):
𝑛 samples in parent node 𝑝 is split into 𝑘 partitions; 𝑛𝑖 is number of records in partition 𝑖
Measures Reduction in Entropy achieved because of the split. Choose the split that achieves most reduction (maximizes GAIN)
k
i
i
splitiEntropy
n
npEntropyGAIN
1
)()(
Problem of splitting using information gain
Disadvantage: Tends to prefer splits that result in large number of partitions, each being small but pure.
Gain Ratio:
Adjusts Information Gain by the entropy of the partitioning (SplitINFO). Higher entropy partitioning (large number of small partitions) is penalized!
Used in C4.5
Designed to overcome the disadvantage of Information Gain
42
SplitINFO
GAINGainRATIO
Split
split
k
i
ii
n
n
n
nSplitINFO
1
log
Stopping Criteria for Tree Induction
Stop expanding a node when all the records belong to the same class
Stop expanding a node when all the records have similar attribute values
Early termination (to be discussed later)
Decision Tree Based Classification
Advantages:
Inexpensive to construct
Extremely fast at classifying unknown records
Easy to interpret for small-sized trees
Accuracy is comparable to other classification techniques for many simple data sets
Example: C4.5
Simple depth-first construction.
Uses Information Gain
Sorts Continuous Attributes at each node.
Needs entire data to fit in memory.
Unsuitable for Large Datasets. Needs out-of-core sorting.
You can download the software from: http://www.cse.unsw.edu.au/~quinlan/c4.5r8.tar.gz
