Upload
rebecca-perry
View
217
Download
0
Embed Size (px)
DESCRIPTION
Assign the most common class among the K-nearest neighbors (like a vote) 8/29/033Instance Based Classification
Citation preview
INSTANCE BASED APPROACH
KNN Classifier
Instance Based Classification 2
Simple classification technique
Handed an instance you wish to classify
Look around the nearby region to see what other classes are around
Whichever is most common—make that the prediction
8/29/03
0 5 100
510
Two Classes
X
Y
Instance Based Classification 3
K-nearest neighbor Assign the most common class among
the K-nearest neighbors (like a vote)
8/29/03
KNN CLASSIFIER
Instance Based Classification 4
How Train?
8/29/03
Don’t
Instance Based Classification 5
Let’s get specific Train
Load training data Classify
Read in instance Find K-nearest neighbors in the training data Assign the most common class among the
K-nearest neighbors (like a vote)
8/29/03
𝑑(𝑥 𝑖 ,𝑥 𝑗)≡√∑𝑟=1
𝑛
(𝑎𝑟 (𝑥 𝑖 )−𝑎𝑟 (𝑥 𝑗 ))¿2¿Euclidean distance: a is an attribute (dimension)
Instance Based Classification 6
𝑑(𝑥 𝑖 ,𝑥 𝑗)≡√∑𝑟=1
𝑛
(𝑎𝑟 (𝑥 𝑖 )−𝑎𝑟 (𝑥 𝑗 ))¿2¿
How find nearest neighbors
Naïve approach: exhaustive For the instance to be classified
Visit every training sample and calculate distance
Sort First K in the list
8/29/03
Euclidean distance: a is an attribute (dimension)
Voting Formula
Where is ’s class, if ; 0 otherwise
Instance Based Classification 7
Classifying a lot of work The Work that Must be Performed
Visit every training sample and calculate distance
Sort Lots of floating point calculations Classifier puts-off work till time to classify
8/29/03
Euclidean distance: a is an attribute (dimension)
𝑑(𝑥 𝑖 ,𝑥 𝑗)≡√∑𝑟=1
𝑛
(𝑎𝑟 (𝑥 𝑖 )−𝑎𝑟 (𝑥 𝑗 ))¿2¿
Instance Based Classification 8
Lazy This is known as a “lazy” learning method If do most of the work during the training stage
known as “eager” Our next classifier, Naïve Bayes, will be eager
Training takes a while but can classify fast Which do you think is better?
8/29/03
Lazy vs. Eager
Training or ClassifyingWhere the work happens
Instance Based Classification 9
Book mentions KD-TreeFrom Wikipedia: space‑partitioning data structure for organizing points in a k‑dimensional space. kd‑trees are a useful data structure for several applications, such as searches involving a multidimensional search key (e.g. range searches and nearest neighbor searches). kd-trees are a special case of BSP trees.
8/29/03
Instance Based Classification 10
If use some data structure …
Speeds up classification
Probably slows “training”
8/29/03
Instance Based Classification 11
How choose K? Choosing K can be a bit of an art What if you could include all data-points
(K=n)? How might you do such a thing?
8/29/03
How include all data points?What if weighted the votes of each training sample by its distance from the point being classified?
Weighted Voting Formula
Where , and is “1” if it is a member of class (i.e. where returns the class of )
Weight Curve
1 over distance squared
Could get less fancy and go linear But then training
data very-far-away still have strong influence
8/29/03 Instance Based Classification 12
0 20 40 60 80 100
020
4060
80100
Distance
Weight
0.0 0.2 0.4 0.6 0.8 1.0
0.0
0.2
0.4
0.6
0.8
1.0
Distance
Weight
Instance Based Classification 13
Could go more fancy Other Radial Basis
Functions Sometimes known as a
Kernel Function One of the more
common
8/29/03
𝐾 (𝑑 (𝑥 , 𝑥𝑡 ))= 1𝑒 √2𝜋
𝑒−(𝑥−𝜇)2 /2𝜎 2
-4 -2 0 2 4
0.0
0.2
0.4
0.6
0.8
1.0
Distance
Weight
Instance Based Classification 14
Issues Work back-loaded
Worse the bigger the training data Can alleviate with data structures
What else?
8/29/03
Other Issues?What if only some dimensions contribute to ability to classify? Differences in other dimensions would put distance between that point and the target.
Instance Based Classification 15
Curse of dimensionality Book calls this the curse of dimensionality More is not always better Might be identical in important dimensions but
distant in others
8/29/03
From Wikipedia:In applied mathematics, curse of dimensionality (a term coined by Richard E. Bellman),[1][2] also known as the Hughes effect[3] or Hughes phenomenon[4] (named after Gordon F. Hughes),[5][6] refers to the problem caused by the exponential increase in volume associated with adding extra dimensions to a mathematical space.
For example, 100 evenly-spaced sample points suffice to sample a unit interval with no more than 0.01 distance between points; an equivalent sampling of a 10-dimensional unit hypercube with a lattice with a spacing of 0.01 between adjacent points would require 1020 sample points: thus, in some sense, the 10-dimensional hypercube can be said to be a factor of 1018 "larger" than the unit interval. (Adapted from an example by R. E. Bellman; see below.)
Instance Based Classification 16
Gene expression data Thousands of genes Relatively few patients Is there a curse?
8/29/03
gene
patient
g1 g2 g3 … gndisea
se
p1 x1,1 x1,2 x1,3 … x1,n Yp2 x2,1 x2,2 x2,3 … x2,n N
.
.
.
.
.
.
pm xm,1 xm,2 xm,3 … xm,n ?
Instance Based Classification 17
Can it classify discrete data?
Bayesian could Think of discrete data as being pre-binned Remember RNA classification
Data in each dimension was A, C, U, or G
8/29/03
How measure distance?A might be closer to G than C or U (A and G are both purines while C and U are pyrimidines). Dimensional distance becomes domain specific.
Representation becomes all importantIf could arrange appropriately could use techniques like Hamming distances
Instance Based Classification 18
Another issue
Redness Yellowness Mass Volume Class4.816472 2.347954 125.5082 25.01441 apple2.036318 4.879481 125.8775 18.2101 lemon2.767383 3.353061 109.9687 33.53737 orange4.327248 3.322961 118.4266 19.07535 peach2.96197 4.124945 159.2573 29.00904 orange5.655719 1.706671 147.0695 39.30565 apple
8/29/03
First few records in the training data See any issues? Hint: think of how Euclidean distance is
calculatedShould really normalize the data
For each entry in a dimension
Instance Based Classification 19
Other uses of instance based approaches
Function approximation Real valued prediction: take average of
nearest k neighbors
If don’t know the function and/or it is too complex to “learn”, just plug-in a new value the KNN classifier can “learn” the predicted value on the fly by averaging the nearest neighbors8/29/03
�̂� (𝑥𝑞)←∑𝑖=1
𝑘
𝑓 (𝑥𝑖)
𝑘-10 -5 0 5
-10
-50
510
X
Y
Why average?
Instance Based Classification 20
Regression Choose an m and b that minimizes the
squared error But again,
computationallyHow?
8/29/03
-10 -5 0 5
-10
-50
510
X
Y
m and b that minimize
Instance Based Classification 21
Other things that can be learned
If want to learn an instantaneous slope Can do local regression Get the slope of a line that fits just the
local data
8/29/03
-10 -5 0 5 10
-3000
-2000
-1000
01000
2000
3000
X
Y
Instance Based Classification 22
The How: Big Picture For each of the training datum
we know what Y should be If we have a randomly
generated m and b, these, along with X will tell us a predicted Y
Know whether the m and b yield too large or too small a prediction
Can nudge “m” and “b” in an appropriate direction (+ or -)
Sum these proposed nudges across all training data8/29/03
∆𝑚∆ 𝑏
Target Y too low
Line represents output or predicted Y
Instance Based Classification 23
Gradient Descent Which way
should m go to reduce error?
8/29/03
y actual
y actual
𝑦 𝑝𝑟𝑒𝑑=𝑚𝑔𝑢𝑒𝑠𝑠 𝑥+𝑏𝑔𝑢𝑒𝑠𝑠
Rise
b
𝑦 𝑝𝑟𝑒𝑑− 𝑦𝑎𝑐𝑡
Could Average
Then do same for bThen do again
Instance Based Classification 24
Back to why we went down this road
Locally weighted linear regression Would still perform gradient descent Becomes a global function approximation
8/29/03
-10 -5 0 5 10
-3000
-2000
-1000
01000
2000
3000
X
Y
�̂� (𝑥 )=𝑤0+𝑤1𝑎1 (𝑥 )+…+𝑤𝑛𝑎𝑛 (𝑥)
Instance Based Classification 25
Summary KNN highly effective for many practical
problems With sufficient training data
Robust to noisy training Work back-loaded Susceptible to dimensionality curse
8/29/03
Instance Based Classification 268/29/03