Embed Size (px)
DESCRIPTION
Machine Learning and its Applications in Bioinformatics. Yen-Jen Oyang Dept. of Computer Science and Information Engineering. Observations and Challenges in the Information Age. A huge volume of information has been and is being digitized and stored in the computer. - PowerPoint PPT Presentation
Citation preview
Machine Learning and its Applications in Bioinformatics
Yen-Jen Oyang
Dept. of Computer Science and Information Engineering
Observations and Challenges in the Information Age
• A huge volume of information has been and is being digitized and stored in the computer.
• Due to the volume of digitized information, effectively exploitation of information is beyond the capability of human being without the aid of intelligent computer software.
An Example ofSupervised Machine Learning
(Data Classification)• Given the data set shown on next slide, can
we figure out a set of rules that predict the classes of objects?
Data Set
Data Class Data Class Data Class
（ 15,33）
O （ 18,28）
× （ 16,31）
O
（ 9 ,23）
× （ 15,35）
O （ 9 ,32）
×
（ 8 ,15）
× （ 17,34）
O （ 11,38）
×
（ 11,31）
O （ 18,39）
× （ 13,34）
O
（ 13,37）
× （ 14,32）
O （ 19,36）
×
（ 18,32）
O （ 25,18）
× （ 10,34）
×
（ 16,38）
× （ 23,33）
× （ 15,30）
O
（ 12,33）
O （ 21,28）
× （ 13,22）
×
Distribution of the Data Set
。。
10 15 20
30
。。。 。。
。 。。
××
××
×
×
×
×
×
×
××
×
×
Rule Based on Observation
.x
o
30
253015 22
class
else
class
, thenand y
yxIf
Rule Generated by a Kernel Density Estimation Based
Algorithm
Let and
If then prediction=“O”.
Otherwise prediction=“X”.
2o
2o
210
12o
o 2
1)( i
icv
i i
evf
.
2
1)(
2
214
12x
x
2x
x
j
jcv
j j
evf
),()( xo vfvf
(15,33)
(11,31)
(18,32)
(12,33)
(15,35)
(17,34)
(14,32)
(16,31)
(13,34)
(15,30)
1.723 2.745 2.327 1.794 1.973 2.045 1.794 1.794 1.794 2.027
ico
io
(9,23) (8,15)(13,37)
(16,38)
(18,28)
(18,39)
(25,18)
(23,33)
(21,28)
(9,32)(11,38)
(19,36)
(10,34)
(13,22)
6.458 10.08 2.939 2.745 5.451 3.287 10.86 5.322 5.070 4.562 3.463 3.587 3.232 6.260
jcx
jx
Identifying Boundary of Different Classes of Objects
Boundary Identified
Problem Definition ofData Classification
• In a data classification problem, each object is described by a set of attribute values and each object belongs to one of the predefined classes.
• The goal is to derive a set of rules that predicts which class a new object should belong to, based on a given set of training samples. Data classification is also called supervised learning.
The Vector Space Model
• In the vector space model, each object is described by a number of numerical attributes/features.
• For example, the outlook of a man is described by his height, weight, and age.
• It is typical that the objects are described by a large number of attributes/features.
Transformation of Categorical Attributes into Numerical
Attributes• Represent the attribute values of the object
in a binary table form as exemplified in the following:
10003
00112
01001School Graduate
Education
College
Education
School High
Education
Female
Male/Objects
• Assign appropriate weight to each column.
• Treat the weighted vector of each row as the feature vector of the corresponding object.
4
21
3
0003
002
0001School Graduate
Education
College
Education
School High
Education
Female
Male/Objects
Transformation of the Similarity/Dissimilarity Matrix
Model• In this model, a matrix records the
similarity/dissimilarity scores between every pair of objects.
P1 P2 P3 P4 P5 P6
P1 - 53 137 862 35 180
P2 - 46 72 816 606
P3 - 447 751 201
P4 - 291 156
P5 - 494
P6 -
• We may select P2, P5, P6 as representatives and use reciprocals of the similarity scores to these representatives to describe an object.
• For example, the feature vectors of P1 and P2 are <1/53, 1/35, 1/180> and
<0, 1/816, 1/606>, respectively.
Applications ofData Classificationin Bioinformatics
• In microarray data analysis, data classification is employed to predict the class of a new sample based on the existing samples with known class.
• Data classification has also been widely employed in prediction of protein family, protein fold, and protein secondary structure.
• For example, in the Leukemia data set, there are 72 samples and 7129 genes.• 25 Acute Myeloid Leukemia(AML) samples.
• 38 B-cell Acute Lymphoblastic Leukemia samples.
• 9 T-cell Acute Lymphoblastic Leukemia samples.
Model of Microarray Data SetsGene1 Gene2 ‧‧‧‧‧‧ Genen
Sample1
Sample2
Samplem
.),( RjiM
Alternative Data Classification Algorithms
• Decision tree (Q4.5 and Q5.0);
• Instance-based learning(KNN);
• Naïve Bayesian classifier;
• RBF network;
• Support vector machine(SVM);
• Kernel Density Estimation (KDE) based classifier.
Instance-Based Learning
• In instance-based learning, we take k nearest training samples of a new instance (v1, v2, …, vm) and assign the new instance to the class that has most instances in the k nearest training samples.
• Classifiers that adopt instance-based learning are commonly called the KNN classifiers.
Example of the KNN Classifiers
• If an 1NN classifier is employed, then the prediction of “” = “X”.
• If an 3NN classifier is employed, then prediction of “” = “O”.
Decision Function of the KNN Classifier
• Assume that there are two classes of samples, positive and negative.
• The decision function of a KNN classifier is:
.or query vect
of samplesnearest theare ,..., , where
,)sgn()sgn(
21
1
v
sss
sv
kk
k
i
i
Extension of the KNN Classifier
• We may extend the KNN classifier by weighting the contribution of each neighbor with a term related to its distance to the query vector:
.or query vect
of samplesnearest theare ,..., , where
,)sgn()()sgn(
21
1
v
sss
ssvv
k
w
k
k
i
iiii
A RBF Network Based Classifier with
Gaussian Kernels• It is typical that all are radial basis
functions of the same form.
• With the popular Gaussian function, the decision function is of the following form:
. )sgn(2
2
2
2
)(2)(2
j
j
i
i
ewewj
j
i
i
svsv
v
k ,...,, 21
The Common Structure of the RBF Network Based Data Classifier
v
)(1 v
)(vk
)(2 v
)(vf
iw
)(1 v
)(vk
)(2 v )(vf jw
2
2
2
2
)(2exp
)(2exp
Classifier DataBased
Network RBF
j
j
j
j
i
i
i
i
w
w
sν
sν
. i.e.,global aemploy and
sampleeach at function kernel a place
kernel RBF with the
ji
SVM
.
orset lly heuristica are and
ji
ji
NetworkstionRegulariza
mh
hw
adaptiveandVariable
KDE
2
1
and sampleeach at
function kernel a place
. and
2
1
h
m
h
hw
Fixed
KDE
Regularization of a RBF Network Based Classifier
• The conventional approaches proceed with either employing a constant σ for all kernel functions or employing a heuristic mechanism to set σi individually, e.g. a multiple of the average distance among samples, and attempt to minimize
where is a learning sample.
,
)(
)(
)(
)(
)(
)(
1 1
2
2
2
1
2
1
21
22221
11211
i
m
j
k
h
jh
ik
i
i
ik
i
i
mkmm
k
k
w
f
f
f
www
www
www
Es
s
s
s
s
s
s
is
• The term
is included to avoid overfitting and γ is to be set through cross validation.
m
j
k
h
jhwE1 1
2
ii
ii
ii
i
i
ii
k
h
hihi
s
ii
k
h
hihi
ii
k
h
hihi
i
k
h
iihh
ii
m
j
k
h
jh
k
h
iihh
fwww
E
fwww
E
fww
wfw
VV
w
wfw
w
E
ss
s
ss
s
s
ssss
ssss
ssss
sss
ss
)()()()(0
)()()()(0
Similarly,
)()()()(
02)()()(2
.tor column vec ofelement first theis )1( where
,0
)()(
0
1121
1
2121
2212
1
1212
1111
1
11
111
1
11
11
1 1
2
2
1
11
11
. and have weThen,
)()()(
)()()(
)()()(
Let W
1
21
22212
12111
21
22212
12111
21
22212
12111
T
ZWZW
fff
fff
fff
Z
www
www
www
TT
ikninin
ikii
ikii
kkkk
k
k
mkkk
m
m
sss
sss
sss
Decision Function of a SVM
• A prediction of the class of a new sample located at v in the vector space is based on the following rule:
.,0
, )(ˆ 2
2
2
2
22
Cww
bewewvf
ji
vn
j
j
vn
i
i
ji
ss
ctors.support ve thecalled are
weghtszero-non with and instances Those
"." class predicted Otherwise,
"." class predicted then ,0)(ˆ if
-ji
vf
ss
The Kernel Density Estimation (KDE) Based
Classifier• The KDE based learning algorithm constructs
one approximate probability density function for one class of objects.
• Classification of a new object is conducted based on the likelihood function:
objects. -class of
functiondensity y probabilit eapproximat theis )(ˆ and
ly,respective classes, all of samples trainingofnumber total theand
class of samples trainingofnumber theare and where
),(ˆ)(
m
f
mSS
vfS
SvL
m
m
mm
m
Identifying Boundary of Different Classes of Objects
Boundary Identified
Problem Definition of Kernel Density Estimation
• Given a set of samples
randomly taken from a probability distribution. We want to find a set of symmetric kernel functions and the corresponding weights such that
nsssS ,...,, 21
),;( iiK
iw
).(),;()(ˆ fKwf ii
i
i
The Proposed KDE Based Classifier
• We determined to employ the Gaussian function and set the width of each Gaussian function to a multiple of the average distance among neighboring samples:
. of samples gneighborin
theamong distance average theis and
where,2
exp2
11)(ˆ
12
2
i
i
ii
n
i i
id
i
s
nf
• can be estimated as follow:
sample.nearest
its to sample from distance theis )( and
space vector ldimensiona- ain )( radiuswith
sphere a of volume theis )1
2(
)(
where,1
)12
(
)()1(
2
1
2
k-thssR
dsR
dsR
dsR
k
iik
ik
dd
ik
d
i
i
dd
ik
i
Accuracy of Different Classification Algorithms
Data setclassification algorithms
KDE SVM 1NN 3NN
Satimage
(4335,2000)92.30 91.30 89.35 90.6
Letter
(15000,5000)97.12 97.98 95.26 95.46
Shuttle
(43500,14500)99.94 99.92 99.91 99.92
Average 96.45 96.40 94.84 95.33
Comparison of Execution Time(in seconds)
KDE without data reduction
KDE with data reduction SVM
Cross validation
Satimage 670 265 64622
Letter 2825 1724 386814
Shuttle 96795 59.9 467825
Make classifier
Satimage 5.91 0.85 21.66
Letter 17.05 6.48 282.05
Shuttle 1745 0.69 129.84
Test
Satimage 21.3 7.4 11.53
Letter 128.6 51.74 94.91
Shuttle 996.1 5.85 2.13
Parameter Setting through Cross Validation
• When carrying out data classification, we normally need to set one or more parameters associated with the data classification algorithm.
• For example, we need to set the value of k with the KNN classifier.
• The typical approach is to conduct cross validation to find out the optimal value.
• In the cross validation process, we set the parameters of the classifier to a particular combination of values that we are interested in and then evaluate how good the combination is based on alternative schemes.
• With the leave-one-out cross validation scheme, we attempt to predict the class of each sample using the remaining samples as the training data set.
• With 10-fold cross validation, we evenly divide the training data set into 10 subsets. Each time, we test the prediction accuracy of one of the 10 subsets using the other 9 subsets as the training set.
Overfitting
• Overfitting occurs when we construct a classifier based on insufficient quantity of samples.
• As a result, the classifier may works well for the training dataset but fail to deliver an acceptable accuracy in the real world.
• For example, if we toss a fair coin two times, there is a 50% chance that we will observe either side up in both tosses.
• Therefore, if we draw our conclusion on how fair the coin is with just two tosses, we may end up with overfitting the dataset.
• Overfitting is a serious problem in analyzing high-dimensional datasets, e.g. the microarray datasets.
Alternative Similarity Functions
• Let < vr,1, vr,2 ,…, vr,n> and < vt,1, vt,2 ,…, vt,n > be the gene expression vectors, i.e. the feature vectors, of samples Sr and St, respectively. Then, the following alternative similarity functions can be employed:• Euclidean distance—
n
hhthr vvitydissimilar
1
2,,
• Cosine—
• Correlation coefficient--
n
hht
n
hhr
n
hhthr
vv
vvSimilarity
1
2,
1
2,
1,,
n
hthtt
n
hrhrr
n
hhtt
n
hhrr
tr
n
hthtrhr
vn
vn
vn
vn
vvn
Similarity
1
2,
1
2,
1,
1,
1,,
ˆ1
1ˆ ˆ
1
1ˆ
1ˆ
1ˆ
where,ˆˆ
ˆˆ1
Importance of Feature Selection
• Inclusion of features that are not correlated to the classification decision may make the problem even more complicated.
• For example, in the data set shown on the following page, inclusion of the feature corresponding to the Y-axis causes incorrect prediction of the test instance marked by “”, if a 3NN classifier is employed.
• It is apparent that “o”s and “x” s are separated by x=10. If only the attribute corresponding to the x-axis was selected, then the 3NN classifier would predict the class of “” correctly.
x=10 x
y
Linearly Separable and Non-Linearly Separable
• Some datasets are linearly separable.
• However, there are more datasets that are non-linearly separable.
An Example of Linearly Separable
x
y
An Example of Non-Linearly Separable
x=10 x
y
A Simplest Case ofLinearly Separable
Feature Selection Based on the Univariate Analysis
Ai
S11 v11
S12 v12
S13 v13
: :
S21 v21
S22 v22
S23 v23
: :
S31 v31
S32 v32
S33 v33
Class 2
Class 1
Class 3
. 3) and 123nfor 3.07 e.g.(
1
ifonly and if
included, is attribute test,-F on the based Then,
. where
;1
;1
;1
Let
2
2...
1
1
1 1
2.
2
1
.
1 1
..
ks
kvvn
A
nn
vvkn
s
vn
vvn
v
i
k
i
i
i
k
i
i
k
i
n
j
iij
n
j
iji
i
k
i
n
j
ij
i
ii
An Example of Univariate Analysis
Sample X Y Class Sample X Y Class
1 7.1 9.1 1 11 10.9 8.8 2
2 6.7 10.2 1 12 10.8 10.3 2
3 7.5 10.6 1 13 11.1 11 2
4 7.6 8.8 1 14 12.3 9.1 2
5 8.1 10.3 1 15 12.1 9.7 2
6 8.0 11.0 1 16 12 10.9 2
7 8.6 8.9 1 17 13.1 8.9 2
8 8.7 9.8 1 18 12.8 10.1 2
9 9.2 11.2 1 19 13.2 11.3 2
10 6.5 10.1 1 20 13.7 9.9 2
Average 7.8 10.0 - Average 12.2 10.0 -
Joint p.m.f. of X, Y, and C
1
11
1
1
1
1
1
1
1
2
2
2
2
2
2
2
2
2
2
8
9
10
11
12
6 8 10 12 14
0
10
0.05). with statisticfor (threshold 4.35
24.16
10
2
2...
2
1
2
2...
2
1
Y
i
i
X
i
i
s
yy
F
s
xx
Blind Spot of the Univariate Analysis
• The univariate analysis is not able to identify crucial features in the following example:
x
y
The Demonstrative Data Set
Joint p.m.f. of X, Y, and C
0
2
4
6
8
10
12
0 2 4 6
1
2
• For Gene X,
• For Gene Y,
)threshold(35.40785.184.1
9845.1
9845.1155.384.210155.347.310
155.320
84.21047.310
22
35.4054.133.5
618.5
618.504.657.61004.651.510
04.620
57.61051.510
22
• However, if we apply the following linear transformation, then we will be able to identify the significance of these two genes:
2(GeneX)(GeneY)
• For “2 Gene X – Gene Y”,
35.4853.4460.0
912.26
912.2627.089.01027.043.110
27.020
)89.0(1043.110
22
• On the other hand, if we employ linear operator (x+2y), then we obtain
.3267.0
10102
2...2
2...1
ws
wwww
• Accordingly, the issue now is that how we can figure out the optimal linear operator of form αx+βy for the projection.
• In the 2-D case, given a set of samples
{(x1,y1), (x2,y2),…, (xn,yn)},
then vi = cosθxi+sinθyi
is the value obtained by projecting (xi,yi) onsinθx-cosθy=0or on the component along vector(cosθ, sinθ) as shown on the following page.
Feature Selection with Independent Components
Analysis (ICA)• In recent years, ICA has emerged as a
promising approach for carrying out multivariate analysis.
Basic Idea
• The ICA algorithm attempts to identify a plane so that when we project the data set on the plane the distribution is most non-gaussian.
A Measure of Non-Gaussianity
• The kurtosis is commonly employed to measure the non-Gaussianity of a data set.
• The kurtosis of a dataset {v1, v2 ,… , vn} is
2
1
2
1
41
4
)(1
1 and
1 where
,3)1(
)(
vvn
svn
v
sn
vv
n
ii
n
ii
n
ii
• The expected value of the kurtosis of a set of random samples taken from a standard normal distribution is 0.
• If the kurtosis of a set of random sample is larger than 0, then the p.d.f. of the distribution is sharper than the standard normal distribution.
• If the kurtosis of a set of random sample is smaller than 0, then the p.d.f. of the distribution is flatter than the standard normal distribution.
• Let kurt(θ) denote the kurtosis of {v1, v2 ,… , vn}
2
1
2
1
41
4
)sin(cos1
1 and
)sin(cos1
where
,3)1(
)sin(cos)(
vyxn
s
yxn
v
sn
vyxkurt
n
iii
n
iii
n
iii
• The issue now is to find the value of θ that minimizes kurt(θ).
• This is an optimization problem.
The Optimization Problem
• The optimization problem is to find the global maximum/minimum of a function.
• There are several heuristic algorithms designed for solving the optimization problem, e.g.• gradient descend;
• genetic algorithms;
• simulated annealing.
The Gradient Descend Algorithm
• In the gradient descend algorithm, a number of random samples are taken as the starting points.
• Then, we compute the gradient at each point and make a move in the direction of which the slope is maximum.
• This process is repeated a number of times until the convergent criterion is met.
An 1-D Example
d
dkurt )( to
right the tomove:
11
1
d
dkurt )( to
left the tomove :
22
2
d
dkurt )( to
right the tomove :
33
3
δ : is a parameter that controls the stepsize
• The gradient descend algorithm can be applied to multidimensional functions. In such cases, partial differentiation is involved.
• If the gradient descend algorithm is to be employed, then we must be able to compute the gradient of the function at any point in the vector space.
Blind Spot of ICA
• However, ICA may fail in the following non-linearly separable dataset
Data Clustering
• Data clustering concerns how to group a set of objects based on their similarity of attributes and/or their proximity in the vector space.
Model of Microarray Data Sets
Gene 1 Gene 2 Gene n
S11
S12
S13
:
S21 vi,j
S22
S23
:
S31
S32
S33
Class 2
Class 1
Class 3
Applications of Data Clustering in Microarray Data Analysis
• Data clustering has been employed in microarray data analysis for
• identifying the genes with similar expressions;
• identifying the subtypes of samples.
• For cluster analysis of samples, we can employ the feature selection mechanism developed for classification of samples.
• For cluster analysis of genes, each column of gene expression data is regarded as the feature vector of one gene.
The Agglomerative Hierarchical Clustering
Algorithms• The agglomerative hierarchical clustering
algorithms operate by maintaining a sorted list of inter-cluster distances/similarity.
• Initially, each data instance forms a cluster.• The clustering algorithm repetitively
merges the two clusters with the minimum inter-cluster distance or the maximum inter-cluster similarity.
• Upon merging two clusters, the clustering algorithm computes the distances between the newly-formed cluster and the remaining clusters and maintains the sorted list of inter-cluster distances accordingly.
• There are a number of ways to define the inter-cluster distance:• minimum distance/maximum similarity (single-link);
• maximum distance/minimum similarity (complete-link);
• average distance/average similarity;
• mean distance (applicable only with the vector space model).
• Given the following similarity matrix, we can apply the complete-link algorithm to obtain the dendrogram shown on the next slide.
g1 g2 g3 g4 g5 g6
g1 - 0.053 0.137 0.862 0.035 0.018
g2 - 0.046 0.072 0.816 0.606
g3 - 0.447 0.751 0.201
g4 - 0.291 0.156
g5 - 0.494
g6 -
• Assume that the complete-link algorithm is employed.
• If those similarity scores that are less than 0.3 are excluded, then we obtain 3 clusters {P1, P4}, {P2, P5, P6}, {P3}.
P1 P4 P2 P5P6P3
0.862 0.816
0.137
0.494
0.018
g1 g4 g2 g5 g3 g6
0.862 0.816
0.751
0.606
0.447
• If the single-link algorithm is employed, then we obtain the following result.
Example of the Chaining Effect
Single-link (10 clusters)
Complete-link (2 clusters)
Effect of Bias towards Spherical Clusters
Single-link (2 clusters) Complete-link (2 clusters)
K-Means: A Partitional Data Clustering Algorithm
• The k-means algorithm is probably the most commonly used partitional clustering algorithm.
• The k-means algorithm begins with selecting k data instances as the means or centers of k clusters.
• The k-means algorithm then executes the following loop iteratively until the convergence criterion is met.• repeat {
• assign every data instance to the closest cluster based on the distance between the data instance and the center of the cluster;
• compute the new centers of the k clusters;
• } until(the convergence criterion is met);
• A commonly-used convergence criterion is
If the difference between the values of two consecutive iterations is smaller than a threshold, then the algorithm terminates.
.cluster ofcenter theis where
,2
ii
C Cpi
Cm
mpEi i
Illustration of the K-Means Algorithm---(I)
initial center
initial center initial center
Illustration of the K-Means Algorithm---(II)
x
x
x
new center after 1st iteration
new center after 1st iteration
new center after 1st iteration
Illustration of the K-Means Algorithm---(III)
new center after 2nd iteration
new center after 2nd iteration
new center after 2nd iteration
A Case in which the K-Means Algorithm Fails
• The K-means algorithm may converge to a local optimal state as the following example demonstrates:
InitialSelection
Conclusions
• Machine learning algorithms have been widely exploited to tackle many important bioinformatics problems.
