Farhad Hormozdiari

Lab for Computational Biology, Simon Fraser University

EXTENDED NEAREST NEIGHBOR CLASSIFICATION METHODS FOR PREDICTING

SMALL MOLECULE ACTIVITY

Outline

Small Molecule Similarity Measure Classification

Kernel Methods Nearest Neighbor classifier Centroid based Nearest Neighbor

Distance / Metric Learning Results

What are small molecules ?

Chemical compounds with small molecular mass

Important in the synthesis and maintenance of larger molecules (DNA, RNA and proteins).

High potential as medicine. Increasing number of databases: PubChem,

ChemDB, ChemBank… Standard task in in silico drug discovery:

Classifying an compound with unknown activity

Representation of small molecules

Chemical (Conventional) Descriptors: A(x)=(25, 0.24, 1, 12.3,….., 5, 2.12,

……..) Chemical structures represented by

labeled graphs

Classification methods for small molecules

Artificial Neural Networks (ANN) Support Vector Machine (SVM) K-Nearest Neighbor Classification Recent works focused on Kernel Methods

SVM (Support Vector Machine) Φ(x) fixed feature transformation tn ϵ{1,-1} Find a decision boundary

Y(x) = WT Φ(x) + b Goal to maximize the distance Dist= Quadratic programming

||||

))((

||||

)(

w

bxwt

w

xyt Tnnn

)]})(([min||||

1{maxarg , bxwtw n

Tnnbw

Recent works on small molecule classification Mariginalized Kernel (MK)

Tsuda et.al 2002, Kashima et al. (ICML 2003)

Features are number of labeled paths of random walks

Improved Mariginalized Kernel Mahe et al. (ICML 2005) Avoid totters (walks that visit a node which

was visited in two previous stages)

Recent works on small molecule Classification Swamidass et al. (Bioinformatics 2003)

Kernels based on 3D Euclidean coordinates of atoms

One histogram per pair of atom labels Similarity between histograms

Cao et al. (ISMB 2008, Bioinformatics 2010) Use Maximum Common Substructure (MCS)

as a measure of similarity Randomly pick ”basis” compounds

Features of a molecule are MCS between that molecule and all basis compounds

Nearest Neighbor Classification

Nearest Neighbor (NN) Classification The label of a molecule is predicted based

on ones of its nearest neighbors NN Error < 2*Bayes error (Cover et al.

1967) One of most used classifiers in small

molecule classification because of its simplicity

Nearest Neighbor Classification Drawbacks

Speed/Memory Distances to all traning set points should be

computed All the traning set is stored in the memory

Overfitting

Centroid based Nearest Neighbor (CBNN) Classificatrion CBNN Classification

Centroids are picked from each class Bioactivity of a small molecule is predicted based on its

nearest centroids

CBNN tackle NN drawbacks

Centroid Selection

Hart et al., 1968 introduced Condensed NN Classification Initially, the set of centroids S includes one point Iteratively go through each remaining point p, if

its nearest neighbor in S has the opposite class, p is added to S

Fast condensed NN Classification (Angiulli et al., ICML 2005) S is assigned to medoids of each class For each point in S their Voronoi cell is build

In each Voronoi cell if there exist a point from different class is added to S

Centroid Selection

Gabriel Graph (Gabriel et al. 1969,1980) There exist an edge between two points u,v

If for any point w dist(u,v) < min{dist(u,w),dist(w,v)} After the graph is built, connected nodes from

different classes are selected

u vw

Removed link

Centroid Selection

Relative Neighborhood Graph (Toussiant et al. 1980) There exists an edge between two points u,v if

for any point w, dist(u,v) < max{dist(u,w),dist(w,v)} After the graph is built, connected nodes from

different classes are selected

Combinatorial Centroid Selection Combinatorial Centroid Selection(CCS)

Given a training set of points (compounds) where distances satisfy triangle inequalities

Asked to find the minimum number of centroids (selected compounds) such that for each point, its nearest centroid is from same class

For simplicity, we only deal with binary classification i.e. C1 first class and C2 second class.

CCS Complexity

k-CCS problem Asked to select a set of points with cardinality less

than k such that for each point, its nearest centroid is from same class

k-CCS is NP-Complete K-Dominating Set (k-DS): given a graph G(V,E), ask

whether there exists V' ⊆ V, |V'| ≤ k and each node v∊V either exist in V' or it is adjecent to a node in V'

k-DS ≤p k-CCS

This reduction states no approximation better than O(log n) exists for CCS unless P = NP

Integer Linear Program Solution

Notations:

To minimize the number of chosen points or compounds (called centroids)

otherwise 0 and centroid a is if )( xx

11 C classin points are s'i

C

22 C classin points are s'j

C

Y and X points obetween tw distance metric a :),( YXd

minimize (C1i)

C1iC1

(C2 j)

C2 jC2

Integer Linear Program Solution

Ensure that for every pair of compounds i of class 1 and j of class 2, if j is chosen as a centroid, a compound k of class 1within the radius of between i and j should be chosen as a centroid as well.

11222),(),(

1

22112),(),(

1

,);()(

,);()(

..

1222

2111

CCCCCC

CCCCCC

ts

qpq

qplp

l

jij

jiki

k

CCdCCd

CCdCCd

Integer Linear Program Solution

Ensure that for each class there is a compound chosen as a centroid

21

2

1

};1,0{)(

1)(

1)(

..

22

11

CCXX

C

C

ts

CC

CC

i

i

i

i

Fixed Size Neighborhood Solution

ILP solution suffers from Huge size

due to pairwise constraints among points Potential trivial solution

Propose a relaxed version of ILP Reduce the number of constraints

for each point p within the radius equal to the distance from p to its k-th nearest neighbor of the different class there must be one centroid of same class of p

We will call this method CCNN1

Special case of CCS

When the majority of the compounds do not exhibit the bioactivity of interest All compounds that exhibit bioactivity of

interest are picked as centroids We minimize the number of compounds

chosen from compounds that does not exhibit the activity of interest

Special case of CCS

It can be reduced to Set Cover O(logn)-approximation algorithm

Set Cover problem Given a Universal Set (U) and a collection of

subsets (C) from U. Goal is to pick the minimum number of sets from C which cover all the elements in U.

NP-Complete Greedy Algorithm

Pick the set which cover the maximum number of uncoverd elements from the universal set

We will call this method CCNN2

Experimental Results - Datasets Mutageniticy dataset

includes aromatic and hetero-aromatic nitro compounds that are tested for mutagenicity on Salmonella

188 compounds with positive levels of log mutagenicity 63 negative examples

Drug dataset includes 958 drug compounds 6550 non-drug compounds including antibiotics,

human, bacterial, plant, fungal metabolites and drug-like compounds

Experimental Results - Descriptors The structures of the compounds have

been used 30 3D inductive QSAR descriptors by

Cherkasov et al. 2005 32 conventional QSAR by MOE:

Number of basic atoms Number of bonds ….

Comparison with other CBNN based methods Drug dataset

Method #Centroids

%Training Set

Accuracy

RNG 1705 28.39 89.00

GG 4804 79.99 92.00

CCNN 1489 24.79 89.89

CCNN2 1052 17.51 92.17

NN 6006 100 91.02

Comparison with small molecule classication methods Mutag Data set

Method Precision Recall Accuracy Running Time(min)

NN 87.80 92.00 86.17 1(?)

CCNN 92.00 92.74 89.94 6

CCNN2 92.13 94.35 90.91 6

SVM-Linear 92.00 92.00 89.36 6

SVM-ploy 91.30 92.00 88.83 6

SVM-Radial 86.60 92.80 85.63 6

Cao et.al. 88.2 77.8 82.35 20

MK Kashima et.al.

94.4 88.7 89.10 6

Comparison with small molecule classication methods Drug

Method Precision Recall Accuracy Running Time(min)

NN 64.70 65.30 91.02 45(?)

CCNN 56.36 61.18 89.89 181

CCNN2 69.12 69.70 92.17 150

SVM-Linear 76.10 8.70 87.89 121

SVM-Poly 77.10 38.30 90.17 180

SVM-Radial 80.10 35.00 90.60 121

Cao et.al. 81.20 56.20 92.00 ~5days

MK Kashima et.al.

53.70 57.00 89.10 ~1days

Learning the Metric Space

Emre Karakoc, Artem Cherkasov, S.Cenk Sahinalp (ISMB 2006)

Quantitative Structure-Activity Relationship(QSAR) Similarity measure

Minkowski distance

Each feature is equally significant But some features should be more

significant and some less Weighted Minkowski distance

ppn

ip iYiXL /1

1)|][][|(

|][][|),(1

1 iYiXwYXwLn

ii

Main Idea

Can weighted Minkowski be useful? Reduce the number of features.

PCA Increase the accuracy

How to learn the right W? Decrease the within-class distance Increase the between-class dist.

Learn the optimal W

Given the training set T let Active set Inactive set Min f(T)

f(T) =

},...,{ 21Am

AAA TTTT },...,{ 21

Iml

III TTTT

2

1 1 1

/|][][| miTiTw Aj

Ah

m

h

m

j

n

ii

2

1 1 1

)/()|][][|( mliTiTwml

h

nl

j

n

i

Ij

Ihi

))(/()|][][|(1 1 1

mlmiTiTwm

h

ml

j

n

i

Ij

Ahi

Learn the optimal W (cont.)

Min f(T) s.t

))(/(|][][|/|][][|1 1

2

1 1

mlmiTiTwmiTiTw Ij

Ah

ml

j

n

ii

Aj

Ah

m

j

n

ii

]1,0[iw

Metric Learning

Weinberger et al. NIPS 2006 Semidefinite program D(xi,xj) = (xi-xj)TM(xi-xj) where M = LTL

s.t. M > 0 The difference between between-class and

within-class distances is pre-fixed It aims to compute the

“best” M

Classification of new compounds Input:

Distances of new compound Q to the ones in the data-set

Assumption: Bioactivity level of Q is likely to be similar

to its close neighbors kNN classifier estimate the bioactivity of

Q: The majority bioactivity among its k-

nearest neighbors

Querying a compound

Naïve Method O(S) which S is the number element in

database. Binary search tree

Vantage Point (VP) tree (Uhlmann 1991) Binary tree that recursively partition data

space using distances of data points to randomly picked vantage point.

VP-Tree

Internal nodes: (Xvp, M, Rptr, Lptr)

M: median distance of among d(Xvp, Xi) for all Xi in the space partitioned.

Xvp: Vantage point.

Leaves: references to data points

Proximity search in VP-tree

Given a query point q, metric distance d(.,.) and a proximity radius r

Goal is to find all points x where d(x,q) < r If d(q,Xvp) – r < M recursively search the

inner partition If d(q, Xvp) + r > M recursively search the

outer partition Else search both

Can we do better?

Select multiple vantage points at each level Space Covering VP (SCVP) Trees (Sahinalp

et.al 2003) Increasing the chance of inclusion of query

in one of the inner partitions.

Can we do much better?

Instead of selecting random vantage points select them more intelligently Deterministic Multiple Vantage Point

(DMVP) Tree Select minimum number of multiple

vantage points that cover the entire data collection (OVPS problem)

Better space utilization (Optimal redundancy)

OVPS problem is NP-hard for any wLp

Conclusion

NN is powerful classifier Small molecule classification NN problem CBNN

CCNN1 and CCNN2

Distance learning Accuracy

DMVP tree

Future work

Further investigation of possible approximation algorithms for selecting centroids

Combining CCNN (selecting centroids) with metric learning

Ideally the problem formulation should ask to ensure the NN of each point in the training set is in the same class with that point

Adapt CCNN to work with regression datasets

References

Phuong Dao*, Farhad Hormozdiari*, Hossien Jowhari, Kendall Byler, Artem Cherkasov, S. Cenk Sahinalp, Improved Small Molecule Activity

Determination via Centroid Nearest Neighbors Classification, CSB 2008.

Emre Karakoc, Artem Cherkasov, S. Cenk Sahinalp Distance Based Algorithm for small Biomolecule

Classification and Structural Similarity Search, ISMB 2006 Lurii Sushko et.al. Applicability domains for classification

problems: benchmarking of distance to models for AMES mutagenicity

set, J. Chemical Informatics 2010.

Acknowledgments

Cenk Sahinalp Artem Cherkasov Zehra Cataltepe Emre Karakoc Phuong Dao Hossien Jowhari Kendall Byler All members of Lab

Questions