Learning with General Similarity Functions

Maria-Florina Balcan

2-Minute VersionGeneric classification problem:Problem: pixel representation not so good.Powerful technique: use a kernel, a special kind of similarity function K( , )

.

Develop a theory that views K as a measure of similarity. General sufficient conditions for K to be useful for learning.

But, standard theory in terms of implicit mappings.

[Balcan-Blum, ICML 2006] [Balcan-Blum-Srebro, MLJ 2008]

Our Work:

[Balcan-Blum-Srebro, COLT 2008]

3

Kernel Methods

A kernel K is a legal def of dot-product: i.e. there exists an implicit mapping such that K( , )= ( )¢ ( ).

E.g., K(x,y) = (x ¢ y + 1)d

(n-dimensional space) ! nd-dimensional space

Why Kernels matter? Many algorithms interact with data only via dot-products.

So, if replace x ¢ y with K(x,y), they act implicitly as if data was in the higher-dimensional -space.

What is a Kernel?

Prominent method for supervised classification today.The learning alg. interacts with the data via a similarity fns

4

x2

x1

OO O

OO

OO O

XX

X

X

XX

X

X X

X

X

X

X

XX

X

XX

z1

z3

OO

O O

O

O

O

OO

X XX X

X

X

X

X X

X

X

X

X

X

X

X X

X

ExampleE.g., for n=2, d=2, the kernel

z2

K(x,y) = (x¢y)d corresponds to

original space space

5

Generalize Well if Good Margin

• If data is linearly separable by margin in -space, then good sample complexity.

|(x)| · 1

+++

++

+

---

--

If margin in -space, then

need sample size of only Õ(1/2) to get confidence in generalization.

6

Kernel Methods

Significant percentage of ICML, NIPS, COLT.

Very useful in practice for dealing with many different types of data.

Prominent method for supervised classification today

7

Limitations of the Current Theory

Existing Theory: in terms of margins in implicit spaces.

In practice: kernels are constructed by viewing them as measures of similarity.

Kernel requirement rules out many natural similarity functions.

Difficult to think about, not great for intuition.

Better theoretical explanation?Better theoretical explanation?

8

Better Theoretical Framework

Existing Theory: in terms of margins in implicit spaces.

In practice: kernels are constructed by viewing them as measures of similarity.

Kernel requirement rules out natural similarity functions.

Difficult to think about, not great for intuition.

Yes! We provide a more general and intuitive theory that formalizes the intuition that a good kernel is a good measure of similarity.

Better theoretical explanation?Better theoretical explanation?

[Balcan-Blum, ICML 2006][Balcan-Blum-Srebro, MLJ 2008]

[Balcan-Blum-Srebro, COLT 2008]

9

More General Similarity Functions

2) Is broad: includes usual notion of good kernel.

We provide a notion of a good similarity function:

1) Simpler, in terms of natural direct quantities.• no implicit high-dimensional spaces• no requirement that K(x,y)=(x) ¢ (y)

K can be used to learn well.

has a large margin sep. in -space

Good kernels

First attempt

Main notion

3) Allows one to learn classes that have no good kernels.

10

A First Attempt

K is (,)-good for P if a 1- prob. mass of x satisfy:

Ey~P[K(x,y)|l(y)=l(x)] ¸ Ey~P[K(x,y)|l(y)l(x)]+

P distribution over labeled examples (x, l(x))

K is good if most x are on average more similar to points y of their own type than to points y of the

other type.

Goal: output classification rule good for P

Average similarity to points of opposite label

gap Average similarity to

points of the same label

11

A First Attempt

E.g., K(x,y) ¸ 0.2, l(x) = l(y) K(x,y) random in {-1,1}, l(x)

l(y)

-1

11

0.50.4Example:

K is (,)-good for P if a 1- prob. mass of x satisfy:

Ey~P[K(x,y)|l(y)=l(x)] ¸ Ey~P[K(x,y)|l(y)l(x)]+

0.3

12

A First Attempt

Algorithm• Draw sets S+, S- of positive and negative examples.

• Classify x based on average similarity to S+ versus to S-.

K is (,)-good for P if a 1- prob. mass of x satisfy:

S+-1

11

0.50.4

S-

xx

Ey~P[K(x,y)|l(y)=l(x)] ¸ Ey~P[K(x,y)|l(y)l(x)]+

13

A First Attempt K is (,)-good for P if a 1- prob. mass of x satisfy:

Ey~P[K(x,y)|l(y)=l(x)] ¸ Ey~P[K(x,y)|l(y)l(x)]+

Theorem

Algorithm• Draw sets S+, S- of positive and negative examples.

• Classify x based on average similarity to S+ versus to S-.

• For a fixed good x prob. of error w.r.t. x (over draw of S+, S-) is ± ²’. [Hoeffding]

• Overall error rate · +’.

If |S+| and |S-| are ((1/2) ln(1/’)), then with probability ¸ 1-, error · +’.

• At most chance that the error rate over GOOD is ¸ ’.

14

A First Attempt: Not Broad EnoughEy~P[K(x,y)|l(y)=l(x)] ¸ Ey~P[K(x,y)|l(y)l(x)]

+

• has a large margin separator;

+ +++++

-- -- --

more similar to - than to typical +

½ versus ¼ 30o

30o

Similarity function K(x,y)=x ¢ ydoes not satisfy our definition.

½ versus ½ ¢ 1 + ½ ¢ (- ½)

15

A First Attempt: Not Broad EnoughEy~P[K(x,y)|l(y)=l(x)] ¸ Ey~P[K(x,y)|l(y)l(x)]

+

+ +++++

-- -- --

R

Broaden: 9 non-negligible R s.t. most x are on average more similar to y 2 R of same label than to y 2 R of other label.[even if do not know R in advance]

30o

30o

16

Broader Definition K is (, , ) if 9 a set R of “reasonable” y (allow probabilistic) s.t. 1-

fraction of x satisfy:

• Draw S={y1, , yd} set of landmarks. F(x) = [K(x,y1), …,K(x,yd)].

RdFF(P)

Property

x !

• If enough landmarks (d=(1/2 )), then with high prob. there exists a good L1 large margin linear separator.

Re-represent data.

w=[0,0,1/n+,1/n+,0,0,0,-1/n-,0,0]w=[0,0,1/n+,1/n+,0,0,0,-1/n-,0,0]

P

At least prob. mass of reasonable positives & negatives.Ey~P[K(x,y)|l(y)=l(x), R(y)] ¸ Ey~P[K(x,y)|l(y)l(x), R(y)]

+

17

Broader Definition

• Draw S={y1, , yd} set of landmarks. F(x) = [K(x,y1), …,K(x,yd)]

RdFF(P)

Algorithm

x !Re-represent data.

OO

OOOX

XXX

X

X X X XX

O O OO O

• Take a new set of labeled examples, project to this space, and and run a good Lrun a good L11 linear separator alg. linear separator alg.

P

K is (, , ) if 9 a set R of “reasonable” y (allow probabilistic) s.t. 1- fraction of x satisfy:

du=Õ(1/(2 ))dl=O(1/(2²acc ln (du) ))

At least prob. mass of reasonable positives & negatives.Ey~P[K(x,y)|l(y)=l(x), R(y)] ¸ Ey~P[K(x,y)|l(y)l(x), R(y)]

+

18

Kernels versus Similarity Functions

TheoremK is also a good similarity function.

Main Technical ContributionsOur Work

Good Kernels

Good Similarities

(but gets squared).

K is a good kernel

If K has margin in implicit space, then for any ,K is (,2,)-good in our sense.

19

Kernels versus Similarity Functions

Can also show a Strict Separation.

Main Technical ContributionsOur Work

Good Kernels

Good Similarities

Strictly more general

For any class C of n pairwise uncorrelated functions, 9 a similarity function good for all f in C, but no such good kernel function exists.

Theorem

TheoremK is also a good similarity function.

(but gets squared).

K is a good kernel

20

Kernels versus Similarity FunctionsCan also show a Strict

Separation.

For any class C of n pairwise uncorrelated functions, 9 a similarity function good for all f in C, but no such good kernel function exists.

Theorem

• In principle, should be able to learn from O(-1log(|C|/)) labeled examples.

• Claim 1: can define generic (0,1,1/|C|)-good similarity function achieving this bound. (Assume D not too concentrated)

• Claim 2: There is no (,) good kernel in hinge loss, even if =1/2 and =1/ |C|-1/2. So, margin based SC is d=(1/|C|).

21

Learning with Multiple Similarity Functions

• Let K1, …, Kr be similarity functions s. t. some (unknown) convex combination of them is (,)-good.

Guarantee: Whp the induced distribution F(P) in R2dr has a separator of error · + at L1 margin at least

Algorithm• Draw S={y1, , yd} set of landmarks. Concatenate features.

Sample complexity only increases by log(r) factor!

F(x) = [K1(x,y1), …,Kr(x,y1), …, K1(x,yd),…,Kr(x,yd)].

Conclusions• Theory of learning with similarity fns that provides a

formal way of understanding good kernels as good similarity fns.

• Our algorithms work for similarity fns that aren’t necessarily PSD (or even symmetric).

Algorithmic Implications

• Can use non-PSD similarities, no need to “transform” them into PSD functions and plug into SVM.

E.g., Liao and Noble, Journal of Computational Biology

Conclusions• Theory of learning with similarity fns that provides a

formal way of understanding good kernels as good similarity fns.

• Our algorithms work for similarity fns that aren’t necessarily PSD (or even symmetric).

Open Questions• Analyze other notions of good similarity fns.

Our Work

Good Kernels

24

25

Similarity Functions for Classification Algorithmic Implications• Can use non-PSD similarities, no need to “transform” them into PSD functions and plug into SVM.

E.g., Liao and Noble, Journal of Computational Biology

• Give justification to the following rule:

• Also show that anything learnable with SVM is learnable this way!