LMS Algorithm in a Reproducing Kernel Hilbert Space

LMS Algorithm in a Reproducing Kernel Hilbert Space

Weifeng Liu, P. P. Pokharel, J. C. Principe

Computational NeuroEngineering Laboratory,

University of Florida

Acknowledgment: This work was partially supported by NSF grant ECS-0300340 and ECS-0601271.

Outlines

Introduction Least Mean Square algorithm (easy) Reproducing kernel Hilbert space (tricky) The convergence and regularization analysis

(important) Learning from error models (interesting)

Introduction

Puskal (2006) –Kernel LMS

Kivinen, Smola (2004) –Online learning with kernels (more like leaky LMS)

Moody, Platt (1990’s)—Resource allocation networks (growing and pruning)

LMS (1960, Widrow and Hoff)

Given a sequence of examples from U×R:

U: a compact set of RL. The model is assumed:

The cost function:

1 1(( , ),..., ( , ))N Nu y u y

)()( nvuwy no

n

2

1

1( ) ( ( ))

N

n ni

J w y w uN

LMS

The LMS algorithm

The weight after n iteration:

0

1

1

0

( )an n n n

an n n n

w

e y w u

w w e u

1

n an i iiw e u

(1)

(2)

Reproducing kernel Hilbert space

A continuous, symmetric, positive-definite kernel ,a mapping Φ, and an inner product

H is the closure of the span of all Φ(u). Reproducing Kernel trick The induced norm

:U U R

, H

, ( ) ( )Hf u f u

1 2 1 2( ), ( ) ( , )Hu u u u 2|| || ,H Hf f f

RKHS

Kernel trick: – An inner product in the feature space– A similarity measure you needed.

Mercer’s theorem:T

M uuuu )](),...,(),([)( 21

Common kernels

Gaussian kernel

Polynomial kernel

)||||exp(),( 2jiji uuauu

( , ) ( 1)T pi j i ju u u u

Kernel LMS

Transform the input ui to Φ(ui):

Assume Φ(ui) R∈ M

The model is assumed:

The cost function:

1 1(( ( ), ),..., ( ( ), ))N Nu y u y

( ( )) ( )on ny u v n

2

1

1( ) ( ( ( )))

N

n ni

J y uN

Kernel LMS

The KLMS algorithm

The weight after n iteration:

0

1

1

0

( ( ))

( )

an n n n

an n n n

e y u

e u

1( )

n an i ii

e u

(3)

(4)

Kernel LMS

1

1

1

1

1

1

1

( ( ))

( ), ( )

( , ),

( ( )),

( ).

n n

n ai i ni H

n ai i ni

an n n n

n an i ii

u

e u u

e u u

e y u

e u

(5)

Kernel LMS

After the learning, the input-output relation:

1

( ( ))

( , )

N

N ai ii

y u

e u u

(6)

KLMS vs. RBF

KLMS:

RBF:

α satisfy

G is the gram matrix: G(i,j)=ĸ(ui,uj) RBF needs regularization. Does KLMS need regularization?

1( ) ( , )

N ai ii

y e u u

(7)

1( , )

N

i iiy u u

(8)

G y

KLMS vs. LMS

Kernel LMS is nothing but LMS in the feature space--a very high dimensional reproducing kernel Hilbert space (M>N)

Eigen-spread is awful—does it converge?

Example: MG signal predication

Time embedding: 10.

Learn rate: 0.2 500 training data 100 test data point. Gaussian noise noise variance: .04

0 100 200 300 400 5000

0.02

0.04

0.06

0.08

0.1mse linearmse kernel

Example: MG signal predication

MSE Linear LMS

KLMS RBF (λ=0)

RBF (λ=.1)

RBF (λ=1)

RBF (λ=10)

training 0.021 0.0060 0 0.0026 0.0036 0.010

test 0.026 0.0066 0.019 0.0041 0.0050 0.014

Complexity Comparison

RBF KLMS LMS

Computation O(N3) O(N2) O(L)

Memory O(N2+N*L) O(N*L) O(L)

The asymptotic analysis on convergence—small step-size theory

Denote The correlation matrix

is singular. Assume

and

1

1 NT

x i ii

R x xN

Mii Rux )(

1 1... ... 0k k M T

xR P P

The asymptotic analysis on convergence—small step-size theory

Denote

we have1( )

Mon i ii

n P

[ ( )] (1 ) (0)ni i iE n

2 2 2min min[| ( ) | ] (1 ) (| (0) | )2 2

ni i i

i i

J JE n

The weight stays at the initial place in the 0-eigen-value directions

If

we have

0i

[ ( )] (0)i iE n

2 2[| ( ) | ] | (0) |i iE n

The 0-eigen-value directions does not affect the MSE

Denote

2 2min minmin 1 1

( ) (| (0) | )(1 )2 2

M M ni i i ii i

J JJ n J

2( ) [| ( ) | ]iJ n E y x

It does not care about the null space! It only focuses on the data space!

The minimum norm initialization

The initialization gives the minimum norm possible solution.

00

1ˆ

M

n i iiw P

2 2 2 2

1 1 1ˆ ˆ ˆ|| || || || || || || ||

M k M

n i i ii i i kw w w

Minimum norm solution

0 2 4-1

0

1

2

3

4

5

Learning is Ill-posed

Over-learning

Regularization Technique

Learning from finite data is ill-posed. A priori information--Smoothness is needed. The norm of the function, which indicates the

‘slope’ of the linear operator is constrained. In statistical learning theory, the norm is

associated with the confidence of uniform convergence!

Regularized RBF

The cost function:

or equivalently2 2

1

1( ) ( ( ( ))) || ||

N

n ni

J y uN

2

1

2

1( ) ( ( ( )))

subject to || ||

N

n ni

J y uN

C

KLMS as a learning algorithm

The model with The following inequalities hold

The proof…(H∞ robust + triangle inequality + matrix transformation + derivative + …)

( ) ( )on ny x v n ( )n nx u

2 1 2 2|| || || || 2 || ||a oe v

2 2|| || 2 || ||ae y

The solution of regularized RBF is

The reason of ill-posedness is the inversion of the matrix (G+λI)

The numerical analysis

1( , )

N

i iiy u u

1( )G I y

1 1|| ( ) || as 0G I

The numerical analysis

The solution of KLMS is

By the inequality we have

ae Ly

1( , )

N ai ii

y e u u

|| || 2L

Example: MG signal predication

weight KLMS RBF (λ=0)

RBF (λ=.1)

RBF (λ=1)

RBF (λ=10)

norm 0.520 4.8e+3 10.90 1.37 0.231

The conclusion

The LMS algorithm can be readily used in a RKHS to derive nonlinear algorithms.

From the machine learning view, the LMS method is a simple tool to have a regularized solution.

Demo

Demo

LMS learning model

An event happens, and a decision made. If the decision is correct, nothing happens. If an error is incurred, a correction is made

on the original model. If we do things right, everything is fine and

life goes on. If we do something wrong, lessons are drawn

and our abilities are honed.

Would we over-learn?

If the real world is attempted to be modeled mathematically, what dimension is appropriate?

Are we likely to over-learn? Are we using the LMS algorithm? What is good to remember the past? What is bad to be a perfectionist?

"If you shut your door to all errors, truth will be shut out."---Rabindranath Tagore