Lecture VII: MLSC - Dr. Sethu Vijayakumar 1

Lecture VII– Regression(Nonlinear/Nonparametric Methods)

Contents:• Local Weighted and Lazy learning techniques• Nonparametric methods

Lecture VII: MLSC - Dr. Sethu Vijayakumar 2

Nonparametric Methods

Working DefinitionThe name “nonparametric” is to indicate that the data to be modeled stem from very large families of distributions which cannot be indexed by a finite dimensional parameter vector in a natural way.

Remarks this does not mean that nonparametric methods have no parameters!nonparametric methods avoid making assumptions about the parametric form of the underlying distributions (except some smoothness properties).nonparametric methods are often memory-based (but not necessarily)sometimes called “lazy learning”

Can be applied to density estimationclassification regression

Lecture VII: MLSC - Dr. Sethu Vijayakumar 3

Locally Weighted Regression (LWR)

Fit locally lower order polynomials, e.g., first order polynomials

Approximate non-linear functions with a mixture of k piecewise linear models

Lecture VII: MLSC - Dr. Sethu Vijayakumar 4

Minimize Weighted Squared Error

Solution: Weighted Least Squares

Weight can be calculated from any weighting kernel, e.g., a Gaussian:

LWR: Formalization

J = wn t n − y n( )T

n= 1

N

∑ t n − y n( ), where y n = x n T β

( )

1

21

0 0 00 0 0

where 0 0 00 0 0

T T

n

ww

w

β−

⎡ ⎤⎢ ⎥⎢ ⎥= =⎢ ⎥…⎢ ⎥⎢ ⎥⎣ ⎦

X WX X WY W

w = exp −12

x − c( )T D x − c( )⎛ ⎝

⎞ ⎠

Lecture VII: MLSC - Dr. Sethu Vijayakumar 5

LWR: Examples (cont’d)

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

D=10.00

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

D=50.00

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

D=100.00

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

D=200.00

The fits exhibit the bias-variance tradeoff effect with respect to parameter D.

Lecture VII: MLSC - Dr. Sethu Vijayakumar 6

How to Optimize the Distance Metric?

Global Optimizationfind the most suitable D for the entire data set

Local Optimizationfind the most suitable D as a function of the kernel location c

Two possible options :

Lecture VII: MLSC - Dr. Sethu Vijayakumar 7

Global Distance Metric Optimization

Leave-one-out Cross Validationcompute the prediction of every data point in the training set by:

centering the kernel at every data pointbut excluding the data point at the center from the training set

find distance metric that minimizes the cross validation errordepending on how many parameters are allowed in the distance metric, this is a multidimensional optimization problem

NOTE: Leave-one-out Cross Validation is very cheap in nonparametric methods (in comparison to most parametric methods) since there is no iterative training of the learning system

J c = t n − y − nn( )T

n =1

N

∑ t n − y − nn( )

Lecture VII: MLSC - Dr. Sethu Vijayakumar 8

Why Cross Validation ?

X

y

w

X

y

w

Without Cross Validation, the kernel could just shrink to zero and focus on one data point only.

*** Avoids degenerate Solutions for D

Lecture VII: MLSC - Dr. Sethu Vijayakumar 9

0 50 100 150 200 250 3000.24

0.245

0.25

0.255

0.26

0.265

0.27

0.275

0.28

D

Global Optimization of Distance Metric: Example

-3 -2 -1 0 1 2 3-3

-2

-1

0

1

2

3

x

D=100.00

Locally Weighted Cost functionResultant fit from Global Optimization

Lecture VII: MLSC - Dr. Sethu Vijayakumar 10

Local Distance Metric Optimization

Why not optimize the distance metric as a function of the location of the kernel center?

Local Cross Validation Criterion

Something Exceptionally Cool: The local leave-one-out cross validation error can be computed analytically—WITHOUT an n-fold recalculation of the prediction for linear local models !

Jc = w n t n − y − nn( )T

n =1

N

∑ t n − y − nn( )

where β = X T WX( )−1X T WY = PXT WY

( ) ( ) ( ) ( )( )2

1 1 1

Tn n n n nN NTn n n n nc n n Tn n nn n

wJ w

w P− −

= =

− −= − − =

−∑ ∑

t y t yt y t y

x xa.k.a.:PressResidualError

Lecture VII: MLSC - Dr. Sethu Vijayakumar 11

A Nonparametric Regression Network

Ideas:Create new (Gaussian) kernels as needed (i.e., when no other kernel in the net is activated sufficiently (=> a constructive network)update the linear model in each kernel by weighted recursive least squaresadjust the distance metric by gradient descent in local cross validation criteriaThe weighted output of all kernels is the prediction

Receptive Field Weighted Regression (RFWR)

Lecture VII: MLSC - Dr. Sethu Vijayakumar 12

Nonparametric Regression Network (cont’d)

y = βxTx +β0 = βT ˜ x where ˜ x = xT 1[ ]T

w = exp −12

x−c( )T D x− c( )⎛ ⎝

⎞ ⎠ where D= MTM

J =1

w ii= 1

p

∑w i y i − ˆ y i , − i

2

i= 1

p

∑ + γ D ij2

i=1 , j =1

n

∑

Elements of each Local Linear Model

Penalized local cross validation error

Regression Slope

Receptive Field

Lecture VII: MLSC - Dr. Sethu Vijayakumar 13

Nonparametric Regression Network (cont’d)

Update of the parameters:Slope of local model :

Distance Metric Adaptation :

( )

1 1

1

1where and

n n n Tcv

n T nn n T n

cvT n

w

w

β β

βλλ

+ +

+

= +

⎛ ⎞⎜ ⎟

= − = −⎜ ⎟⎜ ⎟+⎜ ⎟⎝ ⎠

P x e

P x x PP P e y xx P x

%

% %%

% %

M n +1 = M n − α∂JdM

Another very cool thing: For linear systems, leave-one-out cross validation can be approximated INCREMENTALLY!Thus, no data has to be kept in memory!

Lecture VII: MLSC - Dr. Sethu Vijayakumar 14

Example of learning

⋅⋅

⋅

⋅⋅⋅⋅⋅

⋅

⋅⋅⋅

⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅

⋅

⋅⋅⋅⋅⋅⋅⋅

⋅

⋅

⋅⋅⋅⋅⋅⋅⋅

⋅⋅

⋅⋅⋅⋅⋅

⋅⋅⋅⋅

⋅⋅

⋅

⋅

⋅⋅⋅⋅⋅

⋅

⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅

⋅⋅⋅

⋅⋅⋅

⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅

⋅

⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅

⋅⋅

⋅⋅

⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅

⋅⋅⋅⋅⋅

⋅⋅

⋅⋅⋅⋅⋅⋅⋅

⋅

⋅⋅⋅

+

++

+

+

+

+

+

++

+

++

++

+

+

++

++

+

+

+

+++

+

++

++

+++

+

++++

+++

++++

+

++

++

+

+++

++++

+++

++

+

+

+

++

-2

-1

0

1

2

3

4

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

y

x

0

0.5

1

-2.5 -2 -1.5 -1 -0.5 0 0.5 1 1.5 2 2.5

w

x

Lecture VII: MLSC - Dr. Sethu Vijayakumar 15

A 2D Example

-1

-0.5

0

0.5

1

x

-1

-0.5

0

0.5

1

y

-0.5

0

0.5

1

1.5

z

-1

-0.5

0

0.5

1

x

-1

-0.5

0

0.5

1

y

-0.5

0

0.5

1

1

Actual Data Generating function Sampled Data with noise

Why is this a tough function to learn ??

Lecture VII: MLSC - Dr. Sethu Vijayakumar 16

-1

-0.5

0

0.5

1

x

-1

-0.5

0

0.5

1

y

-0.5

0

0.5

1

1.5

z

-1

-0.5

0

0.5

1

x

-1

-0.5

0

0.5

1

y

-0.5

0

0.5

1

1

-1

-0.5

0

0.5

1

x

-1

-0.5

0

0.5

1

y

-0.5

0

0.5

1

1.5

z

-1

-0.5

0

0.5

1

x

-1

-0.5

0

0.5

1

y

-0.5

0

0.5

1

1

⊕

⊕

⊕

⊕

⊕

⊕⊕

⊕

⊕ ⊕⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕

⊕⊕

⊕

⊕⊕

⊕

-1.5

-1

-0.5

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5

y

x

⊕

⊕

⊕

⊕

⊕

⊕⊕

⊕

⊕ ⊕⊕

⊕

⊕

⊕

⊕

⊕

-1.5

-1

-0.5

0

0.5

1

1.5

-1.5 -1 -0.5 0 0.5 1 1.5

y

x

A 2D Example (cont’d)Results after one Epoch of Training Results after 50 Epochs of Training

Lecture VII: MLSC - Dr. Sethu Vijayakumar 17

Nonparametric Regression Network (Summary)

The LWR scheme we developed (RFWR)--can incrementally deal with the bias-variance dilemmagrows with the data (constructive)learns very fastis similar to a mixture of experts, but does not need a pre-specified number of experts (no competitive learning)is similar to committee networks (by averaging the outputs over many independently trained networks)

… but still has problems with the curse of dimensionality, as all spatially localized networks

Lecture VII: MLSC - Dr. Sethu Vijayakumar 18

Handling curse of Dimensionality

We developed a method to make Local Learning methods like RFWR scale

…the resultant system is called Locally Weighted Projection Regression (LWPR)

LWPR module

We will learn more about this and other

dimensionality reduction techniques

in the next class.