Linear Regression: Basis Functions, Vectorization

Robot Image Credit: Viktoriya Sukhanova © 123RF.com

These slides were assembled by Byron Boots, with grateful acknowledgement to Eric Eaton and the many others who made their course materials freely available online. Feel free to reuse or adapt these slides for your own academic purposes, provided that you include proper attribution.

Last Time: Linear Regression• Hypothesis:

• Fit model by minimizing sum of squared errors

2

x

y = ✓0 + ✓1x1 + ✓2x2 + . . .+ ✓dxd =dX

j=0

✓jxjy = ✓0 + ✓1x1 + ✓2x2 + . . .+ ✓dxd =dX

j=0

✓jxj

Figures are courtesy of Greg Shakhnarovich

Last Time: Gradient Descent• Initialize • Repeat until convergence

3

✓

✓j ✓j � ↵@

@✓jJ(✓) simultaneous update

for j = 0 ... d

J(✓)

✓

0

1

2

3

-0.5 0 0.5 1 1.5 2 2.5

↵

RegressionGiven:– Data where

– Corresponding labels where

4

0

1

2

3

4

5

6

7

8

9

1975 1980 1985 1990 1995 2000 2005 2010 2015

Sept

embe

r Arc

tic S

ea Ic

e Ex

tent

(1

,000

,000

sq k

m)

Year

Data from G. Witt. Journal of Statistics Education, Volume 21, Number 1 (2013)

Linear RegressionQuadratic Regression

X =nx(1), . . . ,x(n)

ox(i) 2 Rd

y =ny(1), . . . , y(n)

oy(i) 2 R

Extending Linear Regression to More Complex Models

• The inputs X for linear regression can be:– Original quantitative inputs– Transformation of quantitative inputs

• e.g. log, exp, square root, square, etc.

– Polynomial transformation• example: y = b0 + b1×x + b2×x2 + b3×x3

– Basis expansions– Dummy coding of categorical inputs– Interactions between variables

• example: x3 = x1 × x2

This allows use of linear regression techniques to fit non-linear datasets.

Linear Basis Function Models

• Generally,

• Typically, so that acts as a bias• In the simplest case, we use linear basis functions :

h✓(x) =dX

j=0

✓j�j(x)

�0(x) = 1 ✓0

�j(x) = xj

basis function

Based on slide by Christopher Bishop (PRML)

Linear Basis Function Models

– These are global; a small change in x affects all basis functions

• Polynomial basis functions:

• Gaussian basis functions:

– These are local; a small change in x only affect nearby basis functions. μj and s control location and scale (width).

Based on slide by Christopher Bishop (PRML)

Linear Basis Function Models• Sigmoidal basis functions:

where

– These are also local; a small change in x only affects nearby basis functions. μjand s control location and scale (slope).

Based on slide by Christopher Bishop (PRML)

Example of Fitting a Polynomial Curve with a Linear Model

y = ✓0 + ✓1x+ ✓2x2 + . . .+ ✓px

p =pX

j=0

✓jxj

Linear Basis Function Models

• Basic linear model:

• More general linear model:

• Once we have replaced the data by the outputs of the basis functions, fitting the generalized model is exactly the same problem as fitting the basic model– Unless we use the kernel trick – more on that when we

cover support vector machines

10

h✓(x) =dX

j=0

✓j�j(x)

h✓(x) =dX

j=0

✓jxj

Based on slide by Geoff Hinton