Math Primer

Outline

• Calculus: derivatives, chain rule, gradient descent, taylor expansions

• Bayes Rule

• Fourier Transform

• Dynamical linear systems

Calculus

• Derivatives

• Derivative=slope

0 0

lim limx x

y f x

dyf x

dxf x x f xy

x x

Calculus

• Derivative: a few common functions

• (xn)’=nxn-1

• (x-1)’=-1/x2 = x-2

• exp(x)’=exp(x)

• log(x)’=1/x

• cos(x)’=-sin(x)

Calculus

• Derivative: Chain rule

• Ex: gaussian• z=h(x)=exp(-x2)’,f(y)=exp(y), y=g(x)=-x2

• f’(y)=exp(y)= exp(-x2), g’(x)=-2x• z’= f’(y)g’(x) =exp(-x2)(-2x)

' '

z f y f g x h x

dz dz dyh x f y g x

dx dy dx

Calculus

• Finding minima: gradient descent

f(x) f’(x0) < 0

x0 x*x0+x

x > 0f’(x*)=0

x = -f’(x0)

Calculus

• Example: minimizing an error function

22

22

* *

* *

2 * '

i ii

i i i ij ji i j

ij

k kk l

kl k k kl

E y y v h y

v g net y v g w a y

h w

E E y h nety y v g a

w y h net w

Calculus

• Taylor expansion

0 00

0 0 0

20 0 0 0

1

2

f x x f xf x

xf x x f x f x x

f x x f x f x x f x x

Calculus

1 21 2

1

21 2

2 2 2 22 2

0 1 2 1 2 2 12 21 2 1 2 2 1

2 2

21

0

1

2

1

2

T

T

T T

f ff f x x

x x

xf ff

xx x

f f

f f f ff x f x x x x x x

x x x x x x

f f

xf x f

o o

o o

o o o o

o

0 0

x x

0

x x

0 0

0

x x x x

x

0

x δx x

x

x x δx

x δx

x δx δx

1 2

2 2

22 1 2

0

1

2

T T

x x

f f

x x x

f x f

o

o o

x

x x

0 0

δx

x δx δx H x δx

Bayes rule

• Example: drawing from 2 boxes• 2 boxes (B1,B2)• P(B1)=0.2,P(B2)=0.8 (Prior)

• Balls with two colors (R,G)• B1=(16R,8G), B2=(8R,16G)• P(R|B1)=2/3, P(G|B1)=1/3 (Conditional)• P(R|B2)=1/3, P(G|B2)=2/3 (Conditional)

Bayes rule

• Joint distributions

• P(G,B1)=P(B1)P(G|B1)=0.2*0.33=0.066

• P(G,B2)=P(B2)P(G|B2)=0.8*0.66=0.528

• P(X,Y)=P(X|Y)P(Y)

• P(Y,X)=P(Y|X)P(X)

• P(Y|X)P(X)=P(X|Y)P(Y)

Bayes rule

• Bayes rule

• P(Y|X)P(X)=P(X|Y)P(Y)

• P(Y|X)=P(X|Y)P(Y)/P(X)

• If you draw G, what is the probability that it came from box1?

• P(B1|G)=P(G|B1)P(B1)/P(G)

How do you get this? Marginalize

Bayes rule

Marginalization

• P(G)=P(G,B1)+P(G,B2)=0.066+0.528=0.6

• P(G)=P(G|B1)P(B1)+P(G|B2)P(B2)

• P(Y)=x P(Y,X)

• P(Y)=x P(Y|X)P(X)

• P(Y|X)=P(X|Y)P(Y)/YP(X|Y)P(Y)

Bayes rule

• Bayes rule

• If you draw G, what is the probability that it came from Box1 or Box2?

• P(B1|G)=P(G|B1)P(B1)/P(G)

=(0.33*0.2)/0.6=0.11

• P(B2|G)=P(G|B2)P(B2)/P(G)

=(0.66*0.8)/0.6=0.89

Sum to one

Bayes rule

• P(A,B|C)=P(A|B,C)P(B|C)

• P(B|A,C)=P(A|B,C)P(B|C)/P(A|C)

Fourier transform

• Basis in linear algebra

• Basis function: dirac

• Basis function: sin

Fourier Transform

• Decomposition in sum of sin and cosine

• Power: first term is the DC

• Phase

• Fourier transform for

Dirac

Sin

Gaussian (inverse relationship)

Fourier Transform

• Convolution and products

Fourier transform

• Fourier transform of a Gabor

Fourier transform

• Eigenspace for liner dynamical system…

Dynamical systems

• Stable if <0, unstable otherwise

0 exp

x x

x t x t

Dynamical systems

0

0

if 0

let

then

o

o o

o

x f x

f x

x x x

x x

x f x x

x f x f x x

x f x x

Fixed Point

Dynamical systems

ox f x x

Stable if f’(x0)<0, unstable otherwise.

Dynamical systems

x F x

1 1 2

2 1 2

,

,

f x x

f x x

F x

Dynamical systems

0 0

0

1 0 1 0

1 21 1

2 22 0 2 0

1 2

1 0 1 01 1 2

1 2

f f

x xx x

x xf f

x x

f fx x x

x x

x F x F x x

F x x J x

x x

x x

x x

1 0 1 0

1 2

2 0 2 0

1 2

f f

x x

f f

x x

x x

Jx x

Dynamical systems

• go into eigen space• Equations decouple

1 1 1 2 2 2

1

1

1 1 1

1 1 1

,

0 exp

if x

x

x x

x t x t

Je e Je e

x e

x J e

e e

e e

Stable if <0, unstable otherwise.

Dynamical systems

1 0 1 0

1 21 1

2 22 0 2 0

1 2

1 11

2 2

1 1 11

2 2 2

1 11

2 22

1 1 1

0

0

0

0

0

0e e

e e

e e

f f

x xx x

x xf f

x x

x

x

x x

x x

x x

x x

x x

x x

x x

P Ρ

Ρ ΡP Ρ

2 2 2e ex x

1

11

2

11 2112

12 22

0

0

e e

e e

1

J P ΛΡ

P Ρ

P e e

Dynamical systems

• go into eigen space• Equations decouple

1 1 1 2 2 2

1 1 2 2

1 1 1 2 2 2

1 1 2 2

,

exp exp

x x

x x

t t t

Je e Je e

x J e e

x e e

x e e

Dynamical systems

• go into eigen space• Equations decouple

1 1 1

2 2 2

0

0

e e

e e

Stable is f’(x0)<0, unstable otherwise.

Dynamical systems

• Fixed point• Saddle point• Unstable point• Stable and unstable

oscillations: complex eigenvalues

1 1 1

2 2 2

0

0

e e

e e

Nonlinear Networks

• Discrete case:

Stable if ||<1, unstable otherwise

1

1

0t

x t x t

x t x

Nonlinear Networks

• Discrete case:

1

1

1

1

1

1

H( )

H

H( )

i

n

ij jj

t t

t t

t t

t t

o t h w o t

O WO

O O W O O

O O WO J O

O J O

Nonlinear Networks

• Dynamics around attractor:

1

1

1

0 0

0 ... 0

0 0

t t

n

t t

-1

O J O

J = P ΛP

Λ

P O ΛP O

Nonlinear Networks

• Stable Fixed point: |1|<1, |2|<1

Nonlinear Networks

• Saddle Point: |1|>1, |2|<1

Nonlinear Networks

• Unstable Fixed point: |1|>1, |2|>1

Nonlinear Networks

• Line Attractor: 1=1, |2|<1

Nonlinear Networks

• Oscillation: complex ’s

Nonlinear Networks: global stability

• Lyapunov Function: function of the state of the system which is bounded below and goes down over time. If such a function exists, the system is globally stable.

• Ex: Hopfield network, Cohen-Grossberg network