Upload
elwyn
View
35
Download
0
Tags:
Embed Size (px)
DESCRIPTION
Math Primer. Outline. Calculus: derivatives, chain rule, gradient descent, taylor expansions Bayes Rule Fourier Transform Dynamical linear systems. Calculus. Derivatives Derivative=slope. Calculus. Derivative: a few common functions (x n )’=nx n-1 (x -1 )’=-1/x 2 = x -2 - PowerPoint PPT Presentation
Citation preview
Math Primer
Outline
• Calculus: derivatives, chain rule, gradient descent, taylor expansions
• Bayes Rule
• Fourier Transform
• Dynamical linear systems
Calculus
• Derivatives
• Derivative=slope
0 0
lim limx x
y f x
dyf x
dxf x x f xy
x x
Calculus
• Derivative: a few common functions
• (xn)’=nxn-1
• (x-1)’=-1/x2 = x-2
• exp(x)’=exp(x)
• log(x)’=1/x
• cos(x)’=-sin(x)
Calculus
• Derivative: Chain rule
• Ex: gaussian• z=h(x)=exp(-x2)’,f(y)=exp(y), y=g(x)=-x2
• f’(y)=exp(y)= exp(-x2), g’(x)=-2x• z’= f’(y)g’(x) =exp(-x2)(-2x)
' '
z f y f g x h x
dz dz dyh x f y g x
dx dy dx
Calculus
• Finding minima: gradient descent
f(x) f’(x0) < 0
x0 x*x0+x
x > 0f’(x*)=0
x = -f’(x0)
Calculus
• Example: minimizing an error function
22
22
* *
* *
2 * '
i ii
i i i ij ji i j
ij
k kk l
kl k k kl
E y y v h y
v g net y v g w a y
h w
E E y h nety y v g a
w y h net w
Calculus
• Taylor expansion
0 00
0 0 0
20 0 0 0
1
2
f x x f xf x
xf x x f x f x x
f x x f x f x x f x x
Calculus
1 21 2
1
21 2
2 2 2 22 2
0 1 2 1 2 2 12 21 2 1 2 2 1
2 2
21
0
1
2
1
2
T
T
T T
f ff f x x
x x
xf ff
xx x
f f
f f f ff x f x x x x x x
x x x x x x
f f
xf x f
o o
o o
o o o o
o
0 0
x x
0
x x
0 0
0
x x x x
x
0
x δx x
x
x x δx
x δx
x δx δx
1 2
2 2
22 1 2
0
1
2
T T
x x
f f
x x x
f x f
o
o o
x
x x
0 0
δx
x δx δx H x δx
Bayes rule
• Example: drawing from 2 boxes• 2 boxes (B1,B2)• P(B1)=0.2,P(B2)=0.8 (Prior)
• Balls with two colors (R,G)• B1=(16R,8G), B2=(8R,16G)• P(R|B1)=2/3, P(G|B1)=1/3 (Conditional)• P(R|B2)=1/3, P(G|B2)=2/3 (Conditional)
Bayes rule
• Joint distributions
• P(G,B1)=P(B1)P(G|B1)=0.2*0.33=0.066
• P(G,B2)=P(B2)P(G|B2)=0.8*0.66=0.528
• P(X,Y)=P(X|Y)P(Y)
• P(Y,X)=P(Y|X)P(X)
• P(Y|X)P(X)=P(X|Y)P(Y)
Bayes rule
• Bayes rule
• P(Y|X)P(X)=P(X|Y)P(Y)
• P(Y|X)=P(X|Y)P(Y)/P(X)
• If you draw G, what is the probability that it came from box1?
• P(B1|G)=P(G|B1)P(B1)/P(G)
How do you get this? Marginalize
Bayes rule
Marginalization
• P(G)=P(G,B1)+P(G,B2)=0.066+0.528=0.6
• P(G)=P(G|B1)P(B1)+P(G|B2)P(B2)
• P(Y)=x P(Y,X)
• P(Y)=x P(Y|X)P(X)
• P(Y|X)=P(X|Y)P(Y)/YP(X|Y)P(Y)
Bayes rule
• Bayes rule
• If you draw G, what is the probability that it came from Box1 or Box2?
• P(B1|G)=P(G|B1)P(B1)/P(G)
=(0.33*0.2)/0.6=0.11
• P(B2|G)=P(G|B2)P(B2)/P(G)
=(0.66*0.8)/0.6=0.89
Sum to one
Bayes rule
• P(A,B|C)=P(A|B,C)P(B|C)
• P(B|A,C)=P(A|B,C)P(B|C)/P(A|C)
Fourier transform
• Basis in linear algebra
• Basis function: dirac
• Basis function: sin
Fourier Transform
• Decomposition in sum of sin and cosine
• Power: first term is the DC
• Phase
• Fourier transform for
Dirac
Sin
Gaussian (inverse relationship)
Fourier Transform
• Convolution and products
Fourier transform
• Fourier transform of a Gabor
Fourier transform
• Eigenspace for liner dynamical system…
Dynamical systems
• Stable if <0, unstable otherwise
0 exp
x x
x t x t
Dynamical systems
0
0
if 0
let
then
o
o o
o
x f x
f x
x x x
x x
x f x x
x f x f x x
x f x x
Fixed Point
Dynamical systems
ox f x x
Stable if f’(x0)<0, unstable otherwise.
Dynamical systems
x F x
1 1 2
2 1 2
,
,
f x x
f x x
F x
Dynamical systems
0 0
0
1 0 1 0
1 21 1
2 22 0 2 0
1 2
1 0 1 01 1 2
1 2
f f
x xx x
x xf f
x x
f fx x x
x x
x F x F x x
F x x J x
x x
x x
x x
1 0 1 0
1 2
2 0 2 0
1 2
f f
x x
f f
x x
x x
Jx x
Dynamical systems
• go into eigen space• Equations decouple
1 1 1 2 2 2
1
1
1 1 1
1 1 1
,
0 exp
if x
x
x x
x t x t
Je e Je e
x e
x J e
e e
e e
Stable if <0, unstable otherwise.
Dynamical systems
1 0 1 0
1 21 1
2 22 0 2 0
1 2
1 11
2 2
1 1 11
2 2 2
1 11
2 22
1 1 1
0
0
0
0
0
0e e
e e
e e
f f
x xx x
x xf f
x x
x
x
x x
x x
x x
x x
x x
x x
x x
P Ρ
Ρ ΡP Ρ
2 2 2e ex x
1
11
2
11 2112
12 22
0
0
e e
e e
1
J P ΛΡ
P Ρ
P e e
Dynamical systems
• go into eigen space• Equations decouple
1 1 1 2 2 2
1 1 2 2
1 1 1 2 2 2
1 1 2 2
,
exp exp
x x
x x
t t t
Je e Je e
x J e e
x e e
x e e
Dynamical systems
• go into eigen space• Equations decouple
1 1 1
2 2 2
0
0
e e
e e
Stable is f’(x0)<0, unstable otherwise.
Dynamical systems
• Fixed point• Saddle point• Unstable point• Stable and unstable
oscillations: complex eigenvalues
1 1 1
2 2 2
0
0
e e
e e
Nonlinear Networks
• Discrete case:
Stable if ||<1, unstable otherwise
1
1
0t
x t x t
x t x
Nonlinear Networks
• Discrete case:
1
1
1
1
1
1
H( )
H
H( )
i
n
ij jj
t t
t t
t t
t t
o t h w o t
O WO
O O W O O
O O WO J O
O J O
Nonlinear Networks
• Dynamics around attractor:
1
1
1
0 0
0 ... 0
0 0
t t
n
t t
-1
O J O
J = P ΛP
Λ
P O ΛP O
Nonlinear Networks
• Stable Fixed point: |1|<1, |2|<1
Nonlinear Networks
• Saddle Point: |1|>1, |2|<1
Nonlinear Networks
• Unstable Fixed point: |1|>1, |2|>1
Nonlinear Networks
• Line Attractor: 1=1, |2|<1
Nonlinear Networks
• Oscillation: complex ’s
Nonlinear Networks: global stability
• Lyapunov Function: function of the state of the system which is bounded below and goes down over time. If such a function exists, the system is globally stable.
• Ex: Hopfield network, Cohen-Grossberg network