Embed Size (px)
Citation preview
Neural networksFeedforward neural network - artificial neuron
ARTIFICIAL NEURON2
Topics: connection weights, bias, activation function•Neuron pre-activation (or input activation):
•Neuron (output) activation
• are the connection weights• is the neuron bias • is called the activation function
...1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• x
1
xd b w
1
wd
• w
• {
• g(·) b
• h(x) = g(a(x))
• a(x) = b
(1) +W
(1)
x
⇣a(x)i = b
(1)
i
Pj W
(1)
i,j xj
⌘
• o(x) = g
(out)(b(2) +w
(2)
>x)
1
ARTIFICIAL NEURON3
Topics: connection weights, bias, activation function
2Reseaux de neurones
-1 1
-1
1
11
1 1
.5
-1.5
.7-.4-1
x1 x2
x1
x2
z=+1
z=-1
z=-1
0
1-1
0
1
-1
0
1
-1
0
1-1
0
1
-1
0
1
-1
0
1-1
0
1
-1
0
1
-1
R2
R2
R1
y1 y2
z
zk
wkj
wji
x1
x2
x1
x2
x1
x2
y1 y2
sortie k
entree i
cachee jbiais
(from Pascal Vincent’s slides)
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = biP
i wixi = bi +w
>x
• h(x) = g(a(x)) = g(biP
i wixi)
• w
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = biP
i wixi = bi +w
>x
• h(x) = g(a(x)) = g(biP
i wixi)
• w
• {
1
range determined by
bias only changes the position of the riff
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• w
• {
• g(·) b
1
Feedforward neural network
Hugo LarochelleD
´
epartement d’informatique
Universit
´
e de Sherbrooke
September 6, 2012
Abstract
Math for my slides “Feedforward neural network”.
• a(x) = b+P
i wixi = b+w
>x
• h(x) = g(a(x)) = g(b+P
i wixi)
• w
• {
• g(·) b
1
