Embed Size (px)
Citation preview
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
1
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
We,
lik
e al
l o
ther
“h
igh
er”
anim
als,
are
ad
apta
ble
cre
atu
res,
we
lear
n.
Lea
rnin
g g
oes
on
at
dif
fere
nt
lev
els
and
usi
ng
dif
fere
nt
mec
han
ism
s, b
ut
it
is c
om
mo
nly
ag
reed
th
at l
earn
ing
in
vo
lves
th
e cr
eati
on
, ch
ang
e an
d d
eath
of
syn
apse
s.
We
mo
del
sy
nap
ses
wit
h a
sin
gle
nu
mb
er,
a sc
alar
, w
hic
h w
e u
sual
ly
den
ote
by
w. L
earn
ing
in
vo
lves
, o
r in
ou
r m
od
elli
ng
co
nsi
sts
of,
chan
gin
g a
ll w
’s .
Tec
hn
iqu
es f
or
cha
ng
ing
w f
orm
a m
ajo
r p
art
of
bra
in a
nd
min
d
mo
del
lin
g.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
2
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
Th
ere
are
thre
e le
arn
ing
par
adig
ms:
S
up
ervi
sed
lea
rnin
g
U
nsu
per
vise
d l
earn
ing
or
self
-org
an
iza
tio
n
R
ein
forc
emen
t le
arn
ing
Su
per
vis
ed l
earn
ing
was
fir
st d
evel
op
ed i
n t
he
195
0’s
an
d i
s w
idel
y u
sed
tod
ay b
ut
ther
e is
no
rea
son
to
bel
iev
e it
is
use
d i
n o
ur
bra
ins.
Sel
f-o
rgan
izat
ion
bec
ame
a fo
cus
of
stu
dy
in
th
e 1
980’s
. T
her
e ar
e g
oo
d
reas
on
s to
bel
iev
e it
is
use
d i
n o
ur
bra
ins.
Rei
nfo
rcem
ent
lear
nin
g i
s o
bv
iou
sly
use
d i
n o
ur
bra
ins,
bu
t h
as n
ot
bee
n
stu
die
d a
s m
uch
as
the
firs
t tw
o.
Th
e re
aso
n m
igh
t b
e th
at i
t is
rel
ativ
ely
un
inte
rest
ing
to
th
e en
gin
eeri
ng
-phy
sics
-mat
hem
atic
s co
mm
un
ity
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
3
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
Th
is c
hap
ter
is f
ull
y d
evo
ted
to
su
per
vis
ed l
earn
ing
. T
his
is
reas
on
able
fo
r
his
tori
c re
aso
ns
– m
any
bel
iev
ed t
hat
th
is w
as n
ot
on
ly a
n i
nte
rest
ing
tech
niq
ue
bu
t al
so t
hat
it
has
bio
log
ical
rel
evan
ce.
We
sho
uld
, ho
wev
er,
be
awar
e o
f th
e d
ang
ers
if w
e tr
y t
o d
raw
con
seq
uen
ces
in t
he
bio
logic
al r
ealm
fro
m e
xp
erim
ents
bas
ed o
n
sup
erv
ised
lea
rnin
g.
Mu
ch w
ork
of
this
kin
d h
as b
een
do
ne,
it
is n
ot
clea
r
wh
at l
asti
ng
val
ue
it m
igh
t h
ave.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
4
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
Th
en w
hat
is
sup
erv
ised
lea
rnin
g o
f a
neu
ral
net
wo
rk?
It a
ssum
es t
hat
a t
each
er (
in t
his
cas
e a
few
lin
es i
n a
com
pute
r pro
gra
m)
is p
rese
nt
and
super
vis
es t
he
resp
onse
of
the
neu
ral
net
work
to a
set
of
stim
uli
. B
ased
on t
he
teac
her
’s
resp
onse
, th
e ch
anges
in a
ll w
eights
w i
n t
he
neu
ral
net
work
wil
l be
calc
ula
ted s
o a
s to
mak
e th
e neu
ral
net
work
res
pond m
ore
lik
e th
e te
acher
than
bef
ore
.
F
rom
Haykin
, N
eura
l N
etw
ork
s A
Com
pre
hen
sive
Foundat
ion1994, p. 5
8
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
5
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
The
erro
r of
the
net
work
’s r
esponse
is
the
teac
her
’s r
esponse
min
us
the
net
work
s re
sponse
.
This
err
or
is u
sed i
n o
ne
of
sever
al e
rror
corr
ecti
ng a
lgori
thm
s to
chan
ge
the
val
ues
of
all
w’s
in t
he
net
work
.
The
Del
ta r
ule
was
dev
eloped
for
the
one-
lay
er p
erce
ptr
ons,
neu
ral
net
work
s popula
r in
the
1950’s
and 1
960’s
. T
he
idea
is
easy
so w
e w
ill
dis
cuss
it
in s
om
e det
ail.
The
Back-
pro
pagati
on r
ule
, or
bac
k-p
rop f
or
short
, w
as d
evel
oped
for
mult
i-la
yer
per
ceptr
ons
and h
as b
een p
opula
r si
nce
the
1980’s
. T
he
idea
is
easy
to u
nder
stan
d b
ut
we
leav
e th
e det
ails
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
6
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
Her
e w
e se
e a
one-
lay
er p
erce
ptr
on w
ith o
nly
one
neu
ron:
Fro
m G
urn
ey, A
n I
ntr
od
uct
ion t
o N
eura
l N
etw
ork
s, 1
997, p. 45
The
wei
ghts
should
be
chose
n t
o m
ake
the
outp
ut
hig
h w
hen
the
input
pat
tern
dis
pla
ys
an
‘A’,
and l
ow
oth
erw
ise
(as
an e
xam
ple
). H
ow
?
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
7
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
The
input
pat
tern
res
ides
on a
model
of
a re
tina
and t
he
pre
des
igned
ass
oci
atio
n u
nit
s
repre
sent
som
e pro
cess
ing i
n t
he
reti
na,
i.e
. “p
repro
cess
ing”.
The
model
neu
ron r
ecei
ves
its
inputs
fro
m t
he
asso
ciat
ion u
nit
s, t
he
acti
vat
ion i
s ca
lcula
ted
as b
efore
and t
he
outp
ut
is a
sim
ple
thre
shold
ing o
per
atio
n.
The
teac
her
know
s if
the
input
pat
tern
is
an ‘
A’
and i
f it
is
the
teac
her
res
ponds
wit
h a
hig
h
outp
ut,
say
1,
oth
erw
ise
by
a 0
.
If t
he
input
pat
tern
is
an ‘
A’
then
ther
e ar
e tw
o p
oss
ibil
itie
s :
if t
he
per
ceptr
on r
esponds
wit
h a
1 t
hen
it
resp
onded
corr
ectl
y a
nd i
ts w
eights
wil
l not
be
chan
ged
.
if t
he
per
ceptr
on r
esponds
wit
h a
0 t
hen
it
resp
onded
inco
rrec
tly
and i
ts w
eights
wil
l be
chan
ged
If t
he
input
pat
tern
is
not
an ‘
A’
then
ther
e ar
e al
so t
wo p
oss
ibil
itie
s.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
8
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
Ther
e ar
e tw
o c
ases
when
the
wei
ghts
should
be
chan
ged
. H
ow
?
The
idea
is
that
the
resp
onse
should
be
mad
e cl
ose
r to
that
of
the
teac
her
.
Let
us
call
the
vec
tor
input
from
the
asso
ciat
ion u
nit
s v
and t
he
wei
ght
vec
tor
of
the
neu
ron i
s w
. T
he
acti
vat
ion o
f th
e neu
ron i
s th
en t
he
dot
pro
duct
wx.
If t
he
teac
her
res
ponded
wit
h a
1 a
nd t
he
net
work
wit
h a
0, th
en t
he
dot
pro
duct
wx
should
be
incr
ease
d. If
w i
s m
ade
a bit
more
lik
e x
then
thei
r dot
pro
duct
wil
l in
crea
se. T
he
foll
ow
ing r
ule
wil
l ta
ke
care
of
that
:
∆
w =
α(t
eacher’
s re
sponse
– n
etw
ork
’s r
esp
onse
)xT
If t
he
teac
her
res
ponded
wit
h a
0 a
nd t
he
net
work
wit
h a
1, th
en t
he
dot
pro
duct
wx
should
be
dec
reas
ed. If
w i
s m
ade
a bit
les
s li
ke
x th
en t
hei
r dot
pro
duct
wil
l dec
reas
e. T
he
foll
ow
ing r
ule
wil
l ta
ke
care
of
that
:
∆
w =
α(t
eacher’
s re
sponse
– n
etw
ork
’s r
esp
onse
)xT
The
rule
is
the
sam
e in
the
two c
ases
so w
e hav
e a
single
rule
for
chan
ges
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
9
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
What
is
α i
n t
he
expre
ssio
n ∆
w =
α(t
eacher’
s re
sponse
– n
etw
ork
’s r
esp
onse
)x?
α i
s ca
lled
the
lear
nin
g r
ate.
A l
arge
α p
rovid
es l
arge
corr
ecti
ve
chan
ges
in t
he
wei
ghts
.
So t
he
larg
er t
he
α, th
e bet
ter?
N
o.
Gen
eral
ly α
should
be
chose
n “
fair
ly l
arge”
in t
he
beg
innin
g o
f a
lear
nin
g p
roce
ss a
nd t
hen
dec
reas
e.
Bio
logy
cer
tain
ly k
now
s how
to t
ake
care
of
that
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
10
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
W
hat
can
you d
o w
ith a
sin
gle
-lay
er p
erce
ptr
on a
nd t
he
del
ta l
earn
ing r
ule
?
You c
an d
ecid
e w
hat
cla
ss a
n o
bje
ct b
elongs
to i
f th
e cl
asse
s ar
e “l
inea
rly
sep
arab
le”.
C
lass
es a
re l
inea
rly
sep
arab
le
C
lass
es a
re n
ot
linea
rly
sep
arab
le
Fro
m H
aykin
, N
eura
l N
etw
ork
s A
Com
pre
hen
sive
Foundat
ion1994, p. 119
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
11
Ch
ap
ter
9,
Su
per
vis
ed L
earn
ing
: D
elta
Ru
le a
nd
Ba
ck-P
rop
ag
ati
on
T
he
single
-lay
er p
erce
ptr
on h
as s
erio
us
lim
itat
ions!
Thes
e li
mit
atio
ns
wer
e w
ell
expose
d i
n 1
969 b
y M
insk
y a
nd P
aper
t in
thei
r book
Perc
eptr
ons.
Most
of
the
mat
hem
atic
s-phy
sics
-engin
eeri
ng c
om
munit
y l
ost
inte
rest
in n
eura
l net
work
s
and f
ew w
ork
ed i
n t
he
fiel
d d
uri
ng t
he
1970’s
. T
he
bio
logy
-psy
cholo
gy
com
munit
y o
f
cours
e w
ere
not
worr
ied b
y t
he
com
puta
tional
lim
itat
ions
of
the
single
-lay
er p
erce
ptr
ons
(it
has
bee
n s
ugges
ted t
hat
thes
e li
mit
atio
ns
are
sim
ilar
to t
hose
of
rats
).
The
com
puta
tional
lim
itat
ions
of
the
single
-lay
er p
erce
ptr
ons
wer
e over
com
e by
the
mult
i-
lay
er p
erce
ptr
ons
and t
he
bac
k-p
ropag
atio
n l
earn
ing r
ule
in t
he
beg
innin
g o
f th
e 1980’s
.
Thes
e net
work
s w
ork
so w
ell
that
one
would
lik
e th
em t
o b
e a
model
of
bio
logy
. In
my
opin
ion t
his
is
a fu
tile
hope
and w
e th
eref
ore
lea
ve
them
to t
hose
who a
re i
nte
rest
ed i
n
lear
nin
g m
achin
es i
n g
ener
al, not
spec
ific
ally
the
centr
al n
ervous
syst
em.
Ther
e is
no t
each
er i
n t
he
bra
in a
nd t
he
erro
r co
rrec
ting l
earn
ing r
ule
s, n
o m
atte
r how
pow
erfu
l,
cannot
be
imple
men
ted i
n t
he
bra
in, so
we
must
mak
e a
fres
h s
tart
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
12
A c
ha
pte
r th
at
I m
iss:
Sel
f-o
rga
niz
ati
on
Wit
h n
o t
each
er p
rese
nt
the
lear
nin
g s
ituat
ion r
educe
s to
:
Fro
m H
aykin
, N
eura
l N
etw
ork
s A
Com
pre
hen
sive
Foundat
ion1994, p. 65
Ther
e is
a s
eem
ing m
agic
her
e. T
he
net
work
does
not
rece
ive
any
inst
ruct
ions,
yet
it
lear
ns
about
the
envir
onm
ent!
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
13
A c
ha
pte
r th
at
I m
iss:
Sel
f-o
rga
niz
ati
on
We
may
saf
ely
ass
um
e th
ere
is s
truct
ure
in t
he
envir
onm
ent.
It
is t
his
str
uct
ure
that
the
net
work
wil
l ev
entu
ally
“pic
k u
p”.
It i
s st
ill
seem
ing m
agic
, ju
st m
ore
pre
cise
ly(?
) phra
sed.
Let
us
thin
k a
bout
sounds.
Som
e of
thes
e ar
e so
und o
f sp
eech
, phonem
es.
They
are
im
port
ant
to
us
from
the
ver
y b
egin
nin
g –
moth
ers
wil
l sp
eak t
o t
hei
r in
fants
in “
moth
eres
e”, a
var
iant
of
ever
y l
anguag
e w
her
e th
e phonem
es a
re e
xte
nded
in t
ime
to g
ive
bab
y a
chan
ce t
o p
ick t
hem
up.
The
firs
t goal
of
self
-org
aniz
atio
n i
n a
udit
ory
cort
ex i
s to
tune
in t
o t
he
sounds
that
mak
e up t
he
phonem
es o
f th
e la
nguag
e in
your
envir
onm
ent.
Oth
er s
ounds
wil
l not
be
giv
en t
he
sam
e
atte
nti
on,
unle
ss o
f co
urs
e y
ou a
re a
buddin
g c
ar m
echan
ic.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
14
A c
ha
pte
r th
at
I m
iss:
Sel
f-o
rga
niz
ati
on
K
oh
on
en’s
ph
on
etic
ty
pew
rite
r
In t
he
1980’s
Kohonen
pre
sente
d t
he
phonet
ic t
ypew
rite
r, a
neu
ral
net
work
that
rec
eived
sound
signal
s, p
icked
up b
y a
mic
rophone
and b
andpas
s fi
lter
ed (
much
lik
e w
hat
hap
pen
s in
our
ear)
.
The
net
work
lea
rned
to d
isti
nguis
h F
innis
h p
honem
es v
ery
quic
kly
(m
uch
fas
ter
than
a h
um
an)
and c
ould
pri
nt
out
spee
ch o
nto
a s
cree
n, w
ith s
om
ethin
g l
ike9
5%
corr
ect
spel
ling.
F
rom
Kohonen
, S
elf-
Org
aniz
ing M
aps,
1995, p.
95
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
15
Sel
f-o
rga
niz
ati
on
Kohonen
inven
ted a
few
tri
cks,
spee
din
g u
p l
earn
ing i
ncr
edib
ly a
nd s
how
ing t
hat
sel
f-
org
aniz
atio
n i
s a
cred
ible
and e
ffic
ient
lear
nin
g p
arad
igm
. W
e w
ill
lear
n a
bout
those
tri
cks
in d
ue
tim
e. B
ut
our
pri
mar
y g
oal
is
to u
nder
stan
d h
ow
lea
rnin
g h
appen
s in
nat
ure
, not
to i
nven
t a
mac
hin
e th
at l
earn
s fa
ster
.
So b
ack t
o b
iolo
gy
!
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
16
S
elf-
org
an
iza
tio
n
W
e hav
e th
e hy
poth
esis
by
Heb
b f
rom
1949:
When
an a
xon o
f ce
ll A
is
nea
r en
ough t
o e
xci
te a
cel
l B
and
repea
tedly
tak
es p
art
in f
irin
g i
t, s
om
e gro
wth
pro
cess
or
met
aboli
c
chan
ges
tak
e pla
ce i
n o
ne
or
both
cel
ls s
uch
that
A’s
eff
icie
ncy
as
one
of
the
cell
s fi
ring B
, is
incr
ease
d.
This
is
one
pie
ce o
f th
e puzz
le.
Anoth
er p
iece
is
the
arch
itec
ture
of
late
ral
exci
tato
ry a
nd i
nhib
itory
fee
dbac
k t
hat
we
met
in
connec
tion w
ith t
he
Lim
ulu
s.
A t
hir
d a
nd m
ore
rec
entl
y d
isco
ver
ed p
iece
is
“volu
me
lear
nin
g”
mad
e poss
ible
by
a
“dif
fusi
ve
agen
t”,
usu
ally
bel
ieved
to b
e nit
ric
oxid
e, N
O.
A f
ourt
h p
iece
would
be
the
arch
itec
ture
of
e.g. neo
cort
ex w
ith i
ts s
ix l
ayer
s an
d
min
icolu
mns.
We
wil
l le
ave
this
fourt
h p
iece
out
of
the
puzz
le f
or
now
though –
it
is
dif
ficu
lt e
nough a
ny
way
!
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
17
S
elf-
org
an
iza
tio
n
Let
us
beg
in w
ith H
ebb’s
law
and t
ry t
o f
orm
aliz
e it
. It
consi
der
s th
e outp
uts
, or
effe
rents
,
from
the
two n
euro
ns
A a
nd B
, so
those
tw
o e
nti
ties
must
fig
ure
in t
he
form
al l
aw.
We
call
the
A’s
eff
eren
t x a
nd B
’s e
ffer
ent
y.
Fir
st w
e tr
y t
he
expre
ssio
n :
∆w
= α
xy
for
the
chan
ge
in t
he
wei
ght
w b
etw
een A
and B
.
If A
is
firi
ng x
is
larg
e an
d i
f B
is
firi
ng y
is
larg
e an
d ∆
w w
ill
be
larg
e. T
his
is
not
in
contr
adic
tion w
ith H
ebb’s
word
ing (
note
that
it
mig
ht
be
form
aliz
ed i
n a
num
ber
of
way
s,
Heb
b w
as n
ot
spec
ific
about
the
mat
hem
atic
s).
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
18
S
elf-
org
an
iza
tio
n
So w
hat
hap
pen
s in
a c
oncr
ete
case
?
Ther
e is
som
ethin
g f
ishy
her
e – t
he
wei
ght
w (
red c
urv
e) j
ust
gro
ws
and g
row
s!
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
19
S
elf-
org
an
iza
tio
n
It i
sn’t
that
Heb
b w
as w
rong.
He
just
pro
duce
d a
hal
f-tr
uth
. In
sci
ence
that
’s q
uit
e an
acco
mpli
shm
ent,
unli
ke
in t
esti
mony
in a
court
of
law
.
We
must
mak
e it
poss
ible
for
the
wei
ght
w t
o d
ecre
ase
also
. “F
org
etti
ng”
is n
eces
sary
!
It i
s re
asonab
le t
o a
men
d H
ebb’s
word
ing a
nd s
ay t
hat
when
A i
s fi
ring a
nd d
oes
n’t
com
e
close
to f
irin
g B
, th
en t
he
synap
se c
onnec
ting t
hem
isn
’t d
oin
g a
ny
thin
g w
ort
hw
hil
e an
d
ther
efore
its
eff
icie
ncy
, or
wei
ght,
should
dec
reas
e. A
fter
all
, a
synap
se s
hould
, ju
st l
ike
a
neu
ron, ea
rn i
ts k
eep.
We
noti
ce t
hat
x a
nd y
are
both
posi
tive
or
zero
. T
hei
r pro
duct
must
then
in o
ur
firs
t
form
aliz
atio
n a
lway
s be
posi
tive
or
zero
. T
her
efore
w w
ill
alw
ays
gro
w o
r at
bes
t ta
ke
a
tem
pora
ry r
est
from
gro
win
g.
If w
e w
ere
to s
ubtr
act
the
mea
ns
of
x a
nd y
we
would
expec
t w
to s
om
etim
es g
row
and
som
etim
es d
ecre
ase.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
20
S
elf-
org
an
iza
tio
n
The
term
∆w
= α
xy
in a
form
aliz
ed c
om
ple
te H
ebbia
n l
earn
ing l
aw i
s univ
ersa
lly
acc
epte
d.
Ther
e ar
e, h
ow
ever
, a
num
ber
of
dif
fere
nt
way
s of
intr
oduci
ng f
org
etti
ng.
One
com
mon v
aria
nt
is t
he
foll
ow
ing:
∆
w =
α1xy
- α
2yw
wher
e α
1 i
s t
he
lear
nin
g r
ate
and α
2 i
s th
e fo
rget
ting r
ate.
Let
us
see
how
it
work
s:
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
21
Sel
f-o
rga
niz
ati
on
We
now
hav
e in
terv
als
when
the
wei
ght
dec
reas
es a
nd i
t w
ill
not
gro
w b
eyond b
ounds.
If
we
look a
t th
e w
eight
over
a l
onger
tim
e w
e se
e th
at i
t does
n’t
set
tle
dow
n n
icel
y, how
ever
.
By
let
ting t
he
lear
nin
g r
ate
α1 a
nd t
he
forg
etti
ng r
ate
α2 i
s dec
reas
e over
tim
e w
e w
ill
achie
ve
that
too.
α1 a
nd
α2 c
onst
ant
over
tim
e
α1 a
nd
α2 d
ecre
ase
over
tim
e
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
22
S
elf-
org
an
iza
tio
n
We
hav
e sh
ow
n t
hat
we
can o
bta
in a
wei
ght
repre
senti
ng t
he
synap
se b
etw
een n
euro
n A
and n
euro
n B
in a
com
ple
te H
ebb’s
lea
rnin
g l
aw.
This
is
far
from
sel
f-org
aniz
ing t
hough.
Any
neu
ral
net
work
, bio
logic
al o
r m
odel
, has
man
y n
euro
ns,
i.e
. m
any
eff
eren
ts y
.
Eac
h n
euro
n h
as m
any
sy
nap
ses,
i.e
. m
any
aff
eren
ts x
and w
eights
w.
The
man
y w
eights
should
dev
elop o
ver
tim
e to
ref
lect
str
uct
ure
s in
the
affe
rents
, e.
g. th
e
exis
tence
of
phonem
es a
nd t
hei
r in
div
idual
char
acte
rist
ics.
Sel
f-org
aniz
atio
n s
hould
dev
elop s
o t
hat
the
neu
ral
syst
em w
ill
mak
e se
nse
of
its
surr
oundin
gs!
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
23
S
elf-
org
an
iza
tio
n
This
mig
ht
be
a su
itab
le t
ime
to q
uote
a p
rovoca
tive
stat
emen
t by
a F
innis
h r
esea
rcher
,
Hei
ki
Hy
öty
nie
mi:
When
the
mec
han
ism
s ar
e se
lect
ed i
n a
n a
ppro
pri
ate
way
, se
lf-o
rganiz
ati
on
auto
mat
ical
ly r
esult
s in
com
ple
x s
truct
ure
s – t
o p
ut
it s
trongly
, th
e sy
stem
cannot
hel
p g
etti
ng i
nte
llig
ent.
Ther
e is
no n
eed o
f ‘m
oti
vat
ion’,
etc
., f
or
expla
inin
g t
he
‘lea
rnin
g’
of
the
stru
cture
s.
Note
of
cauti
on:
T
her
e is
a n
eed f
or
som
e m
eta-
inte
llig
ence
in t
he
stru
cture
– w
hat
is
w
ort
hw
hil
e le
arnin
g? M
oti
vat
ion w
ill
pla
y a
n i
mport
ant
role
ther
e.
B
ut
Hy
öty
nie
mi’
s st
atem
ent
is n
ever
thel
ess
thoughtw
ort
hy
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
24
Sel
f-o
rga
niz
ati
on
A
net
wo
rk a
rch
itec
ture
We
rem
ember
fro
m t
he
Lim
ulu
s ch
apte
r how
ther
e w
as i
nhib
itory
lat
eral
fee
dbac
k a
nd t
hat
this
arc
hit
ectu
re c
ould
be
com
ple
men
ted w
ith e
xci
tato
ry l
ater
al f
eedbac
k i
n a
clo
se
nei
ghbourh
ood. A
lin
ear
arra
y o
f neu
rons
wit
h t
he
feed
bac
k c
onnec
tions
to t
he
neu
ron i
n
the
mid
dle
is
show
n b
elow
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
25
S
elf-
org
an
iza
tio
n
Th
e M
exic
an H
at
The
stre
ngth
s of
the
feed
bac
k a
s a
funct
ion o
f th
e dis
tance
bet
wee
n n
euro
ns
is s
how
n
bel
ow
. W
hen
we
hav
e a
two-d
imen
sional
arr
ay o
f neu
rons
this
funct
ion i
s know
n a
s th
e
“Mex
ican
Hat
”, f
or
obvio
us
reas
ons.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
26
S
elf-
org
an
iza
tio
n
H
ow
co
uld
it
all
wo
rk t
og
eth
er?
Ass
um
e th
at w
e ar
e in
the
beg
innin
g o
f th
e se
lf-o
rgan
izat
ion p
roce
ss.
An i
nput
is p
rese
nte
d t
o
the
neu
rons
in t
he
arra
y, not
to a
ll o
f th
em b
ut
to a
subst
anti
al n
um
ber
. M
any
neu
rons
wil
l fi
re
but
since
thei
r w
eights
are
not
“att
uned
” to
the
input,
the
firi
ng w
ill
be
unev
en a
nd s
pre
ad.
Som
ewher
e th
ere
is a
neu
ron t
hat
fir
es m
ost
. T
hat
neu
ron w
ill
cause
its
clo
se n
eighbours
to f
ire
more
bec
ause
of
its
exci
tato
ry f
eedbac
k t
o t
hose
clo
se n
eighbours
. A
t th
e sa
me
tim
e th
at n
euro
n
wil
l ca
use
its
more
dis
tant
nei
ghbours
to f
ire
less
bec
ause
of
its
inhib
itory
fee
dbac
k t
o t
hose
more
dis
tant
nei
ghbours
. T
his
eff
ect
wil
l ev
iden
tly
hav
e a
“spir
alli
ng”
effe
ct a
nd w
ill
cause
a
colu
mn o
f fi
ring n
euro
ns
to d
evel
op,
surr
ounded
by
sil
ent
neu
rons.
Now
the
Heb
bia
n l
earn
ing k
icks
in. T
he
firi
ng n
euro
ns
in t
he
colu
mn a
ll h
ave
hig
h o
utp
uts
and
wil
l hav
e th
eir
synap
ses
stre
ngth
ened
whic
h w
ill
mak
e th
em m
ore
sim
ilar
to t
he
input.
The
nex
t
tim
e th
at s
ame
input,
or
one
sim
ilar
, is
pre
sente
d t
he
neu
rons
in t
his
colu
mn w
ill
be
more
adap
ted t
o t
he
input
than
bef
ore
and w
ill
hin
der
oth
er n
euro
ns
from
fir
ing m
ore
eff
ecti
vel
y.
Did
the
oth
er n
euro
ns
loose
out?
Yes
, fo
r th
at i
nput
they
did
, but
they
are
fre
sh t
o a
dap
t to
oth
er
inputs
. A
nd s
o d
iffe
rent
gro
ups
of
neu
rons,
in c
olu
mnar
org
aniz
atio
n,
tend t
o a
dap
t to
dif
fere
nt
inputs
. W
e hav
e sp
ecia
list
s in
det
ecti
ng d
iffe
rent
inputs
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
27
S
elf-
org
an
iza
tio
n
A
n e
xam
ple
: b
efo
re s
elf-
org
aniz
atio
n
Bel
ow
is
the
acti
vit
y o
f 81 n
euro
ns
in a
squar
e ar
ray
. E
ight
ver
y d
iffe
rent
inputs
are
pre
sente
d t
o
the
net
work
. T
he
acti
vit
y l
ooks
rath
er u
norg
anis
ed.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
28
S
elf-
org
an
iza
tio
n
A
n e
xam
ple
: af
ter
self
-org
aniz
atio
n
The
eight
inputs
are
the
sam
e as
bef
ore
. N
ow
ther
e is
cle
arly
a c
olu
mn f
or
each
of
the
inputs
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
29
Sel
f-o
rga
niz
ati
on
A
n e
xam
ple
: af
ter
self
-org
aniz
atio
n
Bel
ow
are
the
sam
e co
lum
ns,
vis
ual
ized
in a
dif
fere
nt
way
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
30
Sel
f-o
rga
niz
ati
on
A
n e
xam
ple
: af
ter
self
-org
aniz
atio
n
Her
e w
e hav
e st
rength
ened
the
exci
tato
ry f
eedbac
k.
The
acti
vit
y s
tart
s to
be
frag
men
tened
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
31
Sel
f-o
rga
niz
ati
on
A
n e
xam
ple
: af
ter
self
-org
aniz
atio
n
Her
e w
e hav
e st
rength
ened
the
exci
tato
ry f
eedbac
k e
ven
more
and s
elf-
org
aniz
atio
n n
o l
onger
work
s ad
equat
ely
. T
he
net
work
is
sile
nt
to s
om
e in
puts
and h
as t
he
sam
e outp
ut
to s
ever
al
dif
fere
nt
inputs
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
32
Sel
f-o
rga
niz
ati
on
A
n e
xam
ple
: af
ter
self
-org
aniz
atio
n
Her
e w
e h
ave
stre
ng
then
ed t
he
inh
ibit
ory
fee
db
ack
. T
he
colu
mn
s h
ave
bec
om
e
nar
row
er.
If w
e fu
rth
er s
tren
gth
en t
he
inh
ibit
ory
fee
db
ack
no
co
lum
ns
wil
l
dev
elo
p a
nd
th
ere
is n
o s
elf-
org
aniz
atio
n:
the
neu
ral
acti
vit
y w
ill
rem
ain
un
org
anis
ed.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
33
Sel
f-o
rga
niz
ati
on
Wit
h t
he
right
bal
ance
bet
wee
n e
xci
tato
ry a
nd i
nhib
itory
lat
eral
fee
dbac
k w
e ac
hie
ved
a v
ery
appea
ling s
elf-
org
aniz
atio
n i
n t
his
sim
ple
cas
e.
Our
case
was
sim
ple
in t
wo w
ays.
Fir
st,
the
inputs
wer
e ver
y d
iffe
rent
from
eac
h o
ther
. T
echnic
ally
spea
kin
g, th
ey w
ere
ort
hogonal
to e
ach o
ther
.
Sec
ond, th
e in
puts
wer
e fe
w.
In c
ases
wher
e w
e hav
e m
any
dif
fere
nt
inputs
we
cannot
hope
to a
chie
ve
self
-org
aniz
atio
n w
ith
the
two m
echan
ism
s w
e hav
e giv
en.
So w
e nee
d m
ore
mec
han
ism
s – a
fter
all
we
are
able
to d
isti
nguis
h p
honem
es w
ell
and s
hould
ther
efore
hav
e nic
e phonoto
pic
map
s. M
ore
about
that
lat
er.
But
firs
t w
e w
ill
acquai
nt
ours
elves
wit
h o
ne
of
the
most
wel
l-know
n n
eura
l net
work
s – K
ohonen
map
s.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
34
S
elf-
org
an
iza
tio
n –
Ko
hon
en m
ap
s
F
un
dam
enta
ls
D
uri
ng s
elf-
org
aniz
atio
n t
her
e dev
elops
a cl
ear
“win
nin
g”
colu
mn o
f neu
rons
when
an i
nput
is
pre
sente
d t
o t
he
neu
ral
net
work
. T
his
pro
cess
is
tim
e-co
nsu
min
g,
it c
onsi
sts
of
enhan
cing t
he
resp
onse
of
the
bes
t ad
apte
d n
euro
ns
and d
imin
ishin
g t
he
resp
onse
of
less
wel
l ad
apte
d n
euro
ns,
adap
ted t
o t
he
par
ticu
lar
input,
that
is,
thro
ugh t
he
mec
han
ism
s of
late
ral
feed
bac
k.
In a
com
pute
r m
odel
of
a neu
ral
net
work
we
hav
e a
quic
ker
way
of
iden
tify
ing t
he
win
ner
– w
e
(or
rath
er o
ur
com
pute
r) j
ust
com
par
e al
l th
e outp
uts
of
the
neu
ron a
nd i
den
tify
the
larg
est.
We
don’t
nee
d a
ny
lat
eral
fee
dbac
k t
o d
o t
hat
, an
d i
n t
he
net
work
we
now
stu
dy
we
ther
efore
don’t
incl
ude
late
ral
feed
bac
k.
Aft
er i
den
tify
ing t
he
win
ner
we
chan
ge
its
wei
ghts
to b
e ev
en m
ore
att
uned
to t
he
input.
That
mea
ns
mak
ing i
t w
eight
vec
tor
more
lik
e th
e in
put
vec
tor
(thei
r sc
alar
pro
duct
is
the
larg
est
when
they
coin
cide)
.
But
that
’s n
ot
all.
We
also
chan
ge
the
wei
ghts
in a
suit
able
nei
ghbourh
ood o
f th
e w
innin
g n
euro
n
in t
he
sam
e w
ay. T
hat
way
we
gai
n a
char
acte
rist
ic o
f to
polo
gic
al o
rder
ing o
f th
e m
ap.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
35
Sel
f-o
rga
niz
ati
on
– K
oh
on
en m
ap
s
Let
us
assu
me
that
the
i:th
neu
ron w
as i
den
tifi
ed a
s th
e w
inner
for
the
input
x. T
hen
the
wei
ght
vec
tor
for
that
neu
ron,
wi,
wil
l be
chan
ged
by
addin
g a
ter
m
∆w
i =
η(x
T -
wi)
Cle
arly
this
tak
es a
way
a l
ittl
e bit
of
the
“old
wi”
and r
epla
ces
it w
ith s
om
ethin
g m
ore
lik
e x,
i.e
.
x i
tsel
f. T
he
tran
spose
does
n’t
chan
ge
any
of
that
– i
t’s
nec
essa
ry f
or
dim
ensi
onal
ity
rea
sons.
But
the
i:th
neu
ron i
s not
the
only
one
to h
ave
its
wei
ghts
chan
ged
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
36
Sel
f-o
rga
niz
ati
on
– K
oh
on
en m
ap
s
We
hav
e al
read
y s
tate
d t
hat
a n
um
ber
of
neu
rons
in a
suit
able
nei
ghbourh
ood o
f th
e i:
th n
euro
n
should
hav
e th
eir
wei
ghts
chan
ged
too. W
hat
is
a su
itab
le n
eighbourh
ood? L
et u
s dis
cuss
the
figure
bel
ow
. (F
rom
Hay
kin
, p.
413)
The
nei
ghbourh
oods
are
squar
es
In t
his
cas
e (t
hat
is
a co
mm
on
choic
e fo
r sh
ape
of
nei
ghbourh
ood).
Ther
e ar
e fo
ur
squar
es,
den
ote
d
by
Λi =
0,…
, Λ
i = 3
.
The
larg
est
squar
e, Λ
i = 3
,
conta
ins
48 n
euro
ns
around
the
win
nin
g n
euro
n (
the
fill
ed c
ircl
e)
The
nex
t la
rges
t sq
uar
e co
nta
ins
24 n
euro
ns
around t
he
win
nin
g n
euro
n.
The
nex
t sq
uar
e co
nta
ins
8 n
euro
ns
around t
he
win
nin
g n
euro
n.
The
smal
lest
squar
e co
nta
ins
no
neu
ron, ex
cept
for
the
win
nin
g n
euro
n.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
37
S
elf-
org
an
iza
tio
n –
Ko
hon
en m
ap
s
O
ne
could
say
that
Kohonen
im
pro
ves
upon n
ature
her
e. I
nst
ead o
f th
e fi
xed
rea
ch o
f th
e la
tera
l
feed
bac
k w
hic
h d
eter
min
e th
e w
idth
of
the
colu
mn o
f neu
rons
we
found e
arli
er,
he
lets
sel
f-
org
aniz
atio
n p
roce
ed i
n s
tages
.
He
firs
t ch
anges
the
wei
ghts
for
all
neu
rons
in t
he
larg
est
nei
ghbourh
ood. H
e does
that
for
all
dif
fere
nt
inputs
, re
pea
tedly
.
In t
he
nex
t st
age
he
chan
ges
the
wei
ghts
for
all
neu
rons
in t
he
seco
nd l
arges
t nei
ghbourh
ood a
nd
then
conti
nues
unti
l he
chan
ges
only
the
wei
ghts
of
the
win
nin
g n
euro
n.
What
does
he
gai
n f
rom
all
this
eff
ort
?
Inputs
that
are
sim
ilar
wil
l hav
e w
inner
s th
at a
re c
lose
– t
his
is
a to
polo
gic
al o
rder
ing.
He
wil
l not
hav
e co
lum
ns
of
neu
rons
in t
he
end b
ut
single
neu
rons
tuned
for
dif
fere
nt
inputs
.
And t
hat
’s O
K –
the
bra
in d
oes
hav
e co
lum
ns
of
neu
rons
but
Kohonen
does
n’t
set
out
to m
odel
that
.
Let
us
study
som
e ex
ample
s.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
38
K
oh
on
en m
ap
s –
ex
am
ple
s
Vis
ual
izin
g K
ohonen
map
s ca
n b
e done
in t
wo “
regim
es”,
low
-dim
ensi
onal
(dim
ensi
ons
1,
2
and 3
) an
d h
igh-d
imen
sional
. In
the
low
-dim
ensi
onal
reg
ime
we
may
see
both
inputs
and w
eights
in t
he
sam
e plo
t. R
emem
ber
that
wei
ghts
of
a w
innin
g n
euro
n s
hould
be
close
to “
its”
input.
If
two-d
imen
sional
inputs
are
man
y a
nd e
ven
ly s
pre
ad o
ver
a s
quar
e as
in t
he
figure
bel
ow
, an
d a
lin
ear
arra
y o
f neu
rons
wit
h s
mal
l in
itia
l w
eights
goes
thro
ugh t
he
pro
cess
of
self
-org
aniz
atio
n w
e w
ould
expec
t th
e neu
rons
to h
ave
thei
r w
eights
spre
ad t
o m
atch
those
of
he
inputs
as
wel
l as
poss
ible
. H
ere
is w
hat
it
mig
ht
look l
ike
(fro
m H
aykin
, p.4
20):
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
39
K
oh
on
en m
ap
s –
ex
am
ple
s T
he
sam
e tw
o-d
imen
sional
dat
a ca
n b
e pre
sente
d t
o a
squar
e ar
ray
of
neu
rons.
By
squar
e w
e
mea
n t
hat
the
neu
rons
are
loca
ted i
n a
squar
e la
ttic
e ra
ther
than
on a
lin
e as
the
exam
ple
above.
What
we
mea
sure
in t
he
coord
inat
e sy
stem
is
the
wei
ghts
of
the
neu
rons,
not
thei
r posi
tion i
n t
he
latt
ice.
The
posi
tion o
f th
e neu
ron i
s in
dic
ated
by t
he
lines
connec
ting t
he
neu
rons.
Fro
m H
aykin
, p. 414.
B
efore
sel
f-org
aniz
atio
n
Aft
er s
elf-
org
aniz
atio
n
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
40
K
oh
on
en m
ap
s –
ex
am
ple
s
Even
ly s
pre
ad d
ata
as i
n t
he
two p
revio
us
exam
ple
s ar
e not
the
most
inte
rest
ing c
ase,
may
be.
In t
he
exam
ple
bel
ow
ther
e ar
e tw
enty
-four
dat
a poin
ts. T
hes
e ar
e not
even
ly s
pre
ad t
hough, but
rath
er a
ppea
r in
pai
rs s
o t
her
e ar
e tw
elve
pai
rs. T
hey
hav
e dif
fere
nt
sym
bols
, o, +
and s
tars
. T
his
is n
ot
import
ant
at t
he
pre
sent.
The
wei
ghts
of
the
4x4 n
euro
ns
are
show
n a
s *.
Aft
er s
elf-
org
aniz
atio
n w
e hav
e so
met
hin
g l
ike:
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
41
K
oh
on
en m
ap
s –
ex
am
ple
s
What
mig
ht
we
hav
e ex
pec
ted i
n t
he
exam
ple
above?
Sin
ce t
her
e ar
e tw
elve
pai
rs o
f dat
a an
d
sixte
en n
euro
ns
we
mig
ht
hav
e ex
pec
ted t
hat
ther
e w
ould
be
one
neu
ron “
assi
gned
to”
each
pai
r
and f
our
neu
rons
mig
ht
bec
om
e w
hat
is
know
n a
s “d
ead n
euro
ns”
.
This
is
alm
ost
what
hap
pen
s. T
her
e ar
e te
n p
airs
of
dat
a w
hic
h e
ach a
re a
ssig
ned
one
neu
ron.
Tw
o p
airs
of
dat
a w
hic
h a
re c
lose
and l
ook a
lmost
lik
e one
gro
up o
f fo
ur
dat
a ar
e as
signed
one
neu
ron o
nly
. F
ive
neu
rons
bec
om
e dea
d n
euro
ns.
Let
us
now
go t
o h
igher
dim
ensi
ons.
We
can t
hen
not
show
the
dat
a or
the
wei
ghts
of
the
neu
rons
so w
e w
ill
hav
e to
show
the
map
in a
more
abst
ract
way
. It
mig
ht
look m
ore
to t
he
poin
t
and m
ore
concr
ete,
but
the
map
s ar
e w
ort
hy
of
som
e th
ought.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
42
H
igh
er d
imen
sio
n K
oh
on
en m
ap
s –
ex
am
ple
s L
et u
s se
e how
a K
ohonen
map
would
dev
elop i
f ch
arac
teri
stic
s of
a num
ber
of
dif
fere
nt
anim
als
are
the
inputs
. T
he
anim
als
are
the
foll
ow
ing:
Gro
up
A:
Prz
ewal
ski’
s hors
e, G
revy
’s z
ebra
, w
olf
, din
go,
whit
e sw
an, bla
ck s
wan
, A
tlan
tic
salm
on,
rain
bow
tro
ut,
pola
r bea
r, K
odia
k b
ear,
whit
e rh
inoce
ros,
hip
popota
mus,
gre
y w
este
rn
kan
gar
oo, sw
amp w
alla
by
, an
aconda,
gre
y w
hal
e. A
tota
l of
sixte
en a
nim
als.
Gro
up
B:
tiger
, li
on, ja
guar
, le
opar
d,
snow
leo
par
d,
bla
ck p
anth
er, co
ugar
, ch
eeta
h, oce
lot,
Eura
sian
ly
nx, se
rval
, fi
shin
g c
at,
bla
ck d
om
esti
c ca
t, e
ven
colo
ure
d d
om
esti
c ca
t,
stri
ped
dom
esti
c ca
t, S
iam
ese
dom
esti
c ca
t. A
tota
l of
sixte
en f
elin
es.
All
anim
als
wer
e ch
arac
teri
zed b
y t
hei
r si
ze, w
het
her
the
are
carn
ivore
s, o
mniv
ore
s or
her
biv
ore
s, i
f th
ey h
ave
four
legs,
tw
o o
r no l
egs,
win
gs
or
fins,
th
eir
soci
abil
ity
(li
vin
g i
n
gro
ups,
pai
rs,
or
soli
tude)
, c
olo
rati
on a
nd h
ead s
hap
e. E
ach a
nim
al i
s des
crib
ed b
y e
ighte
en
num
ber
s, i
.e. by
a v
ecto
r of
dim
ensi
on e
ighte
en.
Ther
e ar
e si
xte
en n
euro
ns
in t
he
map
, so
me
com
pro
mis
es m
ust
be
mad
e. L
et u
s se
e w
hat
sel
f-
org
aniz
atio
n a
chie
ves
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
43
H
igh
er d
imen
sio
n K
oh
on
en m
ap
s –
ex
am
ple
s
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
44
Hig
her
dim
ensi
on
Ko
ho
nen
ma
ps
– e
xa
mp
les
W
hat
char
acte
rist
ics
do w
e fi
nd i
n t
his
map
?
We
see
the
anim
als
list
ed i
n o
val
s. E
ach o
val
rep
rese
nts
a n
euro
n. T
her
e is
als
o a
cro
ssed
-over
neu
ron, th
is i
s a
dea
d n
euro
n.
Eac
h a
nim
al i
s li
sted
at
the
neu
ron w
hic
h h
as t
he
wei
ght
vec
tor
whic
h m
ost
clo
sely
res
emble
the
char
acte
rist
ic v
ecto
r of
the
anim
al.
Aft
er e
ach a
nim
al w
ee s
ee a
num
ber
. T
his
num
ber
rep
rese
nts
the
dis
tance
bet
wee
n t
he
neu
ron
wei
ght
vec
tor
and t
he
anim
al c
har
acte
rist
ic v
ecto
r. A
sm
all
num
ber
is
a cl
ose
r m
atch
than
a l
arge
num
ber
.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
45
Hig
her
dim
ensi
on
Ko
ho
nen
ma
ps
– e
xa
mp
les
Fir
st w
e se
e th
at t
he
real
ly “
odd”
anim
als,
the
whal
e an
d t
he
anac
onda,
eac
h h
as b
een a
ssig
ned
one
neu
ron. T
hat
see
ms
sensi
ble
. If
an a
nac
onda
is t
he
input
we
wan
t th
e m
ap t
o r
eport
this
and
we
don’t
wan
t th
e neu
ron i
n e
ffec
t si
gnal
ling “
an a
nac
onda
or
…(s
om
ethin
g s
imil
ar)
is t
he
input”
.
We
also
see
a n
um
ber
of
pai
rs o
f an
imal
s w
ho s
har
e one
neu
ron, li
ke
the
whit
e sw
an a
nd t
he
bla
ck s
wan
. T
his
is
a good c
om
pro
mis
e fo
r a
map
that
does
n’t
hav
e th
e ca
pac
ity
to a
ssig
n o
ne
neu
ron t
o e
ach a
nim
al i
nput.
We
furt
her
see
that
the
four
var
iants
of
dom
esti
c ca
ts s
har
e one
neu
ron.
That
see
ms
like
a good
“choic
e” t
oo, if
we
wan
t to
use
our
scar
ce n
euro
ns
in a
n e
conom
ic w
ay.
The
tree
big
ges
t ca
ts s
har
e one
neu
ron.
The
mat
ch i
s not
at a
ll a
s cl
ose
as
for
the
dom
esti
c ca
ts u
t
that
is
to b
e ex
pec
ted –
a b
lack
dom
esti
c ca
t an
d a
str
iped
dom
esti
c ca
t hav
e m
ore
char
acte
rist
ics
in c
om
mon t
han
a t
iger
and a
lio
n.
That
the
wolf
and t
he
din
go e
ach h
as o
ne
neu
ron i
s an
oddit
y o
f a
kin
d t
hat
is
not
unco
mm
on.
Len
nar
t G
ust
afss
on pag
e
Cours
e note
s C
SE
2330
46
Hig
her
dim
ensi
on
Ko
ho
nen
ma
ps
– e
xa
mp
les
B
ut
ther
e is
ord
er i
n t
he
ma
p t
oo
! T
he
feli
nes
are
rep
rese
nte
d a
long t
he
right
and p
art
of
the
upper
edge,
i.e
. th
ey a
re c
lose
to e
ach
oth
er.
The
feli
nes
are
rep
rese
nte
d i
n s
ize
ord
er.
The
oth
er c
arniv
oro
us
mam
mal
s ar
e re
pre
sente
d b
y n
euro
ns
close
to t
he
cats
’ neu
rons.
The
big
ger
anim
als
are
in t
he
upper
par
t of
the
map
and t
he
smal
ler
anim
als
are
in t
he
low
er p
art
of
the
map
.
In t
he
self
-org
aniz
atio
n s
om
e ch
arac
teri
stic
s hav
e bee
n f
ound t
o b
e of
such
im
port
ance
that
they
are
refl
ecte
d i
n t
he
ord
erin
g o
f th
e m
ap. B
ut
oth
ers
hav
e not.
So w
e m
ay a
sk w
hat
is
not
repre
sente
d in
the
map
.
The
soci
al p
refe
rence
s e.
g. L
ions
live
in p
acks,
tig
ers
are
soli
tary
and j
aguar
s oft
en l
ive
in p
airs
,
if I
am
corr
ectl
y i
nfo
rmed
. B
ut
they
sti
ll s
har
e one
neu
ron.
