Upload
others
View
2
Download
0
Embed Size (px)
Citation preview
Vshuu/Work/Genova/PPTW_gen 14/05/051
To be is the only way to knowBuddhaa
To be is the only way to knowBuddhaa
Vishwanathan Mohan
PhD Student, IIT
Vshuu/Work/Genova/PPTW_gen 14/05/052
CHNNElectronic Model
DynamicsEnergetic CostsIEEE MWSCAS
AHNNOnline Adaptation
Dynamics CHAR Recog
JACII
Link Between Energy
And Computation
NIPS 2001
HOPFIELDNEURAL
NETWORK(HNN)
Spike Based
Neuromorphic Designs
SOMSOM
Multilevel Activation based HNN
6 Level AVLSI CKT
Pattern Recognition
Phantom Limbs
Pattern Recognition
Phantom Limbs
Vshuu/Work/Genova/PPTW_gen 14/05/053
Low Power Chip DesignLow Power Chip Design
Device levelDevice level System levelSystem level
Obvious culpritsObvious culprits
Computer as a Computer as a
Thermodynamic EngineThermodynamic Engine
A B Out0 0 00 1 01 0 01 1 1
A B Out0 0 00 1 01 0 01 1 1
AND GateAND Gate Conventional Digital Designs
are Logically Irreversible
Physical Irreversibility
dU = dQ +dW
Conservative Logic
Dynamic Charge Recovery Logic
Electronic Letters
Jour. Ckts and Sys, 2004
Vshuu/Work/Genova/PPTW_gen 14/05/054
A model of associative memory.Fully connected net of MP neuronsDissipative dynamics: has Lyapunov fn.Dynamics can be modeled into an electronic equivalent circuit.
Hopfield Neural NetworkHopfield Neural Network
Vshuu/Work/Genova/PPTW_gen 14/05/055
Vshuu/Work/Genova/PPTW_gen 14/05/056
Hpofeild Neuarl Netwrok
(W (i ,j)= W (j, i))(W (i ,i)= 0)
)( ii ugV λ=Vi = state of i’th neuron;
Wij = strength of connection between
Output of Jth neuron to inputOf Ith neuron;
g(i) = sigmoid function;
∑=
+−=N
jjiji
i Vwudtdu
1τ
Ui
Vshuu/Work/Genova/PPTW_gen 14/05/057
a1 = [1 -1 1 -1]
∑=
=P
p
pj
piij NSSw
1/
0 -1 1 -1-1 0 -1 11 -1 0 -1
-1 1 -1 0
Equivalent ResistancesStorage Algorithm
One Shot
Retrieval
Network’s initial state, V(0), is a noisy version of a stored pattern, Sp. Noise is added by flipping a fixed fraction (15%) of bits in Sp. Ideally, for V(0) close to Sp, the final state, Vfinal, of the network must equal Sp.
Sequence of states in a recurrent network is Non deterministic .In addition there can be many equilibrium states that can be potentially reached after many such nondeterministic amount of transitions.
HNN as Associative Memory
Vshuu/Work/Genova/PPTW_gen 14/05/058
Does a solution exist ? If it does ,will the system converge in finite time ? What is the solution.
“ If a bounded function of state variables of a dynamical system can be found, such that all state changes result in a decrease in the value of the function , then the system has a stable solution”.
∑ ∑∫∑∑= =
−
= =
−+−=N
i
N
iii
VN
i
N
jjiij VIdVVgVVwE i
1 10
1
1 1)(1
21
λ
0))(('
)(
2
11
1 1
≤−=−=
+−−=
∑∑
∑ ∑
==
= =
dtdu
ugdt
dVdtdu
dtdV
IuVwdtdE
iN
ii
iN
i
i
iN
i
N
jiijij
Vshuu/Work/Genova/PPTW_gen 14/05/059
∑=
+−=N
jjiji
i Vwudtdu
1τ
)( ii ugV λ=
0)(
1=
−++ ∑
=
N
j ij
jiji
t
ii
RVu
Ru
dtdu
Cγ
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=
∑=
N
j ijt RR
C
1
11τ
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=
∑=
N
j ijt
ijij
ij
RR
Rw
1
11
)1(γ
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=
∑=
N
jijt
ij
ij wR
w
R
11
1
Taking modulus on both sides,
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=
∑=
N
j ijt
ijij
RR
Rw
1
11
1
Taking summation over j,
⎟⎟⎠
⎞⎜⎜⎝
⎛+
=
∑
∑∑
=
=
= N
j ijt
N
j ijN
jij
RR
Rw
1
1
1 11
1
Rij, expressing it in termsof network weights.
⎟⎟⎠
⎞⎜⎜⎝
⎛−
=
∑=
N
jijt
ij
ij wR
w
R
11
1
∑∑==
+⎟⎟⎠
⎞⎜⎜⎝
⎛+−=
N
j ij
jijN
j ijti
i
Rv
RRu
dtdu
C11
11 γ
Vshuu/Work/Genova/PPTW_gen 14/05/0510
∑∑==
+⎟⎟⎠
⎞⎜⎜⎝
⎛+−=
N
j ij
jijN
j ijti
i
Rv
RRu
dtdu
C11
11 γ
∑=
+−=N
jjiji
i Vwudtdu
1τ
Visual C++Visual C++
Foreign NetlistForeign Netlist
SCAD (Linear Technology)SCAD (Linear Technology)
Vshuu/Work/Genova/PPTW_gen 14/05/0511
Is there a relationship between: 1) HNN performance as an associative memory and 2) the concomitant energy dissipation in the electronic model?
‘Link between energy and computation in a physical model of Hopfield Network’ VS Chakravarthy, Vishwanathan Mohan, Uday Shankar NIPS 2001, Singapore.
1) L .Sokoloff, Metabolic Probes of Central Nervous SystemActivity in Experimental Animals and Man, Sunderland, MA:Sinauer Associates, 1984.2)J.G.McElligott and R Melzack, “Localized Thermal Changesevoked in the brain by Visual and Auditory Stimulation,”Experimental Neurology, vol. 17, pg 293-312, 1967
Vshuu/Work/Genova/PPTW_gen 14/05/0512
In the brain there is a tight coupling between neural activity, local
cerebral blood flow, and local glucose metabolism .
Our experiments on possibility of such a link in neural network
models show a consistent correlation between Energy dissipated and
performance of Hopfield Neural Network.
IS the brain exploiting some relationship between computation and
energy dissipation?
If there is a minimal cost of computation ,has the brain evolved so
as to optimize its energy expenditure in order to perform its
functions ?
Irreversible computational process have inevitable energetic costs
Rolf Launder (1961), IBM Joun. Of Research
Vshuu/Work/Genova/PPTW_gen 14/05/0513
COMPLEX HOPFIELD NEURAL NETWORKCOMPLEX HOPFIELD NEURAL NETWORK : : OverviewOverview
Extension of Hopfield model from real number to Complex numberExtension of Hopfield model from real number to Complex numberdomain .domain .
Interesting as real Neural systems also exhibit a range of Interesting as real Neural systems also exhibit a range of phenomenon like oscillations ,Phase lock and Chaos seldom touchephenomenon like oscillations ,Phase lock and Chaos seldom touched d by dissipative models with fixed point dynamics.by dissipative models with fixed point dynamics.
Exhibits both Fixed point and Oscillatory behavior controlled bExhibits both Fixed point and Oscillatory behavior controlled by a y a continuous mode parameter continuous mode parameter -- ‘‘٧٧’’
Can be used as associative memoryCan be used as associative memory
At one extreme network has conservative dynamics and at other At one extreme network has conservative dynamics and at other extreme , dynamics are dissipative and governed by Lyapunov extreme , dynamics are dissipative and governed by Lyapunov function.function.
Work with mammalian olfactory cortex revealed that odors are stored as oscillatory states.
Freeman (1987)
Vshuu/Work/Genova/PPTW_gen 14/05/0514
COMPLEX HOPFIELD NEURAL NETWORKCOMPLEX HOPFIELD NEURAL NETWORK : : DynamicsDynamics
ZJ : Complex variable representing state of Jth
Neuron.
Vj : Output of Jth Neuron
TJK :Strength of connection between Jth and Kth neurons
α and β are complex coefficients such that , β = γ + i (1 – γ ) and α = λ βWhere λ is a large positive number, g(.) is Tanh activation function
jjkjktZj IZVT
k+−∑=∂
∂ βτa
)( *jj ZgV α=
Vshuu/Work/Genova/PPTW_gen 14/05/0515
Vishwanathan Mohan and VS Chakravarthy .,”Inevitable Energy Costs of Storage Capacity Enhancement in an Oscillatory Neural Network,” IEEE Circuits and Syst, 2003
When the oscillations in CHNN When the oscillations in CHNN stabilize outputs of individual neurons stabilize outputs of individual neurons attain fixed phase relationships. attain fixed phase relationships.
Situation changes dramatically Situation changes dramatically when 2 or more patterns are stored. when 2 or more patterns are stored. Instead of oscillating between a Instead of oscillating between a specific pattern and its negative, the specific pattern and its negative, the network wanders chaotically from one network wanders chaotically from one stored pattern to another.stored pattern to another.
When the same circuit is When the same circuit is operated in static mode however, operated in static mode however, We are able to successfully store We are able to successfully store and recall much more patterns in and recall much more patterns in the presence of noise (0.14, the presence of noise (0.14, Hopfield)Hopfield)
Interesting thing is that the Interesting thing is that the same electronic circuit also same electronic circuit also dissipates much higher energy to dissipates much higher energy to do so!do so!
Vshuu/Work/Genova/PPTW_gen 14/05/0516
Adaptive HNN (Online Hebbian adaptation during retrieval phase at a slower timescale)
•Two phases: Storage phase and retrieval phase•In connectionist networks weights are frozen once the training is completed and network is used for retrieval •In AHNN weights also change during retrieval, Hence losing information during retrieval
( , , )ijij i j
dw F w V Vdt
=
Vshuu/Work/Genova/PPTW_gen 14/05/0517
w
ccw R
V VC−
= jiVVdt
d
jiVVdtd
+−= cc
ww VV
CR
cij vW ∝
( )TGSiDSij VVKW
LRR−
≅=
ijij R
W 1∝
ijij i j
dw w V Vdt
ρ α= − +
Vshuu/Work/Genova/PPTW_gen 14/05/0518
Comparative retrieval performance of the HNN and AHNN
Vishwanathan Mohan, , Joshi, Y.V, Anand, I., “Studies on an Electronic analog of a Recurrent Neural Network with retrieval Phase weight adaptations,” Journal of Advanced Computational Intelligence and Intelligent Informatics ,Editors L A Zadeh, T.Fukuda
Vshuu/Work/Genova/PPTW_gen 14/05/0519
SCIS 2004, Yokohama
Vshuu/Work/Genova/PPTW_gen 14/05/0520
Multi Level AMSpike Based Electronic Models
Vshuu/Work/Genova/PPTW_gen 14/05/0521
Fixed PointHNN
AHNNFixed Point
HNN + Learning
GHNNMulti Level Associative
Memory
CHNNOscillatoryAssociative
Memory
Spike Based
Neural Computation
Vshuu/Work/Genova/PPTW_gen 14/05/0522
Self Organizing MapsSelf Organizing Maps
Vshuu/Work/Genova/PPTW_gen 14/05/0523
900 Neurons Competing, Cooperating, and Adapting
Inspired by ‘PHANTOMS IN THE BRAIN’
Vshuu/Work/Genova/PPTW_gen 14/05/0524
A Glance at NatureA Glance at NatureWhen When the only the only tool you tool you have is a have is a Hammer Hammer ,Every ,Every problem problem begins begins and ends and ends with a with a Nail.Nail.
Pentium IV: 108 Transistors100W /CM2
Pentium 4: 10Pentium 4: 108 8 TranTran100W/cm100W/cm22
Vshuu/Work/Genova/PPTW_gen 14/05/0525
Vishwanathan Mohan, Joge.A. ‘Efficient Datapath Design using 6*6 Conservative reversible gates’, Journal of Circuits and Systems (World Scientific)
Vshuu/Work/Genova/PPTW_gen 14/05/0526
Dynamic Charge Recovery Logic
Vishwanathan Mohan, ‘Dynamic Charge Recovery Logic’ Submitted to the Journal: Electronic Letters, Dec 2005
CL=0.1pf
0.1 Micron NWELL Process
Dev density :
CMOS : 2N SCRL:4N+4 DCRL: 2N+8
Power : 90% CMOS and 33% SCRL
Vshuu/Work/Genova/PPTW_gen 14/05/0527
Vish to acknowledge the help, suggestions and encouragement provided by :
Govt of India (MHRD, AICTE)
Dr. M.Frank (FSU), Dr. Hiroshi Tsujimo (HRI),
Dr. V.S.Chakravarthy (IITM), M.J.Patil (Texas Inst, USA)
Dr.Y.V.Joshi (SGGSIE&T), GowriShankar (SHARP SD Labs )
Family, Friends……………………………………………………………………………
………………………………. Billions of Stars and the Moon……………..
www.geocities.com/vish_m2000/VISHUU
Vshuu/Work/Genova/PPTW_gen 14/05/0528
Mathematical AppendixMathematical Appendix
Lemma 1. If Lemma 1. If λλııRe [Z]Re [Z]ıı >> 1 and g(>> 1 and g(λλ z)=tanh(z)=tanh(λλz) ,then z) ,then g(g(λλ z)z)≈≈ tanh(tanh(λλ(Re [Z]))(Re [Z]))
Vshuu/Work/Genova/PPTW_gen 14/05/0529
Single Neuron oscillatorSingle Neuron oscillator
Dynamics of single neuron is described byDynamics of single neuron is described by ::
Separating real and imaginary parts and invoking Lemma1 we have,Separating real and imaginary parts and invoking Lemma1 we have,
(a) (a) which is a conservative system of form which is a conservative system of form
and describes motion of a particle in a potential Vand describes motion of a particle in a potential Vpot pot given bygiven by
))log(cosh(21)( 2 ywyyfVpot λ
λ+=−= ∫
ForFor γγ=0 the resistors disappear and =0 the resistors disappear and circuit conserves energy. Applying circuit conserves energy. Applying KCL at node y gives,KCL at node y gives,
Same as (a)Same as (a)
Figure 1 : Electrical Analog of Neuron near Figure 1 : Electrical Analog of Neuron near γγ > 0> 0
Vshuu/Work/Genova/PPTW_gen 14/05/0530
Vshuu/Work/Genova/PPTW_gen 14/05/0531
Vshuu/Work/Genova/PPTW_gen 14/05/0532
viHow do living beings extend their genetically predetermined set of actions and percepts with new ones, thus responding effectively to anever-changing, potentially hostile environment? What is really going on in their nervous systems while they sense, act and adapt –in other words, while their intelligence builds up?Medium is the message
Marshal Mc Luhan (1948)
Vshuu/Work/Genova/PPTW_gen 14/05/0533
Vshuu/Work/Genova/PPTW_gen 14/05/0534