Neural Computation of Decisions in Optimization Problems_3

8/14/2019 Neural Computation of Decisions in Optimization Problems_3

1/12

Biol. Cybern: 52, 1 41-152 (1985) B i o l o g i c a lC y b e r n e t i c s9 Springer-Verlag 1985

"Neural" Computation of Decisions in Optimization ProblemsJ . J. H o p f i e l d 1 '2 a n d D . W . T a n k 21 Divisions of Chemistry and Biology, California Institute of Technology, Pasadena, CA 91125, USA2 Department of M olecular Biophysics,AT & T Bell Laboratories, Murray H ill, NJ 07974, USA

Abstract . H i g h l y - in t e r c o n n e ct e d n e t w o r k s o f n o n -l i n e a r a n a l o g n e u r o n s a r e s h o w n t o b e e x t r e m e l ye f fe c ti v e i n c o m p u t i n g . T h e n e t w o r k s c a n r a p i d l yp r o v i d e a c o l l e c t i v e l y - c o m p u t e d s o l u t i o n ( a d i g i t a lo u t p u t ) t o a p r o b l e m o n t h e b a s i s o f a n a l o g i n p u ti n f o r m a t i o n . T h e p r o b l e m s t o b e s o l v e d m u s t b ef o r m u l a t e d i n t e r m s o f d e si r e d o p t i m a , o f t e n s u b j e c t t oc o n s t r a i n t s . T h e g e n e r a l p r i n c i p l e s i n v o l v e d i n c o n -s t r u c ti n g n e t w o r k s t o s o l v e sp e c if ic p r o b l e m s a r e d i s -c u s s ed . R e s u l t s o f c o m p u t e r s i m u l a t io n s o f a n e t w o r kdes igned to so lve a d i f f icu l t bu t we l l -de f ined opt imiza-t i o n p r o b l e m - t h e T r a v e l i n g - S a l e s m a n P r o b l e m - a r ep r e s e n t e d a n d u s e d t o i l l u s t r a t e t h e c o m p u t a t i o n a lp o w e r o f t h e n e t w o r k s. G o o d s o l u ti o n s t o t h is p r o b l e ma r e c o l le c t iv e l y c o m p u t e d w i t h i n a n e l a p s e d t i m e o fonly a few neura l t ime co ns tan t s . The e f fec tiveness oft h e c o m p u t a t i o n i n v o l v e s b o t h t h e n o n l in e a r a n a l o gr e s p o n s e o f t h e n e u r o n s a n d t h e la r g e c o n n e c t i v i t ya m o n g t h em . D e d i c a t e d n e t w o r k s o f b i o lo g i c al o rm i c r o e l e c t r o n i c n e u r o n s c o u l d p r o v i d e t h e c o m p u -ta t iona l capabi l i t i e s desc r ibed for a wide c las s ofp r o b l e m s h a v i n g c o m b i n a t o r i a l c o m p l e x i t y . T h ep o w e r a n d s p e e d n a t u r a l l y d i s p l a y e d b y s u c h c o l l e ct i v en e t w o r k s m a y c o n t r i b u t e t o t h e e f f e c ti v e ne s s o f b i o l o g -ica l in format ion proces s ing .

I Introduct ion

A la rge c las s of log ica l proble m s a r i s ing f rom rea lw o r l d s i t u a t i o n s c a n b e f o r m u l a t e d a s o p t i m i z a t i o np r o b l e m s , a n d t h u s q u a l i t a t i v e ly d e s c r i b e d a s a s e a r c hf o r t h e b e s t s o l u t i o n . T h e s e p r o b l e m s a r e f o u n d i ne n g i n e e r i n g a n d c o m m e r c e , a n d i n p e r c e p t u a l p r o b -l e m s w h i c h m u s t b e r a p i d l y s o l v e d b y t h e n e r v o u ss y s t e m s o f a n i m a l s. W e l l - st u d i e d p r o b l e m s f r o m c o m -m e r c e a n d e n g i n e e r i n g i n c l u d e : G i v e n a m a p a n d t h ep r o b l e m o f d r i v in g b e t w e e n t w o p o i n t s , w h i c h i s t h e

b e s t r o u t e ? G i v e n a c i r c u i t b o a r d o n w h i c h t o p u tchips , wh a t i s the bes t wa y to loca te the ch ips for a goo dwi r ing l ayout (Ki rkpa t r i ck e t a l . , 1983)? Ana logous ,b u t o n l y p a r t ia l l y c h a r a c t e r iz e d p r o b l e m s i n b i o l o g ic a lp e r c e p t i o n a n d r o b o t i c s in c l u d e : ~ G i v e n a m o n o c u l a rp ic ture , wha t i s the bes t th ree -d imens iona l desc r ip t iono f t h e l o c a t i o n s o f t h e o b j e c t s ? I n d e e d , w h a t a r e t h e" o b j e c t s " ? I n e a c h o f t h e s e o p t i m i z a t i o n p r o b l e m s , a na t t e m p t c a n b e m a d e t o q u a n t i f y t h e v a g u e c r i te r i o n" b e s t " b y t h e u s e o f a s p e ci fi c m a t h e m a t i c a l f u n c t i o n t ob e m i n im i z e d .

Whi le a cos t func t ion may be spec i f i ed , rea l wor ldda ta used to eva lua te i t i s genera l ly not prec i se . A l so ,c o m p l e x c o s t f u n c t i o n s u s u a l l y i n v o l v e s o m e w h a ta r b i t r a r y w e i g h t in g s a n d f o r m s o f t h e v a r i o u s c o n t r i -b u t i o n s . F r o m a n e n g i n e e r i n g v i e w p o i n t , t h e s e c o m -p l i c a t i o n s i m p l y t h a t l i t t l e m e a n i n g c a n b e a t t a c h e dto "bes t " . Of ten , wha t i s t ru ly des i red i s a ve ry goods o l u t io n , w h i c h w i l l b e u n i q u e l y b e s t o n l y f o r s i m p l et a sk s . I n m a n y s i tu a t i o n s , a v e r y g o o d a n s w e r c o m -p u t e d o n a t i m e s c al e s h o r t e n o u g h s o t h a t t h e s o l u t i o nc a n b e u s e d i n t h e c h o i c e o f a p p r o p r i a t e a c t i o n i s m o r eiLmpor tan t tha t a nomina l ly-be t t e r "bes t " so lu t ion .Thi s i s e spec ia l ly t rue in the b io logica l and robot i cst a s k s o f p e r c e p t i o n a n d p a t t e r n r e c o g n i t io n , b e c a u s et h e s e p r o b l e m s t y p i c a ll y h a v e a n i m m e n s e n u m b e r o fv a r i a b l e s a n d t h e t a s k o f se a r c h i n g f o r th e m a t h e m a t -i ca l o p t i m u m o f t h e c r i te r i o n c a n o f t e n b e o f c o n s i d -e rable combina tor i a l d i f f i cu l ty , and hence t imec o n s u m i n g .

T h e c o m p u t a t i o n a l p o w e r s r o u ti n e l y u s e d b y n e r-v o u s s y s t e m s t o s o l v e p e r c e p t u a l p r o b l e m s m u s t b et r u l y im m e n s e , g iv e n t h e m a s s i v e a m o u n t o f se n s o r yd a t a c o n t i n u o u s l y b e i n g p r o c e s s e d , t h e i n h e r e n t d i f -f i c u lt y o f t h e r e c o g n i t i o n t a s k s t o b e s o l v e d , a n d t h es h o r t t i m e ( m s e c - se c s ) i n w h i c h a n s w e r s m u s t b e f o u n d .1 (Poggioand Torre, 1985; Terzopoulos, 1984; Ikeuchi andHorn , 1981i Julesz, 1971; M arr, 1982; Sebestyn, 1962)


2/12

14 2M o st ge ne r a l pu r pose d ig i t a l c om pu te r s w ou ld f a il t op r ov ide t h is c om bina t ion o f pow e r a nd spe e d . O n e o fthe cent ra l go a ls of resea rch in neurosc ie nce i s tound e r s t a nd h ow the b iophys i c a l p r ope r ti e s o f ne u r onsa nd ne u r a l o r ga n iz a t ion c om bine to p r ov ide suc him pr e s s ive c om pu t ing pow e r a nd spe e d . A n unde r -s t a nd ing o f b io log ic al c om p u ta t ion m a y a l so l e a d toso lu t ions f o r r e l a t e d p r ob le m s in r obo t i c s a nd da t ap r oc e s s ing u s ing non - b io log ic a l ha r dw a r e a nd so f t -ware .I t i s c l e a r f r om s tud ie s i n a na tom y , ne u r ophys i -o logy , a nd p syc hophys i c s t ha t pa r t o f t he a nsw e r t oh o w n e r v o u s s y s te m s p r o v id e c o m p u t a t i o n a l p o w e ra nd spe e d i s t h r ough pa r a l l e l p r oc e s s ing . The m a m -m a l i a n v i sua l sy s t e m c om pu te s e l e m e n ta r y f e a tu r erecogn i t ion mass ive ly in pa ra l le l ( Ju lesz, 1981; Ba l la rde t a l ., 1983) . At the leve l of neu ra l a rchi tec ture ,a n a t o m y a n d n e u r o p h y s i o lo g y h a v e r e v ea l ed t h a t t h eb r oa d c a t e go r y o f pa ra l l el o r ga n iz a t ion i s m a n i f e s t insevera l d i f fe rent but in te r re la ted forms. Para l le l sen-so r y inpu t c ha nne l s , suc h a s t he ind iv idua l r ods a n dc one s i n t he ve r t e b r a t e r e t i na , a l l ow r a p id r e m o tese ns ing o f t he e nv i r on m e n t a nd da t a t r a nsm is s ion toprocess ing cente r s . Likewise , pa ra l le l ou tpu t ch anne ls ,f o r e xa m ple i n c o r t i c oc o r t ic a l p r o j e c tions i n t he c o r t e x ,con nec t d i f fe rent process ing m odu les ( see , for exam pleG o ldm a n- R a k ic , 1984 ). A n o the r m a n i f e s t a t ion o f pa r -a lM ism oc c u r s i n t he l a r ge de g r e e o f fe e dba c k a nd in -te rconn ec t iv i ty in the " loca l c i rcui t ry" of spec if ic p ro-cess ing a reas ( see, for ex amp le Sh epherd , 1978). Theide a tha t t h i s l a r ge de g r e e o f loc a l c onne c t iv i t y be tw e e nthe s imple process ing uni ts (neurons) in a spec i f icp r oc e s s ing a r e a o f t he ne r vous sy s t e m is a n im p or t a n tc on t r ibu t ion to i t 's c om pu ta t ion a l pow e r ha s l e d t o t hes tudy o f the ge ne r a l p r ope r t i e s o f ne u r a l ne tw or ks 2 a nda lso severa l "connec t ionis t" theor ies in pe rcept ion(Bal la rd , in press ; Fe ldm an and Ba l la rd , 1982). Th ec onne c t ion i s t the o r i e s e m p loy log i c a l ne tw or ks o f tw o-s t a t e ne u r ons i n a d ig i t a l c loc ke d c om pu ta t iona lf r a m e w or k to so lve m o de l pa t t e r n r e c ogn i t i onp r ob le m s .The r e i s a m a jo r f e a tu r e o f ne u r a l o r ga n iz a t ionw h ic h i s no t i nc lude d in c onne c t ion i s t m ode l s bu twh ich can ac t syn ergis t ica lly wi th pa ra l le l f eedbacka nd c onne c t iv i t y t o g r e a t ly e nha nc e c om pu ta t iona lpower . This fea ture i s tha t the b io logica l sys temopera tes in a col lec tive analo9 m o d e , w i t h e a c h n e u r o ns u m m i n g t h e i n p u ts o f h u n d r e d s o r t h o u s a n d s o fo the r s i n o r de r t o de t e r m ine i t s g r a de d ou tpu t . A na na log sys t e m is m a de pow e r f u l in c om pu ta t io n by i t sabi l i ty to adjus t s imul taneously and se l f -cons is tent lym a ny in t e r a c t ing va r ia b l e s (P ogg io a nd K oc h , 1984 ;Ja c kson , 1960 ; H u ske y a nd K o r n , 1962 ). A l thou gh2 (Hopfield, 1984; Gelperin et al., in press; Hop field, 1982;Hi nto n and Sejnowski, 1983; Arbib , 1975)

very fas t , ana lo g su mm at ion is inevi tably less accura tetha n d ig i t a l sum m a t ion . Th i s c om pr om ise i s no tc r i t ica l , however , in pe rceptua l ta sks formula ted asop t im iz a t ion p r ob le m s . The c om pu ta t iona l l oa d o fr a p id ly r e duc ing th i s s e nso r y inpu t t o t he de s i r e d" good " so lu t ion i s a l r e a dy im m e nse ; i na c c u r a c ie s a ndunc e r t a in t i e s a r e a l r e a dy p r e se n t a nd the c om pu ta -t iona l loa d is mean ingless ly inc reased by h igh d ig i ta la c c u r a c y . P a r a l le l a na log c om pu ta t ion in a n e tw or k o fne u r ons i s t hus a natural w a y to o r ga n iz e a ne r voussys t e m to so lve op t im iz a t ion p r ob le m s .

I n t h i s pa pe r w e qua n t i t a t i ve ly de m ons t r a t e t hec om pu ta t ion a l pow e r a nd spe e d o f c o l le c t ive a na logne tw o r ks o f ne u r ons i n so lv ing op t im iz a t ion p r ob le m sr a p id ly . W e d e m o ns t r a t e t ha t e ve n f o r ve r y d if fi c ul tproblem s, mak ing a col lec t ive dec is ion i s r apid , r equ ir -ing an e lap sed t ime of only a few charac te r is t ic t imes ofthe " ne u r ons" c om pr i s ing the ne tw or k . Th i s spe e d ,ne e de d f o r r e a l -t im e p r oc e s sing o f s e nso r y in f o r m a t ionby a ne r vous sy s t e m , c a n be p r ov ide d by c o l l e c t ivea na log c o m p u ta t iona l c i rc u i ts be c a use a l l o f thene u r ons s im u l t a ne ous ly a nd c on t inuou s ly c ha nge the i ra na log s t a t e s i n pa r a l l e l . W he n c om pa r e d to m ode r nd ig i t a l ge ne r a l pu r pose c om pu te r s c ons t r uc t e d w i thconvent iona l s i l icon in tegra ted c i rcui ts (VLSI) , the" ne u r a l " c om pu ta t iona l c i r c u i t s w e de sc r ibe ha vequa l i ta t ive ly d i f fe rent f ea tures and organiza t ion . InV LS I the u se m a de o f a na log c a l c u l a t i ons i n m in im a l( M e a d a nd C onw a y , 1980 ) . Ea c h log i c ga t e w i llt yp i c a lly ob ta in i npu t s f r om tw o o r t h r e e o the r s , a nd ahuge nu m b e r o f i nde pe n de n t b ina r y de c i s ions a r em a d e in t he c ou r se o f a c om pu ta t ion . I n c on t r a s t , e a chnonl inear neura l processor (neuron) in a col lec t ivea na log c om pu ta t iona l ne tw or k ge t s i npu t s f r om t e nso r h u n d r e d s o f o t h e r s a n d a collective solu t ion i sc om pu te d on the ba s is o f t he s im u l t a ne ous i n t e ra c t ionsof hund reds of devices .R e c ogn iz ing tha t t h e nature of pe r c e p tua l p r ob le m si s s im i l ar t o o the r op t im iz a t ion p r ob le m s ( P ogg io a ndTor r e , 198 5 ; H in ton a nd S e jnow sk i , 1983 ; Te r -z opou los , 1984 ) a nd tha t c om p u t ing pow e r is be s ti l lus t ra ted on a d i f f icu l t but we l l un ders too d problem ,w e w i l l show he r e how to o r ga n iz e a c om pu ta t iona lne tw or k o f e x te ns ive ly i n t e r c onne c te d non l ine a r a na -log neurons so tha t i t wi l l so lve a we l l charac te r ized ,bu t non - b io log ic a l , op t im iz a t ion p r ob le m . W e ha vec hose n a s a n i l l u s t r a t i on the " Tr a ve l ing - S a le sm a nP r ob le m " ( TS P ) , f o r w h ic h the c om pu ta t iona l d i f -f icu l ty has been m uc h s tu died (Law ler e t a l. , in press ;G a r e y a nd Joh nson , 1979 ). The so lu t ion to t he TS Ppr ob le m , a nd inde e d , t he so lu t ion to m a ny op t im i -z a t ion p r ob le m s i s a d i s c r e t e a nsw e r . H ow e ve r , t hene u r ons i n t he ne tw or ks w e de sc r ibe ope r a t e i na n a na log m ode . H e nc e , un l ike " c onne c t ion i s t " a p -p r oa c he s t o so lv ing pe r c e p tua l p r ob le m s in ne tw or k s


3/12

14 3

whi c h use s t ri c tl y t wo- s t a t e ne u r ons , t he f o r m ul a t i onof p r ob l e m s t o be so l ve d by a n a na l og c om put a t i ona ln e t w o r k r e q ui re s e m b e d d i n g w h a t s e e m e d to b e d i s -c r e t e pr ob l e m s i n a c o n t i n u o u s dec i s ion space in wh icht h e n e u r o n a l c o m p u t a t io n o p e ra te s . W e d e m o n s t r a t ehe r e how a c on t i nuous de c i s ion spac e a nd c o n t i nuou sc om put a t i on c a n be r e l a t e d t o d i sc r e t e c om put a t i ona nd why a c on t i nuous spa c e c a n im pr o ve t he qua l i ty o ft he so l u t i ons ob t a i ne d by h i gh l y - i n t e r c onne c t e dne ur a l ne t wor ks .II Analog Com putational NetworksThe ge ne r a l s t r uc t u r e o f t he a na l og c om put a t i ona lne t wor ks wh i c h c a n so l ve op t i m i z a t i on p r ob l e m s i sshow n i n F ig . l b . The se ne t wor ks ha ve t he t h r e e m a j o rf o r m s o f pa r al le l o r ga n i z a t ion f ound i n ne ur a l sys te m s :para l l e l i npu t channe l s , para ll e l outpu t channe l s , and al a r ge a m o un t o f i n t e r c onne c t iv i t y be t we e n t he ne ur a lprocess ing e l ement s . The process ing e l ement s , or"neurons" , a re model l ed as ampl i f i e r s in conjunc t ionwi th feedb ack c i rcui ts com pr i sed of wires , res i s tors andc a pa c i t o r s o r ga n i z e d so a s t o m ode l t he m os t ba s i c( a )

(b)

// J , 1 ,-u 0 0 +Uou

~ _ i n p u t s . . . . ~

[2 1

~ ' 1 1 3

ff] i

V a m p l if ie r V i n v e r t i n g a m p l if ie r9 r e s i s to r n T i j n e t w o r kF i g . l a a n d b. a T h e i n p u t - o u t p u t r e l a t i o n f o r t h e " n e u r o n s " o ra n a l o g a m p l i fi e r s, b T h e a n a l o g c i r c u it . T h e o u t p u t o f a n y n e u r o nc a n p o t e n t i a ll y b e c o n n e c t e d t o t h e i n p u t o f a n y o t h e r n e u r o n .B l a c k c i r c l e s a t i n t e r s e c t i o n s r e p r e s e n t r e s i s t i v e c o n n e c t i o n s( T q 's ) b e t w e e n o u t p u t s a n d i n p u t s . C o n n e c t i o n s b e t w e e n i n v e r t e do u t p u t s a n d i n p u t s r e p r e s e n t n e g a t i v e ( i n h i b i t o r y ) c o n n e c t i o n s

c om put a t i ona l f e a t u r e s o f ne u r ons , na m e l y a xons ,de ndr i t i c a r bo r i z a t i on , a nd syna pse s c onne c t i ng t hedi f fe rent neuron s . The am pl i fi e r s have s igm oid mo no -tonic input -output re l a t ions , as shown in F ig . l a . Thef unc t i on V j= g j ( u j ) which charac te r i zes th i s input -ou t pu t r e l a t i on de sc r i be s t he ou t pu t vo l t a ge o fampl i f i e r Vj due to an inpu t vo l t age uj. The t imeconstant s of the ampl i f i e r s a re assumed negl ig ible .How e ve r , l ike t he i npu t i m pe da nc e c a use d by t he c e llm e m b r a ne i n a b i o log i c a l ne u r on , e a c h a m pl i fi e r j ha san inp ut res i s tor Qj l eading to a re fe rence gro un d a nda n i npu t c a pa c i t o r C j. The se c om pone n t s pa r t i a l l yde f ine (see be low) t he t i m e c ons t a n t s o f t he ne ur on s a ndpr ov i de f o r i n t e g r a t i ve a na l og sum m a t i on o f t hesyna p t i c i npu t c u r r e n t s f r om o t he r ne u r ons i n t hene t wor k . I n o r de r t o p r ov i de f o r bo t h e xc i t a t o r y a ndi nh i b i t o r y syna p t i c c onne c t i ons be t we e n ne ur onswhi l e us ing convent iona l e l ec t r i ca l component s , eacha m pl i fi e r is g i ven t wo ou t pu t s , a n o r m a l ( + ) ou t pu ta n d a n i n v e r te d ( - ) o u t p u t. T h e m i n i m u m a n dm a x i m um ou t pu t s o f the no r m a l a m pl i f ie r a r e t a ke n a s0 a nd 1 , wh i l e t he i nve r t e d ou t pu t ha s c o r r e spond i ngva l ue s o f 0 a nd - 1 . A syna pse be t we e n two n e ur ons i sde f i ne d by a c on duc t a nc e Tu whi c h c onne c t s one o f thetwo outpu t s o f ampl i f i e r j t o the inpu t of ampl i f i e r i.Th i s c onne c t i on i s m a de wi t h a r e s i s t o r o f va l ueRij= 1/[Tu[ . I f t he synapse i s exc i t a tory (T u>0 ) , t h i sr e s is t o r i s c onne c t e d t o t he no r m a l ( + ) ou t pu t o fampl i f i e r j . F or an inh ibi tory synap se (Tu


4/12

144

T h e e q u a t i o n o f m o t i o n d e s c r ib i n g t h e t i m e e v o -lu t i on o f t h i s c i r cu i t i s :

N6 (d u J d t ) = Z T j V s-u ,/ R i + I i ( 1 )j = l= g j ( u j )

R i i s a p a r a ll e l c o m b i n a t i o n o f r a n d t h e R~j:N1 / R ~ = I / o , + Z 1 / R , . (2 )j = l

F o r s i m p l i c i ty , i n t h e p r e s e n t w o r k w e h a v e a l w a y sc h o s e n g i = g , Ri=R, a n d C~ = C , i n d e p e n d e n t o f i ,t h o u g h t h i s i s n o t n e c e s s a r y . D i v i d i n g b y C a n drede f in ing T q / C a n d I ] C a s T q a n d I i , t h e e q u a t i o n s o fm o t i o n b e c o m e :

Nd u d & = - u J z + Z T i j V j + [ i (3 )j = lz = R CVs = g ( u s) .F o r a n " i n i t i a l- v a l u e " p r o b l e m , i n w h i c h t h e i n p u tvo l t ages o f t he ne uro ns a re g iven va lues u~ a t t ime t = 0 ,t h i s e q u a t i o n o f m o t i o n p r o v i d e s a f u l l d e s c r i p t i o n o ft h e t i m e e v o l u t i o n o f th e s t a t e o f th e c i r c u it . I n t e g r a t i o no f t h i s e q u a t i o n i n a d i g i t a l c o m p u t e r a l lo w s a n yh y p o t h e t i c a l n e t w o r k t o b e s i m u l a t e d .

I n e a r l i e r w o r k ( H o p f i e ld , 1 9 84 ) i t w a s s h o w n t h a tt h e e q u a t i o n s o f m o t i o n f o r a n e t w o r k w i t h s y m m e t r i ccon nec t io ns (T is = T ii ) a lwa ys l ead t o a co nve rg ence t os t a b l e s t a t e s , i n w h i c h t h e o u t p u t s o f a ll n e u r o n sr e m a i n c o n s t a n t . A l s o , w h e n t h e w i d t h o f th e a m p l i fi e rg a i n c u r v e i n F i g . l a i s n a r r o w - t h e h i g h - g a i n l i m i t -t h e s t a b le s t a t e s o f a n e t w o r k c o m p r i s e d o f N n e u r o n sa r e t h e l o c a l m i n i m a o f t h e q u a n t i t y

N N Ne = - 1/2 E Z TqV~V - Z Vd~. (4)i= 1 j= l i = lT h e s t a t e s p a c e o v e r w h i c h t h e c i r c u i t o p e r a te s i s t h ei n t e r io r o f th e N - d i m e n s i o n a l h y p e r c u b e d e f i n e d b yV~= 0 o r 1 . H o w e v e r , i n t h e h i g h - g a i n li m i t, th e m i n i m ao n l y o c c u r a t c o r n e r s of t h i s space . He nce t h e s t ab l es t a te s o f t h e n e t w o r k c o r r e s p o n d t o t h o s e l o c a t i o n s i nthe d i sc re t e space cons i s t i ng o f t he 2 N corne r s o f t h i sh y p e r c u b e w h i c h m i n i m i z e [ E q . ( 4) ].N e t w o r k s o f n e u r o n s w i t h th i s b a si c o r g a n i z a t i o nc a n b e u s e d t o c o m p u t e s o l u t i o n s t o s p e c i f i c o p t i m i -z a t i o n p r o b l e m s b y f i r s t c h o o s i n g c o n n e c t i v it i e s ( Tq )a n d i n p u t b i a s c u r r e n t s ( I~ ) w h i c h a p p r o p r i a t e l y r e p r e -s e n t t h e f u n c t i o n t o b e m i n i m i z e d . T h e m e t h o d s i n -vo lved i n t h i s se l ect i on a re d i scussed be low. Fo l low ingt h is c o n s t r u c t i o n o r " p r o g r a m m i n g " o f th e n e t w o r k , a nin i t i a l se t o f i np u t v o l t ages u i a re p rov id ed , an d t hea n a l o g s y s t e m t h e n c o n v e r g e s t o a s t a b l e s t a te w h i c h

m i n i m i z e s t h e f u n c t i o n . W e i n t e r p r e t t h e s o l u t i o n t ot h e p r o b l e m f r o m t h e f i n a l s t a b l e s t a t e . F o r t h ep r o b l e m s c o n s i d e r e d h e r e , t h e s o l u t i o n s a r e d i s c r e t e( d ig i ta l ) a n d t h e g a i n i s c h o s e n h i g h e n o u g h s o t h a t i nt h e f i n a l s t a b l e s t a t e e a c h n e u r o n h a s a n o r m a l ( + )o u t p u t V~ v e r y n e a r 0 o r 1 . T h e s e t o f o u t p u t s t h e np r o v i d e s a d i g i t a l a n s w e r w h i c h r e p r e s e n t s a s o l u t i o nt h e p r o b l e m .

B e f o r e w e c o n s i d e r t h e f o r m o f a n e t w o r k w h i c hs o l v e s t h e T S P , i t i s i n s t r u c t i v e t o c o n s i d e r h o w as i m p l e r o p t i m i z a t i o n p r o b l e m c a n b e s o l v e d b y o n e o ft h e s e c o m p u t a t i o n a l n e t w o r k s . A l t h o u g h n o t i n t e r -p r e t e d a s a n o p t i m i z a t i o n p r o b l e m a t t h a t t i m e , a ne x a m p l e w a s a c t u a l l y p r o v i d e d i n e a r l i e r w o r k ( H o p -f ie l d, 19 84 ) w h e r e i t w a s s h o w n h o w t h e s a m e c o m p u -t a t i ona l c i r cu i t de sc r ibed above cou ld , wi th t he p rope rc h o i c e o f c o n n e c t i o n s t r e n g t h s , o p e r a t e a s a C o n t e n t -A d d r e s s a b l e - M e m o r y (C A M ) . T h e n o r m a l o u t p u t s o ft h e N a m p l i f e r s c o m p r i s in g t h e m e m o r y c i r cu i t -w h i c h f o r t h a t a p p l i c a t i o n w e r e a l l o w e d th e r a n g e - 1t o + 1 , i n s t e a d o f th e 0 t o 1 r a n g e d e s c r i b e d a b o v e -w e r e a l w a y s - 1 o r 1 i n th e h i g h - g a i n l im i t a n d t h e s t a t eo f t h es e o u t p u t s r e p r e se n t e d a b i n a r y w o r d i n m e m o r y .A m e m o r y , s t o r e d i n t h e n e t w o r k b y a n a p p r o p r i a t ec h o i c e o f T q el e m e n t s , c o u l d b e " r e t r i e v e d " b y s e t t in gt h e o u t p u t s o f t h e a m p l if i e rs i n t h e b i n a r y p a t t e r n o f th er e c al l k e y a n d t h e n a l l o w i n g t h e s y s t e m t o c o n v e r g e t oa s t a b l e s t a te . T h i s s t a b l e s t a t e w a s i n t e r p r e t e d a s t h em e m o r y w o r d e v o k e d b y t h e k e y . E a c h r e ca l l " s o l v e d "t h e " p r o b l e m " o f w h i c h o f th e s t o r e d b i n a r y w o r d s w a s" c l o s e s t " t o t h e k e y w o r d .

W e c a n u n d e r s t a n d h o w t o c o n s t r u ct a n a p p r o p r i -a t e c o m p u t a t i o n a l c i r c u i t f o r t h e C A M , c o n s i d e r e dn o w a s a s i m p l e e x a m p l e o f a n o p t i m i z a t i o n p r o b le m ,by examin ing t he E func t ion . S ince E [Eq . (4 ) ] de t e r -m i n e s t h e s t a b l e s t a te s o f t h e n e t w o r k , t h e n i f w e w i s ht h e s t a b l e s t a te s t o b e a p a r t i c u l a r s e t o f m m e m o r ys t a t e s V ~ s = { 1 , 2 , . . . , m } w e m u s t c h o o s e t h e c o n n e c -t i o n s t r e n g t h s ( T q) a n d t h e i n p u t b i a s c u r r e n t s ( Ii ) o f t h en e t w o r k s u c h t h a t E q . ( 4 ) i s a l o c a l m i n i m a w h e n t h esys t em i s i n each one o f t he s t a t e s VL Since Eq . (4 ) i sq u a d r a t i c a g u e s s m i g h t b e :E = - 1 /2 ~ (V s - V) 2 . (5 )S = II f t he s t a t e ve c to r V (wi th com po nen t s V~) i s a r a nd omv e c t o r, t h e n e a c h p a r e n t h e s i z e d t e r m i s v e r y s m a l l. B u ti f V i s o n e o f th e m e m o r i e s V *, t h e n o n e t e r m i n t h e s u mis N z. H e n c e t h e n e t w o r k h a s a n e n e r g y m i n i m a o fd e p t h a p p r o x i m a t e l y - 1 / 2 N z a t e a c h o f t h e a s s i g n e dm e m o r i e s . E q u a t i o n (5 ) c a n b e r e w r i t t e n i n t h e s t a n -d a r d f o r m [ E q . ( 4) ] o f t h e e n e r g y f u n c t i o n i f al l I i = 0and the T~j e l emen t s a re de f ined a s :r ,= v:v; .

S = l


5/12

145This equat ion for T~s s the storage algorithm presentedearlier (Hopfield, 1984) except for an additive constant.It is derived above by thinking of the CAM as anoptimization problem and then making a judiciouschoice of the representation of the energy function interms of the desired memories. In a practical applica-tion of a CAM, for example a network used to storetelephone numbers, in addition to the storage al-gorithm above, a transformation used to code the real-world information into the binary word memory datarepresentation is required. Taken together, the datatransformation and the algorithm for the T~s can beconsidered the "map " of this problem onto the analogcomputational network.The basic property of the analog computationalnetworks described above is the minimization of E.The CAM example illustrates that through the con-struction of an appropriate energy function for thecircuit, and a strategy for interpreting the state of theoutputs as a solution, a simple optimization problemmay be "mapped" onto the network. We have recentlyfound tha t these circuits can also rapidly solve difficultoptimization problems which have both contraints inthe possible solutions and also combinatorial com-plexity. A network designed to solve the "Traveling-Saleman Problem" illustrates this computationalpower.

I I I T h e T S P P r o b l e mThe "Traveling-Salesman Problem" (TSP) is a classicof difficult optimization. It is simple to describe,mathematica lly well characterized and makes a goodvehicle for describing the used of neural analog compu-tational networks to solve difficult optimization prob-lems. A set of n cities A, B, C ... have (pairwise) dis-tances of separation dA B , dac . . . . dB c . . . . The prob-lem is to find a closed tour which visits each city once,returns to the starting city, and has a short (orminimum) total path length. A tour defines somesequence B , F , E , G , . . . , W in which the cities arevisited, and the total path length d of this tour isd = d~v + dF E + . . . + dwB .

The actual bes t solution to a TSP problem iscomputationally very hard - the problem isnp-complete (Garey and Johnson, 1979), and the timerequired to solve this problem on any given computergrows exponentially with the number of cities.The solution to the n-city TSP problem consists ofan ordered list of n cities. To "map " this problem ontothe computationa l network, we require a representa-tion scheme which allows the digital output states ofthe neurons operating in the high-gain limit to be

decoded into this list. We have chosen a represen tationscheme in which the final location of any individualcity is specified by the output states of a set of nneurons. For example, for a 10-city problem, if city A isin position 6 of the tour which is the solution to theproblem then, as shown below, this is represented bythe sixth neuron out of a set of ten having an o utputV6 = 1 with all other outputs at 0:0 0 0 0 0 1 0 0 0 0 .

This representation scheme is natural, since anyindividual city can be in any one of the n positions inthe tour list. For n cities, a total of n independent se ts ofn neurons are needed to represent a complete tour.This is a total of N = n 2 neurons. The output state ofthese n z neurons which we will use in the TSPcomputational network is most conveniently dis-played as an n x n square array. Thus, for a 5-cityproblem using a total of 25 neurons, the neuronal stateB 1 2 3 4 5A 0 1 0 0 0

0 0 0 1 01 0 0 0 0 (7)

D O 0 0 0 10 0 1 0 0

shown above would represent a tou r in which city C isthe first city to be visited, A the second, E the third, etc.[The total length of the 5-city path is dcA + dAE + deB+ d B o + d v c . ] Each such final state of the array ofoutput s describes a part icular tour of the cities. Anycity cannot be in more than one position in a valid tour(solution) and also there can be only one city at anyposition. In the n x n "square" representation thismeans that in an output state describing a valid tourthere can be only one "1" output in each row and eachcolumn, all other entries being zero. Likewise, any sucharray of output values, called a permutation matrix,can be decoded to ob tain a tour (solution). An exampleof the final state of a 10-city problem is shown inFig. 2d.

For an n-city TSP problem, there are n! states of thegeneral form [Eq. (7)] above. However, a tour de-scribes an or der in which cities are visited. Fo r an n-cityproblem, there are 2n tours of equal path-length, foreach path has an n-fold degeneracy of the initial city ona tour and a 2-fold degeneracy of the tour sequenceorder. There are thus n! /2n d i s t inc t paths for closedTSP routes.

Because of our representation of neural outputs ofthe TSP computational network in terms of n rows of nneurons, the N symbols Vii will be described by doubleindicies Vx, j. The row subscript has the interpretationof a city name, and the column subscript the position of


6/12

146( o ) . . . . . 9 9 , . 9 ( b) . . . . . . 9 9 9 9

. . . . . . . . " . . . . . . . . m m . . . . . . . m . . . . . . . . , , , . . . . , , . .

, m , . . . . . . . m m . . . . .. . . . 9 . . . . . . . . . mm . . . . . . . . , m . . . . . . . . , . . . . . . . m m m . . . . . . .. . . . . . . . . . . . . . i N , .

( c ) . . . . . . m 9 9 99 . . . . . . . i N

. . . . 9 l . . . .

( d . . . . . . . . . ,~

CD

. . . . . . . . . E'm - i i i l -m . P C r T Y

. . . . Gt i : : : : : : :y

. . . . . . . m . 9 dt 2 '3 4 5 6 7 8 91 0\ /

P O S I T I O N ' IN P A T HP A T H = D H W G E A d C 8

Fig. 2a-d. The convergenceof he 10-cityanalog circuit to a tour.The linear dimension of each square is proportional to the v alueof Vx v a, b, e intermediate times, d the final state. The indices ll dillustrate how the f inal state is decoded into a tour (solution ofTSP)tha t c i t y i n a t ou r [ c f. (7) ]. W e wi l l u se t he se two ind i ce si n s t e a d o f o n e t o l a b e l a ll o f t h e n e u r o n s b e c a u s e i ts i m p li f ie s t h e i n t e r p r e t a t i o n o f t h e e q u a t i o n s d e s c r ib -i n g th e e n e r g y f u n c t i o n . L i k e i n t h e C A M p r o b l e m , t h i sE f u n c t i o n w i l l a i d c o n s t r u c t i o n o f a n a p p r o p r i a t ec o m p u t a t i o n a l n e t w o r k f o r t h e T S P .T h e T S . P E n e r g y F u n c t i o nT o e n a b l e t h e N n e u r o n s i n t h e T S P n e t w o r k t oc o m p u t e a s o l u t i o n to t h e p r o b le m , t h e n e t w o r k m u s tb e d e s c r i b e d b y a n e n e r g y f u n c t i o n i n w h i c h t h e l o w e s te n e r g y s t a t e ( t h e m o s t s t a b l e s t a t e o f t h e n e t w o r k )c o r r e s p o n d s t o t h e b e s t p a t h . T h i s c a n b e s e p a r a t e di n t o t w o r e q u i r e m e n t s . F i r s t , t h e e n e r g y f u n c t i o n m u s tf a v o r s t r o n g l y s t a b l e s t a t e s o f t h e f o r m o f a p e r m u -t a t i o n m a t r i x , s u c h a s t h o s e s h o w n i n E q . ( 7 ) o r i nF i g . 2 d , r a th e r t h a n m o r e g e n e r a l s t at e s . S e c o n d , o f -t h en ! s u c h s o l u t i o n s , a l l o f w h i c h c o r r e s p o n d t o v a l i dt o u r s , it m u s t f a v o r t h o s e r e p r e s e n t i n g s h o r t p a t h s . A na p p r o p r i a t e f o r m f o r t h i s f u n c t i o n c a n b e f o u n d b yc o n s i d e r i n g t h e h i g h g a i n l i m i t , i n w h i c h a l l f i n a ln o r m a l ( + ) o u t p u t s w i ll b e 0 o r 1 . A s b e f o r e, t h e s p a c eo v e r w h i c h t h e e n e r g y f u n c t i o n [ E q . ( 4) ] is m i n i m i z e din t h i s l im i t i s t he 2 N c o r n e r s o f t h e N - d i m e n s i o n a lh y p e r c u b e d e f i n e d b y V ~ =0 o r 1 . C o n s i d e r t h o s ec o r n e r s o f th i s s p a c e w h i c h a r e t h e l o c a l m i n i m a ( s ta b l es t a te s ) o f t h e e n e r g y f u n c t i o nE = A / Z Z Z Z V x i V x j + B / 2 E E Z V x,V rfX i j* i i X X* Y

where A, B , and C a re pos i t i ve . The f i r s t t r i p l e sum i sz e r o i f a n d o n l y i f e a c h c i ty r o w X c o n t a i n s n o m o r et h a n o n e " 1 " , (t h e r e s t o f th e e n t r i e s b e i n g z e r o ). T h es e c o n d t r i p l e s u m i s z er o i f a n d o n l y i f e a c h " p o s i t i o n i nt o u r " c o l u m n c o n t a i n s n o m o r e t h a n o n e " 1 " ( th e re s to f t he en t r i e s be ing ze ro ). The t h i rd t e rm i s z e ro i f an do n l y i f t h e r e a r e n e n t r ie s o f " t " i n t h e e n t i r e m a t r i x .T h u s t h is e n e r g y fu n c t i o n e v a lu a t e d o n t h e d o m a i n o ft h e c o r n e r s o f t h e h y p e r c u b e h a s m i n i m a w i t h E = 0 f o ra l l s t a t e m a t r i c e s w i t h o n e " 1 " i n e a c h r o w a n d c o l u m n .A l l o t h e r s t a t e s h a v e h i g h e r e n e r g y . H e n c e , i n c l u d i n gt h e s e t e r m s i n a n e n e r g y f u n c t i o n d e s c r i b i n g a T S Pn e t w o r k s t r o n g l y f a v o r s s t ab l e s t a t e s w h i c h a r e a t l e a s tv a l i d t o u r s i n t h e T S P p r o b l e m , a n d f u lf il ls t h e f i rs tr e q u i r e m e n t f o r E .T h e s e c o n d r e q u ir e m e n t , th a t E f a v o r va l id t o u r sr e p r e s e n t i n g s h o r t p a t h s , i s f u lf il le d b y a d d i n g o n ea d d i t i o n a l t e r m t o t h e f u n c t i o n . T h i s t e r m c o n t a i n si n f o r m a t i o n a b o u t t h e l e n g t h o f t h e p a t h c o r r e s p o n d -i n g t o a g i v e n t o u r , a n d i t s f o r m c a n b e c h o s e n a s1 / 2 D Z Z Z d x y V x i ( V r , i + l + V r, i - O . (9)

X Y*X iF o r n o t a t i o n a l c o n v e n i e n c e , s u b s c r i p t s a r e d e f i n e dm o d u l o n , i n o r d e r t o e x p r e s s e a s il y " e n d e f f ec t s " s u c ha s t h e f a c t t h a t t h e n ' t h c i t y o n a t o u r i s a d j a c e n t i n t h et o u r t o b o t h c i t y ( n - 1 ) a n d c i t y 1 : i . e . , V y , , + j = V r . j .W i t h i n t h e d o m a i n o f s t at e s w h i c h c h a r a c t e r i z e a v a l idt o u r , t h e a b o v e t e r m E E q . ( 9) ] is n u m e r i c a l l y e q u a l t ot h e l e n g t h o f t h e p a t h f o r t h a t t o u r .

A n a p p r o p r i a t e t o t a l e n e rg y f u n c t i o n f o r t h e T S Pn e t w o r k c o n s i s ts o f t h e s u m o f E q . (8 ) a n d E q . (9 ) . I fA, B , an d C a re su f f i c i en t ly l a rge , a l l t he r ea l l y l owe n e r g y s t a t e s o f a n e t w o r k d e s c r i b e d b y t h i s f u n c t i o nw i l l h a v e t h e f o r m o f a v a l id t o u r . T h e t o t a l e n e r g y o ft h a t s t a t e w i ll b e th e l e n g t h o f t h e t o u r , a n d t h e s t a t e sw i t h t h e s h o r t e s t p a t h w i l l b e t h e l o w e s t e n e r g y s t a t e s .

T h r o u g h E q s . ( 3 ) a n d ( 4) , t h e q u a d r a t i c t e r m s i n t h i se n e r g y f u n c t i o n d e f i n e a c o n n e c t i o n m a t r i x a n d t h el i n e a r t e r m s d e f i n e i n p u t b i a s c u r r e n t s . U s i n g t h er o w / c o l u m n n e u r o n l a b e l i n g s c h e m e d e s c r i b e d e a r l i e rfo r each o f t he two ind i ce s , t he impl i c i t l y de f inedc o n n e c t i o n m a t r i x i s ( w i t h b r i e f d e s c r i p t i o n s o f t h em e a n i n g s o f t h e v a r i o u s t e rm s ) :Txi, r j =- - AO xy(1 - bij) " i n h i b i t o r y c o n n e c t i o n s w i t h ine a c h r o w "

- B 3 u ( 1 - 3 x r ) " i n h ib i t o ry c o n n ec t io n s w it h ine a c h c o l u m n "- C " g l o b a l i n h i b i t io n "- D dxr(Sj , i + 1 + 5j , i - 1) "da ta te r m "E 3 u = l i f i = j a n d i s 0 o t h e r w i s e ] . ( 10 )

T h e e x t e r n a l i n p u t c u r r e n t s a r e :I x i = + C n " e x c i t a t i o n b i a s " . ( 11 )


7/12

1 4 7

T h e " d a t a t e r m " c o n t r i b u t i o n , w i t h c o e f f ic i e n t D , t oT x , , r j i s t h e i n p u t w h i c h d e s c r i b e s w h i c h T S P p r o b l e m( i.e. , whe re the c i t ie s ac tua l ly a re ) i s to be so lved . Thete rms w i th A , B , and C coef f i ci en t s pro vide the g enera lc o n s t r a i n t s r e q u i r e d f o r a n y T S P p r o b l e m . W i t h th i s Ef u n c t i o n g u i d i n g t h e d y n a m i c s o f t h e c ir c u it , t h en e t w o r k s h o u l d c o m p u t e t h e s o l u t i o n b y c h o o s i n g af in a l s ta t e w h i c h h a s t h e f o r m o f a p e r m u t a t i o n m a t r i x[Eq. (7) ] a f t e r s t a r t ing in som e in i ti a l unb iased s t a t e .T h e " d a t a t e r m " c o n t r i b u t i o n c o n t r o l s w h i c h o n e o fthe n! s e t of these pro per ly con s t ra ined f ina l s t a t es isa c t u a l l y c h o s e n a s t h e " b e s t " p a t h .

IV TS P Simulat ion Result s

A n e t w o r k f o r a 1 0 - c it y p r o b l e m u s i n g t h e c o n n e c t i o nm a t r i x d e f i n e d i n E q . (1 0 ) a n d t h e i n p u t b i a s t e rm s o fEq. (11) was s im ula ted on a d ig i t a l com pute r . Th el o c a t i o n s o f t h e 1 0 c i ti e s w e r e c h o s e n a t r a n d o m ( w i thu n i f o r m p r o b a b i l i t y d e n s i ty ) o n t h e i n t e r io r o f a t w o -d i m e n s i o n a l s q u a r e o f ed g e l e n g t h 1 . T h e s e c h o i c e sdef ined a pa r t i cu la r s e t of d x r a n d h e n c e T x i , r j t h r o u g hE q . ( 1 0 ). T h e a n a l o g n e t w o r k f o r t h is p r o b l e m g e n -e r a t ed t h e e q u a t i o n s o f m o t i o nd u x i / & = - U x i /Z - A 2 V x j - B Z V r ij * i Y * X

- D Z dxy(V , ,+l + i ) (12)YV x i = g ( U x i ) = 891 + t a n h ( u x i / U o ) ) (for al l X , i ) .T h e s e e q u a t i o n s o f m o t i o n h a v e t h e f o r m d e s c r ib e d i nan ea r l i e r s ec t ion , bu t show the spec i f i c cont r ibut ionsm a d e b y t he T x ~ , r j a n d Ix ~ t e r m s . T h e p a r a m e t e r " n "w a s n o t f i x e d a s 1 0 , b u t w a s u s e d t o a d j u s t t h e n e u t r a lp o s i t i o n o f t h e a m p l i f ie r s w h i c h w o u l d o t h e r w i s e a l s on e e d a n a d j u s t a b l e o f f se t p a r a m e t e r i n t h e i r g a i nf u n c t io n s . T h e o f f se t h y p e r b o l i c t a n g e n t f o r m o f th eg a i n c u r v e w a s c h o s e n t o r e s e m b l e r e a l n e u r a l i n p u t -ou tpu t re l a t ions as we l l a s the charac te r i s t i c s of as imple t rans i s tor ampl i f ie r . The se t of pa ram ete rs int h e s e e q u a t i o n s o f m o t i o n i s o v e r c o m p l e t e , f o r t h e t im ei t ta k e s t o c o n v e r g e is i n a r b i t r a r y u n i ts . W i t h o u t l o s sof genera l ity , z can be se t to 1 .

I n o u r s i m u l a t i o n s , a n a p p r o p r i a t e g e n e r a l s i z e o ft h e p a r a m e t e r s w a s e a s il y f o u n d , a n d a n a n c e d o t a le x p l o r a t i o n o f p a r a m e t e r v a l u e s w a s u s e d t o f in d ag o o d ( b u t n o t o p t i m i z e d ) o p e r a t i n g p o i n t . R e s u l t s i nt h is s e c t i o n r e fe r t o p a r a m e t e r s e ts a t o r n e a rA = B = 500 C = 200D = 5 0 0 Uo = 0 . 0 2 n = 1 5 .

S i n ce w e h a v e n o a p r i o r i k n o w l e d g e o f w h i c h t o u r sa r e b e s t , a n d t h e n e t w o r k a l r e a d y h a s i n t h e " d a t at e r m " t h e n e c e s s a r y i n p u t t o s o l v e t h e p r o b l e m , w ew a n t t o p i c k t h e i n i ti a l v a l u e s o f t h e n e u r a l i n p u tv o l t a g e s ( Ux i ) w i t h o u t b i a s in f a v o r o f a n y p a r t i c u l a rt o u r . A s e n s i b l e c h o i c e m i g h t s e e m t o b e U x i = U o o ,wh ere Uoo i s a con s tan t which i s chose n so th a t a t t = 0Z Z Vxi= 10x i

which i s a lso , approx ima te ly , the de s i red va lue of th i ss u m a t t = ~ . H o w e v e r , t h is u n b i a s e d c h o i c e i s ad i s a s t e r t o t h e c o m p u t a t i o n . S i n c e e a c h p a t h h a s 2 ne q u i v a l e n t t o u r s d e s c r i b in g i t , t h e s y s t e m h a s n o w a y t oc h o o s e o n e o f t h e m g i v e n a n u n b i a s e d s t a r t, a n d t h u sc a n n o t c o n v e r g e t o a t o u r a t a l l . T h e p r o b l e m i sequiv a lent to the fac t ( in c l as s ica l phys ics ) tha t a penc i lp o i s e d e x a c t l y v e r ti c a ll y o n i ts p o i n t m u s t n o t f a l l o v e r ,s in c e to d o s o w o u l d b e t o c h o o s e a d i r e c t i o n i n w h i c ht o f al l. A s im i l a r p r o b l e m o f " b r o k e n s y m m e t r y "a p p e a r s i n m a g n e t i c p h a s e t r a n s i t i o n s ( A n d e r s o n ,1984) . I t i s the re fore neces sa ry to add some noi se ~ U x ~Uxi =- Uoo - t - ~U xit o the in i t i a l va lues . Thi s has the des i red e f fec t ofb r e a k i n g t h e s y m m e t r y a n d a l l o w i n g t h e s y s t e m t oc h o o s e a t o u r , b u t a l s o in s e r ts a s m a l l r a n d o m b i a s in t ot h e c h o i c e o f p a t h .F i g u r e 2 s h o w s t h e r e s u l ts o f a s i m u l a t i o n w h i c hi l lus t ra t e the co nverg ence of a typ ica l such s t a r t ings t a t e t o a f in a l p a th . T h e s y m m e t r y - b r e a k i n g ~ U x ~ w e r ee a c h r a n d o m l y c h o s e n u n i f o r m l y i n t h e i n t e r v a l :- O . l u o ~ 6 U x i < O . l u o .T h e l i n e ar d i m e n s i o n s o f t h e s q u a r e s i n F i g . 2 a r ep r o p o r t i o n a l t o t h e o u t p u t s o f t h e " n e u r o n s " i n t h ea r r a y . I n i ti a l ly t h e y a r e v e r y n e a r l y u n i f o r m a n d a s t im epasses (F igs . 2a -c ) they conv erge to a f ina l t imeinde pen den t s t a t e (F ig. 2d) . The se t of V x~ a re n o t ap e r m u t a t i o n m a t r i x o f f o r m [ E q . (7 )] t h r o u g h o u t t h ec o m p u t a t i o n . T h i s i s b e c a u s e t h e a c t u a l d o m a i n o ff u n c t i o n o f th e n e t w o r k i s n o t a t t h e c o r n e r s o f t h eN - d i m e n s i o n a l h y p e r c u b e d e f in e d b y Vx~ = 0 or 1 , bu tra the r in i t ' s i n t e r i o r . H o w e v e r , n o t i c e t h a t t h e f i n a lo u t p u t s ( Fi g. 2 d ) p r o d u c e a p e r m u t a t i o n a r r a y w i t ho n e n e u r o n " o n " a n d t h e r es t " o f f ' i n e a c h ro w a n d,column, and th i s s t a te thus rep resent s a l eg i t imate tour .'T h e c h o i c e o f n e t w o r k p a r a m e t e r s w h i c h p r o v i d e s: g o o d s o l u t i o n s i s a c o m p r o m i s e b e t w e e n a l w a y s o b -ta in ing l eg i t imate tours (D smal l ) and weight ing thedi s t ances heavi ly (D la rge). A l so , a s expec ted , to o l a rgeUo ( low ga in) resu l t s in f inal s t a t es in w hich the v a lues o fV x i a r e n o t n e a r 1 o r 0 . T h e s e s t a te s a r e n o t p e r m u t a -t i o n m a t r i c e s a n d h e n c e r e p r e s e n t i n v a l i d t o u r s . T o osmal l Uo y ie lds a po ore r s e lec t ion of goo d pa ths .


8/12

148( o )

t0. 8(L 60 .40 .2

"r

A D= 217.I I [ I I0 . 2 0 . 4 0 . 6 0 . 8 " t

( b )

D : 2 . 8 3I I ~ I I0 . 2 0 . 4 0 . 6 Q 8 1

( c )

; 02 04 06 08Fig. 3a-c. a, b Paths found by the analog convergence on 10random cities. The exam ple in a is also the shortest path. Th ecity names A .. . J used in Fig. 2 are indicated, c A typical pathfound using a two-state network instead o f a continuous o ne

( a )

0 .S0 . 60 , 40 , 2

I I I lo ; o 2 o 4 o 6 o 6

(b)D = 4 . 2 6

I I I i I I0 0 . 2 0 . 4 0 , 6 0 . 8 t

(c }D = 5 . 0 7

0 0 . 2 0 . 4 0 . 6 0 . 8 '1

Fig. 4a-c. a A random tour for 30 rando m cities, b The Lin-Kernighan tour. c A typical tour obtained from the analognetwork by slowing increasing the gain

(a} (b)4000" 1 2 5 0 0

`10000 3000re- a~W Wm 7 5 0 0 m 2000D Dz 5 0 0 0 z t 0 0 02 5 0 0

0 00 2 ~ 4 6

T O U R L E N G T H

0 5 t 0 t 5TOUR LENGTH

Fig. 5a and b. a A histogram of the number of different pathsbetween length L and L+ 0.1 for the TSP w ith 10 cities. The *'sbelow the x-axis give the histogram fo r the number of times apath between L and L + 0.1 was found by the analog network in20 tries (conditions as in text). The region for L< 3.0 has beenmagnified by a factor o f 100 for clarity, b A histogram of thenumber of different paths between length L and L + 0.1 fo r theTSP problem with 30 cities. The arrow indicates the to ur lengthfor the Kernigh an-Lin so lution while the asterisk at 5.6 indicatesthe path length of a solution obtained by the analog network atfixed gain width. The asterisk at 5.0 indicates a better solutionobtained by slow ly increasing the gain

A c o n v e r g e n c e f r o m a g i v e n s t a r t in g s t a t e isd e t e r m i n i s t i c , b u t s t a r t i n g s t a t e s w h i c h a r e d i f f e r e n td u e t o a d i f f e r e n t c h o i c e o f 6x~ m a y l e a d t o d i f f e r e n tf i n a l s t a t e s . F i g u r e s 3 a a n d 3 b s h o w t w o t y p i c a l p a t h so b t a i n e d w i t h d i f f e r e n t 5x ~ . A l t h o u g h d i f f e r en t , b o t ha r e g o o d s o l u t i o n s t o t h e p r o b l e m . F i g u r e 3 a is a ls o t h e

b e s t p a t h , f o u n d b y e x h a u s t i v e s e a r c h o f al l p a t h s i n as e p a r a t e c a l c u l a t i o n .

T o s ee h o w w e l l a s e l e c ti o n w a s b e i n g m a d e o f g o o dp a t h s , w e c o m p a r e t h e p a t h s c h o s e n b y t h e n e t w o r kw i t h t h e l e n g t h s o f a l l p o s s ib l e p a t h s . T h e r e a r e 1 0 ! / 2 0= 1 8 1 ,4 4 0 t o t a l d i s t i n c t p a t h s , a n d a h i s t o g r a m o f t h e i rl e n g t h d i s t r i b u t i o n i s s h o w n i n F i g . 5 a . T h e p a t h sf o u n d i n 2 0 c o n v e r g e n c e s f r o m r a n d o m s t at e s a r e al s os h o w n a s t h e h i s t o g r a m ( * s y m b o l s ) b e l o w t h e x - a x is .( O f t h e s e 2 0 s ta r t i n g s t a t e s , 1 6 c o n v e r g e d t o l e g i t i m a t et o u r s. ) A b o u t 5 0 % o f t h e t ri a ls p r o d u c e d o n e o f t h e 2s h o r t e s t p a t h s. H e n c e t h e n e t w o r k d i d a n e x c e l l en t j o bo f s e l e c t in g a g o o d p a t h , p r e f e r r i n g p a t h s i n t h e b e s t1 0 - 5 o f al l p a t h s c o m p a r e d t o r a n d o m p a th s .

B e c a u s e a t y p i c a l b i o l o g i c a l n e u r o n m a y b e c o n -n e c t e d t o 1 0 0 0 - 1 0 , 0 0 0 o t h e r s , i t i s r e l e v a n t t o i n v e s t i -g a te h o w t h e c o m p u t a t i o n a l p o w e r o f th e n e t w o r kg r o w s w i t h t h e n u m b e r o f n e u r o n s . W e t h e r ef o r es t u d i e d a 9 0 0 n e u r o n s y s t e m d e s c r i b i n g a T S P o n3 0 c i ti e s. B e c a u s e t h e t i m e t o simulate t h e d i f f e r e n t ia le q u a t i o n s i n a d i g i ta l c o m p u t e r s c a l es s o m e w h a t w o r s et h a n n 3 , o u r r e s u lt s a r e f r a g m e n t a r y . W e a r e n o t y e tw e l l l o c a t e d i n p a r a m e t e r s p a c e a n d p a r a m e t e r c h o i c es e e m s t o b e a m o r e d e l i c a t e is su e w i t h 9 0 0 n e u r o n s t h a nw i t h 1 00 . T h e p a r t i c u l a r s e t o f 3 0 r a n d o m c i ti e s w eu s e d 3 a r e b e l ie v e d to h a v e t h e m i n i m u m p a t h l e n g t h o f4 . 2 6 f o r t h e p a t h s h o w n i n F i g . 4 b . T h e 3 0 - c i t y s y s t e mc o n v e r g e d t o p a t h s o f l e n g t h l es s t h a n 7 c o m m o n l y ,a n d l e ss t h a n 6 o c c a s i o n a l l y . F o r 3 0 c i ti e s, t h e r e a r e3 0 ! / 6 0 = 4 . 4 x 1 0 3o p a t h s . A d i r e c t e v a l u a t i o n o f t h el e n g t h o f 1 0 s r a n d o m p a t h s f o u n d a n a v e r a g e o f 1 2 .5 ,a n d n o n e s h o r t e r t h a n 9 .5 . A p a t h o f a b o u t a v e r a g ep a t h l e n g t h i s s h o w n i n F i g . 4 a . T h e p a t h l e n g t hh i s t o g r a m o f t h e r a n d o m s a m p l i n g i s s h o w n i n F ig . 5 b .F r o m a s t a t i s t i c a l e s t i m a t e a n d t h e k n o w n s h o r t e s tp a t h , t h e r e s h o u l d b e a b o u t 1 0 s p a t h s s h o r t e r t h a nl e n g t h 7 . T h u s , i n a s in g l e c o n v e r g e n c e , t h e n e t w o r kp r o v i d e d a v e r y g o o d s o l u t i o n t o t h e p r o b l e m , e x c l u d -i n g p o o r p a t h s b y a f a c t o r o f 1 0 - z 2 t o 1 0 - z 3 .

V T h e C o m p u t a t i o n a l P r o c e s sT h e c o l l e c ti v e c o m p u t a t i o n s w e h a v e d e s c r i b e d u s in gn o n l i n e a r a n a l o g c i r c u i ts h a v e a s p e c ts f r o m b o t hc o n v e n t i o n a l d ig i t al a n d a n a l o g c o m p u t e r s . I n c o n v e n -t i o n a l a n a l o g c o m p u t a t i o n , t h e d i f f er e n ti a l e q u a t i o nw h i c h i s s o l v e d b y t h e e l e c t r i c a l c ir c u i t is g e n e r a l l y t h es a m e e q u a t i o n t h a t t h e p r o g r a m m e r w i s h e s t o s o lv e i nt h e r e a l w o r l d ( T o m o v i c a n d K a r p l u s , 1 96 2). T h ev a r i a b l es i n t h e a n a l o g c o m p u t e r a r e c l o se l y r e l a t e d t o3 The list of 30 cities used in these experiments and the solutionshown in F ig. 4b computed using the Lin/Kernighan algorithm(Lin and Kernighan, 1973) were provided by D avid Johnson


9/12

149t h e r e a l - w o r l d v a r i a b l e s w h o s e b e h a v i o r i s s o u g h t . I nthe present case , however , the c i rcu i t d i f fe rent i a lequ a t ion to be so lv ed is of no in t r ins ic in te res t . Thi sd i f fe renti a l equa t ion i s e s sent ia l ly a program b y w h i c ha n a n s w e r t o a q u e s t i o n c a n b e fo u n d . D i g i t a l c o m p u -t a t i o n c o n v e n t i o n a l l y i n v o l v e s fi n d in g a d a t a r e p r e s e n -t a t i o n a n d a l g o r i t h m f o r t h e p r o b l e m , b y w h i c h t h ef ix e d h a r d w a r e w i ll e v e n t u a l l y c o n s t r u c t t h e d e s i r e da n s w e r . T h e c o l l e c ti v e m o d e w e h a v e d e s c r i b e d c o m -b i n e s t h e p r o g r a m m i n g a n d d a t a r e p r e s e n t a t i o n a s -p e c t s c h a r a c te r i s ti c o f d ig i t al c o m p u t a t i o n , b u t r e -p l a c e s th e u s u a l s t e r e o t y p e d l o g ic a l b e h a v i o r o f ad i g i ta l s y s t e m b y a s t e r e o t y p e d f o r m o f a n a l o gc o m p u t a t i o n .W h y i s th e c o m p u t a t i o n s o e ff e c ti v e ? T h e s o l u t i o nt o a T S P p r o b l e m i s a p a t h a n d t h e d e c o d i n g o f t h eT S P n e t w o r k ' s f in a l s ta b l e s t a te t o o b t a i n t h is d i s c r e ted e c i s i o n o r s o l u t i o n r e q u i r e s h a v i n g t h e f i n a l Vxj v a l u e sb e n e a r 0 o r 1 . H o w e v e r , th e a c t u a l a n a l o g c o m p u t a -t i o n o c c u r s i n th e c o n t i n u o u s d o m a i n , 0 < Vx j< 1. Th ed e c i s i o n - m a k i n g p r o c e s s o r c o m p u t a t i o n c o n s i s t s o ft h e s m o o t h m o t i o n f r o m a n i n i t ia l s t a t e o n t h e in t e r i o ro f t h e s p a c e ( w h e r e t h e n o t i o n o f a " t o u r " i s n o t e v e nd e f i n e d ) t o a n u l t i m a t e s t a b l e p o i n t n e a r e n o u g h t o ac o r n e r o f t h e c o n t i n u o u s d o m a i n t o b e a b l e t o id e n t i fyw i t h t h a t c o r n e r . I t is a s t h o u g h t h e l o g ic a l o p e r a t i o n so f a c a l c u l a t io n c o u l d b e g i v e n c o n t i n u o u s v a l u e sb e t w e e n " T r u e " a n d " F a l s e " , a n d e v o l v e t o w a r d c e r -t a i n t y o n l y n e a r t h e e n d o f t h e c a l c u l at i o n .

A l t h o u g h t h e r e m a y b e n o p r e c i s e " t o u r " i n t e r p r e -t a t i o n o f a s t a t e v e c t o r w h i c h d o e s n o t h a v e t h e f o r m[Eq. (7)] , a qua l i t a t ive in te rpre ta t ion can b e made .S u p p o s e r o w C h a s a n a p p r e c i a b l e v a l u e i n c o l u m n s 5a n d 6 a n d n o w h e r e e ls e, a n d n o o t h e r r o w A h a s m u c hg r e a t e r v a l u e in t h e s e s a m e c o l u m n s . T h e n i t m i g h t b es a i d t h a t c i t y C w a s b e i n g c o n s i d e r e d ( s i m u l t a n e o u s ly )f o r b o t h p o s i t i o n 5 a n d p o s i t i o n 6, t h a t o t h e r p o s s i-b i l i t i e s were not as l ike ly , and tha t l a t e r in t ime th i sp o s i t i o n a l a m b i g u i t y s h o u l d b e r e s o l v e d . F i g u r e 2 a i sa n i l l u s tr a t i o n o f a n i n t e r m e d i a t e t i m e s t a te i n a T S Pca lcula t ion on 10 c i ti e s us ing rand om noi se in i t i a lcond i t ions . A t th i s s t age of the ca lcu la t ion , i t appear st h a t A w a n t s t o b e i n p o s i t i o n 6 o r 7 i n t h e t o u r . C i t ie sB , C , and D w ant to be in pos i t ion s 9 , 10, or 1 , bu t i t i sno t a t a l l c l ea r which pa i r ing orB , C , D wi th 9 , 10 , 1 wi llbe pre sent in the f ina l s ta t e . S imi la r ly , pos i t io n 5 isg o i n g t o b e c a p t u r e d b y e i th e r c i t y F o r E , b u t a g a i n t h eo r d e r i s n o t c l ea r . A d e c i s io n i s a lr e a d y a p p a r e n t a s t or o u g h l y w h e r e o n t h e t o u r v a r i o u s c i t i e s w i l l b e , a n dt h is i s o f i ts e l f i m p o r t a n t i n f o r m a t i o n t o c o n v e y t o t h eothe r c i ti es : i t sugges t s res t r i c t ions on the po ss ib i l it i e sw h i c h t h e s e o t h e r s s h o u l d b e c o n s i d e r i n g . T h e r o u g hass ignments in th i s example a re p laus ib le , a s can bes e e n f r o m l o o k i n g a t t h e 1 0 c i t y m a p i n F i g . 3 a. I n d e e d ,t h e c o m p u t a t i o n w o r k s b e c a u s e t h e i n t e r m e d i a t e s t a t e s

s o i n t e r p r e t e d are r e a s o n a b l e . T h o u g h n o t p r e c i s e l ydef ined in t e rms of a tour , they repre sent the s imul ta -n e o u s c o n s i d e r a t i o n o f m a n y s i m i la r t o u r s . I n t e r p r e -t e d i n t h i s w a y , d u r i n g a c o n v e r g e n c e , t h e n e t w o r km o v e s f r o m s t a t e s c o r r e s p o n d i n g t o very r o u g h l ydef ined tours to s t a t es of h igher re f inemen t , un t i l as ingle tour i s l e f t . Thi s genera l computa t iona l s t ra t egyw i l l w o r k w e l l i n o p t i m i z a t i o n p r o b l e m s f o r w h i c hg o o d s o l u t i o n s c l u s te r , a n d e a c h e x c e l le n t s o l u t i o n h a sm a n y a l m o s t a s g o o d w h i c h a r e s i m i l a r t o i t .I n a d i r e c t t e s t o f t h e c o n t r i b u t i o n w h i c hi n t e r m e d i a t e - s t a t e a n a l o g p r o c e s s i n g m a k e s t o t h ea b i l it y o f t h e c o m p u t a t i o n a l n e t w o r k t o s o l v e t h e T S Pp r o b l e m , s e p a r a t e s i m u l a t i o n s o f a 1 0 - c i ty p r o b l e mw e r e p e r f o r m e d u s i n g a d e t e r m i n i s t i c n e t w o r k w h i c hminimized E us ing a dec i s ion space which cons i s t edonly o f t h e c o r n e r s o f t h e 2 N d i m e n s i o n a l c u b e . S u c h ap r o c e d u r e l e d t o t o u r s l i t t l e b e t t e r t h a n r a n d o m . A ne x a m p l e o f a s o l u t i o n f o r t h e 1 0 - c i ty p r o b l e m i s s h o w nin F ig . 3c . Thus the ana log charac te r i s t i c s and in te r -m e d i a t e s t a t e p r o c e s s i n g a r e i m p o r t a n t f o r g o o d T S Psolu t ions .

U n l i k e o u r a n a l o g n e t w o r k p r o c e d u r e , K i r k p a t r i c kh a s a p p r o a c h e d c o n s t r a i n t s a t i s f a c t i o n p r o b l e m s o n ad i s c r e t e d e c i s i o n s p a c e b y a M o n t e C a r l o a p p r o a c hu s i n g a n e f fe c ti v e t e m p e r a t u r e a n d a n a n n e a l i n g p r o c e -dure (K i rkpa t r i ck e t al ., 1983). Thi s " s imu la ted annea l -ing" method has s evera l impor tan t fea tures . F i r s t , i tc a u s e s m a n y c o n f i g u r a t i o n s t o b e a v e r a g e d n e a r ag i v e n o n e , w h i c h h a s t h e e f f ec t o f s m o o t h i n g t h e s u r f a c ea l o n g w h i c h a s e a r c h i s b e i n g d o n e . T h i s p r e v e n t s t h es y s t e m f r o m b e c o m i n g s t u c k in m i n o r e n e r g y m i n im a ,s ince these a re smoothed out . Second, i t g ives thep o s s i b i li t y o f c l im b i n g o u t o f a lo c a l m i n i m u m i n t oa n o t h e r o n e i f t h e a n n e a l i n g g o e s o n l o n g e n o u g h . ( A sthe pro blem s ize ge t s l a rger , t he t ru ly bes t so lu t ion to ap r o b l e m u s i n g s i m u l a t e d a n n e a l i n g i s g e n e r a l l y n o tf o u n d b e c a u s e t h e a n n e al in g p r o c e d u r e w o u l d t a k e a ni n fi n it e a m o u n t o f t im e .) T h e a n a l o g p r o c e d u r e u s e d b yt h e c o m p u t a t i o n a l n e t w o r k s a l s o s m o o t h s t h e e n e r g ys u r f ac e d u r in g t h e s e a r c h b u t d o e s n o t a l l o w r e c o v e r yf r o m l o c a l m i n i m a i n t h e s o l u t io n s p a c e . T h r o u g h t h e" s p i n r e p r e s e n t a t i o n " w e h a v e c o n s t r u c t e d i n e a r li e rw o r k ( H o p f i e l d , 1 9 8 2 ) , t h e r e i s a d i r e ct m e a n s o fshowing the smooth ing e f fec t . Cons ider the e f fec t ivef i e ld so lu t ion to the ex pec ta t io n va lu e of Vj for a s e t ofIs ing spins , e ach res t r ic te d to a val ue Vi = 0 or 1, a tt e m p e r a t u r e T w i t h a n e n e r g y E a s in E q . (8 ) a n dEq. (9) . Ef fec t ive f i e ld m ode l s ( see W annie r , 1966)replace a va r i ab le ( such as a pa r t i cu la r Vi) wh ich oc cursi n a n e n e r g y b y i t s e x p e c t a t i o n v a l u e ( V i ) w h e ne v a l u a t i n g t h e p r o b a b i l i t y d i s t r i b u t i o n o f a n y o t h e rv a r i a b l e . T h i s w e l l - k n o w n a p p r o x i m a t i o n a l l o w s t h es t a ti s ti c a l m e c h a n i c s o f c o m p l i c a t e d s y s t e m s t o b ea p p r o x i m a t e d b y a c l o s e d s et o f e q u a t i o n s r e l at i n g t h e


10/12

150e x p e c t a t i o n v a l u e s. F o r t h e T S P p r o b l e m , t h e s e e q u a -t ions a re


11/12

151d e s c r i b e s a t r a n s p o s i t i o n c o d e . T h e r e f o r e t h e a p p r o p r i -a t e e n e r g y f u n c t i o n n e e d s t h e s a m e A , B , C t e r m s a sEq. (8) and in addi t ionD Z ( P A s - P ~ j ) 2 VAiVBj+ E ~ . (P A --P i) : VA*" (15)A , B A , i

i , jThis energ y func t ion shou ld suff ice to f ind the "cor rec t "c o d e s , o r a t l e a s t o n e n e a r l y c o r r e c t . W h i l e i t w o u l d b en i c e a ls o t o u s e h i g h e r o r d e r c o r r e l a t i o n s o f le t t erf r e q u e n c y , t h e p r o b l e m o f i m p l e m e n t i n g t r ip l e s T ~ , i nh a r d w a r e i s s e v e r e . T h e t r a n s p o s i t i o n c o d e p r o b l e mc a n t h u s b e m a p p e d o n t o a n e t w o r k u s i n g a sl ig h t lym o d i f i e d T S P E f u n c t i o n . O t h e r p r o b l e m s c a n a l s o b em a p p e d o n t o t h e n e t w o r k , b u t u t i l i z e q u i t e d i f f e r e n ts p e c i f i c E f u n c t i o n s . F o r e x a m p l e , w e h a v e f o u n dc o n s t r u c t i o n s b y w h i c h t h e V e r t e x C o v e r i n g P r o b l e m( se e f o r e x a m p l e G a r e y a n d J o h n s o n , 1 9 79 ) a n d t h e b e s tm a t c h w i t h g a p s b e t w e e n t w o l i n e a r s e q u e n c e s p r o b -l em c a n b e m a p p e d o n t o a n a l o g c o m p u t a t i o n a lcircui ts .

VII Conclusion

B o t h a n a l o g n e t w o r k s o f b i o l o g ic a l n e u r o n s a n dn e t w o r k s o f m i c r o e l e e t r o n i c n e u r o n s c o u l d r a p i d l ys o l v e d i f f i c u l t o p t i m i z a t i o n p r o b l e m s u s i n g t h es t ra t e g i es w e h a v e p r e s e n t e d . " R a p i d " i s m e a s u r e d o nthe t ime sca le of the de vices used . S ince the con ver -g e n c e t im e o f t h e n e t w o r k w i ll b e a f e w c h a r a c t e r is t i ct i m e s o f t h e d e v i c e s fr o m w h i c h i t is b u i lt , o n e m a ye x p e c t c o n v e r g e n c e t im e s o f 1 0 - 1 0 0 m s f o r n e t w o r k s o fb i o l o g i c a l n e u r o n s , w h i l e s e m i c o n d u c t o r c i r c u i t ss h o u l d c o n v e r g e i n 1 0 - s t o 1 0 - 7 S . T h e t i m e s c a l ee x p e c t e d f o r b i o l o g i c a l s y s t e m s i s c o n s i st e n t w i t h t h ek n o w n c o m p u t a t i o n t i m e s i n p e r c e p t i o n a n d p a t t e r nr e c o g n i t i o n p r o b l e m s w h i c h o r g a n i s m s s o l v e q u i c k l y .

T h e p o w e r o f t h e c o m p u t a t i o n c a r ri e d o u t i sd e m o n s t r a t e d b y t h e s e l e c t i o n i t m a k e s b e t w e e n t h ep o s s i b l e a n s w e r s i t m i g h t g i v e . I n t h e T S P n e t w o r kcons i s t ing o f 100 neuro ns , the s e lec t iv i ty was 10 -4 to10 - 5 . Thi s i s the f rac t ion of al l pos s ib le so lu t ion s wh ichw e r e p u t f o r w a r d b y t h e n e t w o r k a s p u t a t i v e " b e s t "answers . S ince the re were only about 2 x 105 poss ib lep a t h s , o n e o f t h e b e s t f e w w a s a l w a y s s e l e ct e d.

T h e c o m p u t a t i o n a l p o w e r o f t h e T S P n e t w o r ks c a le s f a v o r a b l y w i t h t h e s iz e o f th e s y s te m . U n d e r t h em o s t f a v o r a b l e c i r c u m s t a n c e s , ( a n d o n l y t h e n ) t h ec o m p u t i n g p o w e r , a s m e a s u r e d b y t h e s e l e c t i v i t yd e f in e d a b o v e s h o u l d b e r a i s e d t o t h e p o w e r o f e w h e nthe s i ze of the sys tem i s mu l t ip li ed by e . W e m ight thenhave expec ted a s e lec t iv i ty of ( 1 0 - 4 " 5 ) + 9 = 10-39.5 fort h e 9 0 0 n e u r o n s y s te m . T h e a c t u a l s e l e c t iv i t y o f a b o u t1 0 - z 2 c o r r e s p o n d s t o ( 1 0 - 4 '5 ) + 5 o r t h u s c o r r e s p o n d sto the s ca l ing expec ted for the idea l case and 500

n e u r o n s . T h i s s h o u l d b e c o n t r a s t e d w i t h t h e c a se w h e nt h e c o m p u t a t i o n i s n o t t r u l y d o n e i n p a r a l l e l a n dcol l ec t ive ly , bu t i s ins t ead s imply par t i t ione d . In th i scase , the s e lec t iv i ty would change by a fac tor of 1 /c~ ,a n d h e n c e w o u l d o n l y b e a b o u t 1 0 - 5 . 5 f o r t h e 3 0 - c i t yp r o b l e m .T h e c o m b i n a t i o n o f s p e e d a n d p o w e r o f t h ec o m p u t a t i o n a l n e t w o r k s i s b a se d o n t h e a n a l o g c h a r -a c t e r o f t h e d e v i c e s i n v o l v ed . R e a l n e u r o n s h a v e t h ekind of respo nse charac te r i s t i c used here , and i t i s to bee x p e c t e d t h a t b i o l o g y w i ll m a k e u s e o f th a t f a ct ."S imula ted Annea l ing" (Ki rkpa t r i ck e t a l . , 1983) on ad i g i ta l c o m p u t e r o r in m o d e l s u s i n g t w o - s t a t e n e u r o n s(H into n a nd Se jnow ski , 1983) i s in t r ins ica lly s loww h e n m e a s u r e d i n u n i t s o f th e t i m e c o n s t a n t s o f d e v i c e sf r o m w h i c h t h e c o m p u t e r i s c o n s t r u c t e d b e c a u s e o f t h el o n g t i m e n e c e s s a r y t o c a l c u l a te c o n f i g u r a t i o n a l a v e r -a g e s a n d t o c l im b f r o m o n e v a l l e y i n t o a n o t h e r . T h i sa p p r o a c h i g n o r e s th e v e r y i m p o r t a n t u s e t h a t c a n b em a d e o f a n a l o g v a r i a b l e s t o r e p r e s e n t p r o b a b i l i t ie s ,e x p e c t a t i o n v a l u e s , o r t h e s u p e r p o s i t i o n o f m a n yp o s s ib i li ti e s. M a k i n g u s e o f th e a n a l o g v a r i a b l e s s e e m sa k e y t o t h e c o m b i n a t i o n o f h i g h s p e e d a n d c o m p u -t a t i o n a l p o w e r i n re a l n e t w o r k s . I t w a s n o t n e c e s s a r y t oplan such a use - the rea l phys ica l sys tems na tura l lyper form in th i s fash ion .

T h e i n p u t s i n t h e T S P p r o b l e m ( t h e d i s t a n c e sb e t w e e n c it ie s) o c c u r a s a m o d u l a t i o n o f t h e c o n n e c -t i o n s b e t w e e n n e u r o n s . T h i s f o r m o f i n p u t i s r a t h e rd i f fe r e n t i n c o n c e p t f r o m t h e u s u a l w a y o f v ie w i n g t h ei n p u t s a s a d d i t i v e l y d r i v i n g a p r o c e s s i n g n e t w o r k . I nr e a l n e u r o n s , s u c h a m o d u l a t i o n c o u l d b e d o n e , f o rexample , by a t t enua t ing d i s t a l s igna l s in the dendr i t i ca r b o r ( K o c h e t a l. , 1 98 3 ) b y m e a n s o f a p r o x i m a li n h i b i t o r y s h u n t i n g i n p u t . T h i s n e w m e c h a n i s m o fi n p u t i n g i n f o r m a t i o n i s b o t h b i o p h y s i c a l ly r e a s o n a b l eand computa t iona l ly e f fec t ive .

T h e e l e m e n ts o f th e c o m p u t a t i o n a l n e t w o r k s w eh a v e d e s c r i b e d w e r e g i v e n p r o p e r t i e s t h a t b i o l o g i c a ln e u r o n s a r e k n o w n t o p o s s e ss , p a r t i c u l a r l y t h e l a rg econn ec t iv i ty and ana lo g ch arac te r . I t i s d i f f icu l t t oi m a g i n e a s y s t e m w h i c h w o u l d m o r e e f fi c ie n t ly so l v es u c h c o m p l e x p r o b l e m s u s i n g a s m a l l n u m b e r o f" n e u r o n s " . B e c a u s e m a n y r e c o g n i t i o n t a s k s a n d p e r -c e p t i o n p r o b l e m s c a n b e s e t i n t h e f o r m o f a c o n -s t r a i n e d o p t i m u m w i t h c o m b i n a t o r i a l c o m p l e x i t y , t h ee f fe c ti v e n es s o f n e u r a l c o m p u t a t i o n i n t h e s e p r o b l e m sm a y r e l y o n c a s t i n g t h e o p t i m i z a t i o n p r o b l e m i n t o af o r m a t w h i c h c a n b e d o n e c o l l e c t iv e l y b y a n e t w o r k .A l t h o u g h w e h av e d e m o n s t r a t e d r e m a r k a b l e c o m -p u t a t i o n a l p o w e r in n e t w o r k s o f s i m p l e n e u r o n s , r e a ln e u r o n s a r e r a t h e r m o r e c o m p l e x . H o w e v e r , a d d i n ga d d i t i o n a l f e a t u r e s t o t h e n e u r o n s c o m p r i s i n g t h en e t w o r k s h o u l d increase h e c o m p l e x i ty o f a c o m p u t a -t i o n a l t a s k w h i c h t h e n e t w o r k c a n d o . N e v e r t h e le s s , it


12/12

152s h o u l d b e r e c o g n i z e d t h a t t h e w o r k i n g t o g e t h e r o f a ne n t ir e n e r v o u s s y s t e m i n v o l v e s a h o s t o f a d d i t i o n a lf e a t u r e s , i n c l u d i n g h i e r a r c h y , a n a t o m y , w i r i n g l i m i -t a t i o n s , n o n - r e c i p r o c a l c o n n e c t i o n s , a n d p r o p a g a t i o nd e l ay s . T h e p r e s e n t w o r k d e s c r ib e s o n l y t h e s i m u l a t i o no f a p a rt i a l, b u t p o w e r f u l , c o m p u t a t i o n w h i c h am o d u l e o f i n t e n se l y i n t e r c o n n e c t e d v e r y s im p l en e u r o n s m i g h t p e r f o r m .v i i i Acknowledgemen ts . The authors thank A. Gelper in and H .Berg for critical readings o f he manu script in draft . Th e wo rk atCal Tech was suppor ted in par t by Nat ion al Science Fou nda t iongrant PCM-8406049.

ReferencesAnderson, P.W. : Ch. 2 . In : Basic not ions of condensed mat terphysics . Menlo Park: Benjamin Cummings 1984Arbib, M.A. :Artificial intell igence a nd brain theo ry: unities anddiversit ies. Ann . Biomed. Eng. 3, 238 -274 (1975)Ballard, D.H.: Cortical connections and parallel processing:structure and function. Behav. Brain Sci. (1985, in press)Ballard, D.H. , Hinton, G.E. , Sejnowski, T.J. : Parallel visualcomp utat ion. Nature 306, 21-26 (1983)Feldman , J .A . , Bal lard , D .H. : Connec t ionis t models and thei rproperties. Cog. Sci. 6, 205-254 (1982)Ga rey, M.R. , John son, D.S.: Comp uters and intractability. NewY ork : W . H . F reem an 1979Gelperin, A.E. , Hopfield, JJ. , Tank, D.W.: The logic of Li m a xlearning. In : M odel neural netwo rks and behavior. Selver-s ton, A ., ed . . New Y ork: Plenum Press 1985Goldm an-Rakic , P .S.: Mo dular o rganizat ion of the prefronta lcortex. Trends Neurosci. 7, 419-4 24 (1984)Gros s, DJ. , M ezard, M .: Th e simplest spin glass. Nucl. Phys. FSB 240, 431-45 2 (1984)Hinton, G .E., Sejnowski, TJ . :Op t ima l perceptual inference. In :Proc. IEEE Comp. Soc. Conf . Comp. Vis ion & Pat ternRecog. , pp.448-453 , Wa shington, D .C . , June 1983Hopfield, J.J. : Neural networks and physical systems withemergent collective computational abili t ies. Proc. Natl .Acad. Sci . US A 79, 2554-2558 (1982)Hopfield, J.J. : Neurons with graded response have collectivecomp utat ional propert ies l ike those o f two-s ta te neurons .Proc. N at l . Acad. S ci . US A 81, 3088-3092 (1984)Huskey, H .D. , Korn, G .A. : Com puter H and boo k, pp. 1-4. NewY ork : M cG raw H i ll 1962

Ikeuchi , K . , Ho rn, B .K.P.: N umerical shape f rom shad ing andocc lud ing bounda ries. Artif. Intell. 17, 141-184 (1981 )Jackson, W.A. : Analog computat ion, pp. 5-8 . New York :M cG raw H i ll 1960Julesz , B .: In : Fo unda t ions ofcyclop ean vision. Ch icago: Univer-sity of Chic ago Press 1971Julesz, B. : Texto ns, the elements o f texture perception, and theirinteractions. Na ture 290, 91-97 (1981)Kirkpatrick, S. , Gelatt , C.D. , Vecchi, M.P. : Optimization bysimulated an nealing. S cience 220, 671-680 (1983)Ko ch, C. , Pog gio, T., Torte, V.: No nline ar interactions in adendr i t ic t ree : Local iza t ion, timing, and role in informat ionprocessing. Proc . Natl . Ac ad. Sci. U SA 80, 2799 (1983)Lawler, E.L. , Lenstra, A. , Rinno oy Ka n, H .G. , Eds. : The travelingsalesman problem. New Yo rk: John Wiley ( in press)Lin, S., Kern ighan, B.W .:An effective heuristic algorithm for thetraveling-salesman problem . Oper. Res. 21, 498-5 16 (1973)Marr , D . : V is ion. New York: W. H . Freeman 1982Mead, C . , Conw ay, L. : In t roduct ion to V LSI sys tems, pp. 12, 265.Reading, M A : Addison-Wesley 1980Poggio, T. , Ko ch, C . : An analog m odel o f com putat ion for the i ll -

pose d prob lems of early vision. M.I.T. A rtif . Intell . Lab. A.I.M em o N o . 783 , 1984Poggio, T. , Torte, V.: Il l-posed problems in early vision. Proc.Roy. B. (1985, in press)Sebestyn, G.S. : Decision-making processes in pattern recog-ni t ion. New York: McMil lan 1962Shepherd, G .M.: Microcircuits in the nervo us system. Sci. Am .238, 92-103 (1978)Terzo pou los, D.: Multilevel recon structio n of visual surfaces:variation al principles and finite element representations. In:Multi-resolution image processing and analysis. Rosenfeld,A. , ed. . Berlin, Heidelberg, N ew Y ork : Springer 1984Thouless, D.J. , Anderson, P.W., Palmer, R.G.: Solution of"solvable m ode l of a spin glass". Phil . Mag. 35, 593-601(1977)Tomo vic , R . , Karplus , W .J. : H igh speed a nalog co mputers , p . 4 .New York: Wiley 1962Wan nier, G.H .: Statistical physics, p. 330ff. New Y ork : Wiley1966

Received: February 16, 1985Prof. J. J. HopfieldDivis ion of Chemis t ry and BiologyCal ifornia Ins t i tu te of TechnologyPasadena, CA 91125U S A

Documents

Neural Computation of Decisions in Optimization Problems_3