PrefaceOneofthegreatjoysofscienceliesinthemomentofshareddiscovery.One person' s half -baked suggestion resonates inthemindofanotherand suddenlytakes ona definiteshape.Aninsightfulcritiqueofone way ofthinkingabouta problemleads toanother,betterunder-standing.Anincomprehensible simulationresultsuddenly makes senseas twopeople trytounderstand ittogether.Thisbookgrew outofmanysuch moments.Theseeds ofthebookwere sown inour jointworkon the interactiveactivationmodel ofwordperception.Since then,each ofus has workedwiththeotherand withothercollaborators.Theresultsofthese collaborationsare reportedinseveralofthechaptersofthisbook.Thebookalsocontainsmanychaptersbyothercolleagueswhoseexplorationshavebecomeintertwinedwithours.Each chapter has itsownby-line,buteach alsoreflectstheinfluencesofothermembersofthegroup.Wehopetheresultreflects some ofthebenefits ofparallel distributedprocessing!Theideaofparalleldistributedprocessing- thenotionthatintelli -gence emerges fromtheinteractionsoflargenumbersofsimplepro-cessing units- hascomeandgonebefore.Theideabegantoseemmoreand moreattractivetous as thecontrast between ourconvictionsabout basic characteristics ofhumanperception, memory, language, andthoughtandtheaccepted formaltoolsforcapturingmentalprocessesbecame moreapparent.Symbol-processing machines,foralltheirTur -ingequivalence,hadfailedtoprovideusefulframeworksforcapturingthesimpleinsightsabouttheinteractivenatureofprocessing thathadlead tosuch models as theHEARSAYmodelofspeech understanding.Moregenerally, theyhad failedtoprovide a frameworkforrepresentingknowledge ina way thatallowedittobe accessedby contentand effec-tivelycombinedwithotherknowledgetoproduceusefulautomaticsyntheses thatwouldallowintelligencetobeproductive.Andtheymadenocontactwiththerealstrengthsandweaknesses ofthehardware inthebrain.ACraycomputercan performontheorderof100 milliondouble-precision multiplicationsina second,butitdoes notexhibitnaturalintelligence.Howthenare wetounderstandthecapa-bilitiesofhumanthought ,giventhetimeconstantsandnoisinessinherentinneural systems?Itseemed obviousthattoget any process-ingdone inreal time,the slow,noisy hardware inthebrainwouldhavetodo massively parallel processing.Asourinterestinparallel mechanisms developed,we began tostudytheworkofotherswhoshared ourconvictionsandtobuildontheirwork.ParticularlyimportantinthisregardwasHintonandJ.A.Anderson' s( 1981)ParallelModels ofAssociative Memory.Indeed,wesee ourbookas a descendant oftheirbookontwoaccounts. First ,thematerialpresented hererepresents furtherdevelopmentsontheworkpresented inHintonand Anderson' s book.Second, we owe a particularintellectualdebttobothHintonand Anderson.Ourinterestindistrib-uted,associative memoriesgoes back tointeractionswithJimAnder -son,beginningas earlyas 1968.Ourinterestinthese topicsbegan inearnest,however,duringtheperiodwhenweweredevelopingtheinteractiveactivationmodelofwordperception,in1979,shortlyafterGeoffreyHintonbegan a postdoctoral fellowshipatUCSD.Geoffrey ' scrispexplanationsshowedusthepotentialpowerandgeneralityofmodelscreated fromconnectionsamongsimpleprocessing units,andfittogethernicelywithourowndevelopingconvictionthatvariousaspects ofperception, language processing, and motorcontrolwere bestthoughtofintermsofmassivelyparallelprocessing (see McClelland,1979, and Rumelhart ,1977, forourearliest steps inthisdirection) .Theprojectculminatinginthisbookformallybegan inDecember,1981 when thetwoofus and GeoffreyHintondecided toworktogetherexploringtheimplicationsofnetworkmodels andtowritea bookout -liningourconclusions.Weexpectedtheprojecttotakeaboutsixmonths.Webegan inJanuary1982 bybringinga numberofourcol-leagues togethertoformadiscussiongrouponthesetopics.Duringthefirstsixmonthswemettwiceweeklyandlaidthefoundationformostoftheworkpresented inthese volumes.Ourfirstorderofbusi-ness was todevelop a name fortheclass ofmodels we were investigat-ing.Itseemed tous thatthephrase parallel distributedprocessing(POPx PREFACEforshort )bestcapturedwhatwehadinmind .Itemphasizedtheparal -lelnatureoftheprocessing ,theuseofdistributedrepresentationsanddistributedcontrol ,andthefactthattheseweregeneralprocessingsys-tems ,notmerelymemorieswewerestudying ,asthephraseassociativememorysuggests.ThusthePOPresearchgroupwasborn .HintonandMcClellandleftafterthefirstsixmonths - HintontoCMUandMcClellandtoMITandlatertoCMU .ThePOPresearchgroup ,how -ever ,hascontinuedregularmeetingsatUCSOuptothepresenttime .Thegrouphasvariedfromfiveorsixofusattimestoasmanyas15ormoreatothertimes ,andthereisnowaparallelgroupofabout15 orsopsychologistsandcomputerscientistsatCMU .ShortlyafterleavingUCSOin1982,HintonbeganworkingwithTerrenceSejnowskiontheBoltzmannmachine(Chapter7)anddecidedtodl"' JP fromtheroleoforganizeroftheprojecttoa contributor ,sohecouldspendmoretimeworkingontheimplicationsoftheBoltzmannmachine .Thus ,theprimaryresponsibilityforputtingthebooktogetherfelltothetwoofus.Atfirstweexpectedtocompletethebookwithinayearafterwebeganourwork .Soon ,however ,itbecameclearthattherewasmuchworktobedoneandmanydirectionstoexplore .Thus ,ourworkcontinuedandexpandedasweandourcol -leaguesfollowedtheimplicationsofthePOPapproachinmanydif -ferentways .Agooddealhashappenedsincewebeganthisproject . Thoughmuchoftheinitialgroundworkwaslaidinearly1982,mostofthematerialdescribedinthesevolumesdidnottakeitspresentformuntilmuchlater .Theworkhasbeeninterdisciplinaryandrepresentswhatweconsideratruecognitivescienceapproach .Althoughthetwoofushavebeentrainedascognitivepsychologists ,thePOPgroupasawholeincludespeoplefromawiderangeofbackgrounds .Itincludespeopletrainedinphysics ,mathematics ,neuroscience ,molecularbiology ,andcomputersciences ,as wellas inpsychology .Wealsoenvisionaninterdisciplinaryaudienceforourbook .Wearecognitivepsychologistsandwehope ,primarily ,topresentPOPmodelstothecommunityofcognitivepsychologistsasalternativestothemodelsthathavedominatedcogni -tivepsychologyforthepastdecadeorso.Wealso,however ,seeour -selvesasstudyingarchitecturesforcomputationandmethodsforartifi -cialintelligence .Therefore ,wehopethatthisbookwillbeseenasrelevanttoresearchersincomputerscienceandartificialintelligence .Also ,thePOPapproachprovidesasetoftoolsfordevelopingmodelsoftheneurophysiologicalbasisofhumaninformationprocessing ,andsowehopeportionsofthesebookswillseemrelevanttoneuroscien -tistsaswell ..XI PREFACE..XII PREFACEORGANIZATION OF THE BOOKOurbookconsistsofsixparts ,threeineachofthetwovolumes .Theoverallstructureisindicatedintheaccompanyingtable .PartIpro-videsanoverview .Chapter1presentsthemotivationfortheapproachanddescribesmuchoftheearlyworkthatleadtothedevelopmentsreportedinlatersections .Chapter2describesthePOPframeworkinmoreformalterms .Chapter3focusesontheideaofdistributedrepresentation ,andChapter4providesadetaileddiscussionofseveralgeneralissuesthatthePOPapproachhasraisedandexplainshowtheseissuesareaddressedinthevariouslaterchaptersofthebook .Theremainingpartsofthebookpresentdifferentfacetsofourexplorationsinparalleldistributedprocessing .ThechaptersinPartIIaddresscentraltheoreticalproblemsinthedevelopmentofmodelsofparalleldistributedprocessing ,focusingforthemostpartonfundamen -talproblemsinlearning .ThechaptersinPartIIIdescribevariousmathematicalandcomputationaltoolsthathavebeenimportantinthedevelopmentandanalysisofPOPmodels .PartIVconsidersACONDENSEDTABLEOFCONTENTSVOLUMEII .THEPOPPERSPECTIVEII .BASICMECHANISMSIII .FORMALANALYSES1. TheAppealofPOP5.CompetitiveLearning9.LinearAlgebra2.AFrameworkforPDP6.HarmonyTheory10.ActivationFunctions3 .Distributed7 .BoltzmannMachines11 .TheDeltaRuleRepresentations8.Learningby12.ResourceRequirements4. GeneralIssuesErrorPropagation13. Parallel NetworkSimulatorVOLUMEII--IV .PSYCHOLOGICALV .BIOLOGICALVI .CONCLUSIONPROCESSESMECHANISMS14 .SchemataandPOP20 .Anatomyand26 .Reflections15. SpeechPerceptionPhysiologyFutureDirections16. ModelofReading21. Computationin17.LearningandMemorytheBrain18.MorphologyAcquisition22.Neuraland19. Sentence ProcessingConceptual Levels23. Place Recognition24.NeuralPlasticity25 .AmnesiaapplicationsandimplicationsofPDPmodelstovariousaspectsofhumancognition ,includingperception ,memory ,language ,andhigher -levelthoughtprocesses.PartVconsiderstherelationbetweenparalleldistributedprocessingmodelsandthebrain ,reviewsrelevantaspectsoftheanatomyandphysiology ,anddescribesseveralmodelsthatapplyPOPmodelstoaspectsoftheneurophysiologyandneuropsychologyofinformationprocessing ,learning ,andmemory .PartVIcontainstwoshortpieces:areflectiononPDPmodelsbyDonNormanandabriefdiscussionofourthoughtsaboutpromisingfuturedirections .Howtoread thisbook?Iti~ toolongtoread straightthrough .Norisitdesignedtobereadthisway.Chapter1isagoodentrypointforreadersunfamiliarwiththePOPapproach ,butbeyondthatthevariouspartsofthebookmaybeapproachedinvariousorders ,asonemightexplorethedifferentpartsofacomplexobjectormachine .Thevari -ousfacetsofthePOPapproachareinterrelated ,andeachpartinformstheothers ~ buttherearefewstrictsequentialdependencies .Thoughwehavetriedtocross-referenceideasthatcomeupinseveralplaces,wehopethatmostchapterscanbeunderstoodwithoutreferencetotherestofthebook .Wheredependenciesexisttheyarenotedintheintro -ductorysectionsatthebeginningofeachpartofthebook .Thisbookchartstheexplorationsweandourcolleagueshavemadeinthemicrostructureofcognition .Thereisalotofterrainlefttobeexplored .Wehopethisbookservesas a guidethathelpsothersjoinusintheseongoingexplorations ....XIII PREFACEDecember1985 JamesL. McClellandPI1TSBURGH , PENNSYLVANIADavid E. RumelhartLA JOLLA, CALIFORNIAAswe have already said,nearlyalltheideas inthisbookwere bornoutofinteractions,and one ofourmostimportantacknowledgments istothe environmentthatmade these interactionspossible.TheInstituteforCognitiveScience atUCSDand themembers oftheInstitutehavemade up the core ofthisenvironment .DonNorman,ourcolleague and friend , theFounderand DirectoroftheInstitute ,deserves special creditformakingICSanexcitingandstimulatingplace,forencouragingourexplorationsinparalleldistrib-utedprocessing, and forhis central roleinarranging muchofthe finan-cialsupport thisbookhas benefitedfrom(ofwhichmorebelow) .Theatmosphere depends as wellonthefaculty,visitingscholars, and gradu-atestudentsinandaroundICS.ThemembersofthePDPResearchGroupitself ,ofcourse,have played themost centralroleinhelpingtoshape theideas foundinthisbook.Allthose whocontributedtotheactual contentsofthebookare listedonthecoverpage;theyhave allcontributed ,as well ,inmanyotherways.Several otherparticipants inthegroupwhodonothaveactualcontributionstothebookalsodeserve mention.MostprominentamongtheseareMikeMozerandYves Chauvin,two graduate students inthe CognitiveScience Lab, andGaryCottrell ,arecentadditiontothegroupfromtheUniversityofRochester.Several othermembers oftheintellectualcommunityinand aroundICShaveplayedveryimportantrolesinhelpingustoshapeourthoughts.These includeLizBates,MichaelCole,Steve Draper,DonAcknowledgmentsACKNOWLEDGMENTSGentner ,EdHutchins ,JimHollan ,JeanMandler ,GeorgeMandler ,JeffMiller ,GuyvanOrden ,andmanyothers ,includingtheparticipantsinCognitiveScience200.Therearealsoseveralcolleaguesatotheruniversitieswhohavehelpedusinourexplorations .Indeed ,theannualconnectionistworkshops(thefirstofwhichresultedintheHintonandAndersonbook )havebeenimportantopportunitiestoshareourideasandgetfeedbackonthemfromothersinthefield ,andtolearnfromthecon -tributionsofothers .JimAnderson ,DanaBallard ,JerryFeldman ,GeoffHintonandTerrySejnowskiallhadahandinorganizingdif -ferentonesofthesemeetings ;andwehavelearnedagreatdealfromdiscussionswiththemandotherparticipants ,particularlyAndyBarto ,ScottFahlman ,ChristofvonderMalsburg ,JohnHopfield ,DaveTouretzky ,andmorerecentlyMarkFantyandGeneCharniak .McClelland ' sdiscussionsatMIT(particularlywithJerryFodorandMollyPotter )helpedintheclarificationofseveralaspectsofourthink -ing ,andvariouscolleaguesatandaroundCMU - particularlyJohnAnderson ,MarkDerthick ,DaveKlahr ,BrianMacWhinney ,andJeffSokolov - havecontributedagreatdealthroughdiscussionsoverthelastyearandahalforso,aswehaveworkedtowardthecompletionofthebook .OthersoneorbothofushaveinteractedwithagreatdealincludeBillBrewer ,NealCohen ,AlCollins ,BillySalter ,EdSmith ,andWalterSchneider .Allofthesepeoplehavecontributedmoreorlessdirectlytothedevelopmentoftheideaspresentedinthisbook .Anoverlappinggroupofcolleaguesdeservescreditforhelpingusimprovethebookitself .JimAnderson ,AndyBarto ,LarryBarsalou ,ChrisReisbeck ,WalterSchneider ,andMarkSeidenbergallreadseveralchaptersofthebookandsentusefulcommentsandsuggestions .Manyotherpeoplereadandcommentedonindividualchapters ,andwearesincerely gratefulfortheircarefulcontributions ,whichwe acknowledgeintheappropriatechapters .ThisprojectowesatremendousamounttothehelpoftheexcellentstaffoftheInstituteforCognitiveScience .KathyFarrelly ,inparticu -lar ,hasplayedanenormousroleinallaspectsoftheproductionofthebook;hercheerful ,thoughtful ,andverycarefulassistance madetheproductionofthebookrunmuchmoresmoothlythanwehavehadanyrighttohopeandallowedustokeepworkingonthecontentofsomeofthechapters evenas thefinalproductionwas rollingforwardonothersections.EileenConway' sassistance withgraphics andformattinghasalsobeeninvaluableandweareverygratefultoheras well .MarkWal -lenkeptthecomputersrunning ,servedaschiefprogrammingconsul -tantanddebuggerparexcellence ,andtamedtroff ,thephototypesetter .Withouthimwewouldneverhavegottenalltheformattingtocomeoutright .KarolLightnerworkedveryhardtowardtheendofthe.XVI. .ACKNOWLEDGMENTSXVIIJLM/ DERprojectonfinalproofingandindexing ,andSondraBuffett ,astheAdministrativeDirectorofICS,heldeverythingtogetherandkepteverythingrunningthroughouttheentirefouryearsoftheproject .Ourprojecthasbeensupportedbyanumberofdifferentagenciesandfoundations .PrimarysupportcamefromtheSystemDevelopmentFoundationandtheOfficeofNavalResearch .TheSystemDevelop -mentFoundationhasprovideddirectsupportforthePDPresearchgroupthroughagranttoNormanandRumelhart ,andhasalsosup-portedseveraloftheindividualmembersofthegroup(Crick ,Hinton ,Sejnowski ,andZipser ) .ONRcontractsthathavecontributedsupportincludeNOOO14- 79 - C-O323 ,NR667 - 437 ~ NOOO14 - 85 - K -0450 ,NR667 -548 ~ andNOOO14 - 82 -C- O374 ,NR667 -483 .ThepeoplebehindbothSDFandONRdeserveacknowledgmenttoo .TheentirePDPenterpriseowesaparticulardebtofgratitudetoCharlieSmith ,formerlyofSDF ,whoappreciatedtheappealofparalleldistrib -utedprocessingveryearlyon ,understoodourneedforcomputingresources ,andhelpedprovidetheentirePOPresearchgroupwiththefundsandencouragementneededtocompletesuchproject .HenryHalff ,formerlyofONR ,wasalsoanearlysourceofsupport ,encouragement ,anddirection .CharlieSmithhasbeensucceededbyCarlYork ,andHenryHalffhasbeensucceededbySusanChipman ,MichaelSi:tafto ,andHaroldHawkins .Wearegratefultoallofthesepeoplefortheircommitmenttothecompletionofthisbookandtotheongoingdevelopmentoftheideas.Severalothersourceshavecontributedtothesupportofindividualmembersofthegroup .TheseincludetheNationalInstituteofMentalHealth ,throughaCareerDevelopmentAward - PHS-MH -00385 - toMcClellandandpost -doctoralfellowshipstoPaulSmolenskyandPaulMunrounderGrantPHS - MH - 14268totheCenterforHumanInforma -tionProcessingatUCSD .SmolenskyreceivedsupportintheformofafellowshipfromtheAlfredP.SloanFoundation ,andsomeofMcClelland ' sworkwassupportedbyagrantfromtheNationalScienceFoundation(BNS- 79-24062) .Theseandothersourcesofsupportforspecificindividualsorprojectsareacknowledgedintheappropriatechapters .Finally ,wewouldliketothankourwives ,HeidiandMarilyn .Theirunderstanding ,encouragement ,andsupportthroughoutthefouryearsofthisprojecthelpedtomaketheprocessofbringingthisbooktolife Imuchmorerewardingthanitmighthavebeen .Addresses of thePDPResearch GroupChisatoAsanumaFrancisH. C. CrickDepartmentof LinguisticsUniversityof California, SanDiegoLa Jolla, CA 92093Jeffrey L. ElmanGeoffreyE. HintonMichael I. JordanAlan H. KawamotoSalkInstituteP.O. Box 85800SanDiego, CA 92138SalkInstituteP.o. Box 85800SanDiego, CA 92138DepartmentofComputerScienceCarnegie - MellonUniversityPittsburgh ,PA15213DepartmentofComputerandInformationScienceUniversityofMassachusettsAmherst ,MA01003DepartmentofPsychologyCarnegie - MellonUniversityPittsburgh ,PA15213RESE) xx ADDRESSESOFTHEPDP LRCHGROUPJamesL. McClellandDepartmentof PsychologyCarnegie-Mellon UniversityPittsburgh, PA 15213Paul W. MunroDonaldA .NormanDanielE. RabinIntellicorp1975El Camino Real WestMountain View, CA 94040David E. RumelhartTerrenceJ. Sejnowski Departmentof BiophysicsJohns Hopkins UniversityBaltimore, MD21218Paul SmolenskyGregory O. StoneRonaldJ. WilliamsDavid Zipser Insitutefor CognitiveScienceUniversityof California, SanDiegoLa Jolla, CA 92093Universityof California , SanDiegoLaJolla, CA 92093Department of Information ScienceUniversity of PittsburghPittsburgh, PA 15260Institute for Cognitive ScienceInstitute for Cognitive ScienceUniversity of California, San DiegoLa Jolla, CA 92093DepartmentofComputerScienceUniversityofColoradoBoulder ,CO80309CenterforAdaptiveSystemsDepartmentofMathematicsBostonUniversityBoston ,MA 02215InstituteforCognitiveScienceUniverSi ~ yofCalifornia ,SanDiegoLaJolla ,CA 92093Whatmakes people smarterthanmachines?Theycertainlyare notquickerormoreprecise.Yetpeople are farbetteratperceiving objectsinnaturalscenes andnotingtheirrelations,atunderstandinglanguageandretrievingcontextuallyappropriateinformationfrommemory,atmakingplans and carryingoutcontextuallyappropriate actions, and at awide range ofothernatural cognitivetasks.People are also farbetter atlearningtodothese thingsmoreaccurately andfluentlythroughpro-..cesslng experIence.Whatisthebasis forthesedifferences?Oneanswer,perhaps theclassic onewe mightexpect fromartificialintelligence,is "software."Ifwe onlyhad therightcomputerprogram,theargument goes, we mightbeabletocapturethefluidityandadaptabilityofhumaninformation.processlng.Certainlythisanswerispartiallycorrect.Therehavebeengreatbreakthroughsinourunderstandingofcognitionasaresultofthedevelopmentofexpressive high-levelcomputerlanguages and powerfulalgorithms.Nodoubttherewillbemoresuchbreakthroughsinthefuture .However, we do notthinkthatsoftware is the whole story.Inourview,people are smarterthantoday' s computersbecause thebrainemploys a basic computationalarchitecturethatis moresuitedtodealwithacentralaspect ofthenaturalinformationprocessing tasksthatpeople are so good at.Inthischapter, we willshow throughexam-ples thatthese tasks generally requirethe simultaneous consideration ofmanypiecesofinformationorconstraints.Eachconstraintmaybeimperfectlyspecifiedandambiguous,yeteachcanplayapotentiallyCHAPTER1TheAppealofParallelDistributedProcessingJ. L. McCLELLAND , D. E. RUMELHART , andG. E. HINTONPERSPECTIVEdecisive roleindeterminingtheoutcomeofprocessing.Afterexamin-ingthesepoints,wewillintroduceacomputationalframeworkformodelingcognitiveprocessesthatseems wellsuitedtoexploitingtheseconstaints and thatseems closer thanotherframeworkstothestyleofcomputationas itmightbe donebythebrain.Wewillreviewseveralearlyexamplesofmodelsdevelopedinthisframework ,andwewillshow thatthemechanisms these models employ can giverise topower-fulemergentpropertiesthatbegintosuggest attractivealternativestotraditionalaccounts ofvariousaspects ofcognition .Wewillalso showthatmodels ofthisclass providea basis forunderstanding how learningcan occur spontaneously, as a by-product ofprocessing activity .4 THEPDPMultipleSimultaneous ConstraintsOnthedesk nexttomyterminalare several objects- a chippedcoffeemug,theendofa computercable,a knobfroma clockradio.Idecide topicktheknobup.AtfirstIhesitate, becauseitdoesn' tseempossible.ThenIjustreachforit ,andfindmyselfgrasping theknobinwhat wouldnormallybe consideredaveryawkwardposition- butitsolvesalloftheconstraints.I ' mnotsure what allthe details ofthemovementwere, so Iletmyselftryita fewtimesmore.Iobserve thatmyrighthand iscarriedupoffthekeyboard,bentattheelbow,untilmyforearmisatabouta300 angle tothedesk topand parallel totheside oftheterminal .Thepalmis facing downward throughmostofthis.Then,myarmextends and lowers downmoreorless paralleltotheedge ofthedesk andparalleltotheside oftheterminaland,asitdrops,itturnsabout900 sothattheReachingandgrasping.Hundredsoftimeseach day wereach forthings.Wenearlyneverthinkaboutthese acts ofreaching.Andyet,each time ,a large numberofdifferentconsiderations appear to jointlydetermineexactly how we willreach fortheobject.Thepositionoftheobject,our . posture atthetime,whatelse we may also be holding,thesize,shape, and anticipated weightoftheobject, any obstacles thatmaybeintheway- allofthese factors jointlydeterminetheexact methodwe willuse forreaching and grasping.ConsiderthesituationshowninFi,gure1.Figure1AshowsJayMcClelland' s hand,intypingpositionathisterminal .Figure18indi -cates thepositionhishand assumed inreaching fora smallknobon thedeskbeside theterminal .Wewilllethimdescribe whathappened inthefirstperson:1.TIlEAPPEAL OF PDP5AFIGURE1.A: Aneveryday situationin which it is necessary totakeinto account a largenumberof constraintstograsp a desiredobject.In thiscase thetargetobjectis thesmallknobtotheleftofthecup.B:Theposturethearmarrivesatinmeetingtheseconstraints.-I sawthe sheepgrazingin the field.Ourknowledge ofsyntactic rulesalone does nottellus what grammati-cal roleis played by theprepositional phrases inthese two cases.Inthefirst , IIflyingtoNewYork"istakenas describingthecontextinwhichthespeaker saw theGrandCanyon - whilehe was flyingtoNewYork .Inthesecond,"grazinginthefield IIcouldsyntacticallydescribeananalogous situation , inwhichthe speaker is grazing inthe field , butthispossibilitydoes nottypicallybecome available onfirstreading.Insteadweassign IIgrazinginthefield IIas amodifierofthesheep(roughly ,'I whoweregrazinginthefield II) .Thesyntacticstructureofeach of6THEPOPPERSPECTIVEThoughthedetailsofwhathappened heremightbequibbledwith ,thebroad outlinesare apparent.Theshape oftheknoband itspositiononthetable~ thestartingpositionofthehandonthekeyboard;thepositionsoftheterminal ,thecup,andtheknob~ andtheconstraintsimposedbythestructureofthearmand themusculatureused tocon-trolit - allthesethingsconspiredtoleadtoasolutionwhichexactlysuitstheproblem.Ifanyofthese constraintshadnotbeen included,themovementwouldhave failed.Thehand wouldhave hitthecup orthe terminal - oritwouldhave missed theknob.palmisfacingthecupandthethumbandindexfingerarebelow.Theturningmotionoccurs justintime,asmyhanddrops,toavoidhittingthecoffeecup.Myindexfingerandthumbcloseinontheknobandgrasp it ,withmyhandcom-pletelyupside down.The mutualinfluenceofsyntaxandsemantics.Multipleconstraintsoperate justas stronglyinlanguage processing as theydoinreachingand grasping.Rumelhart( 1977)has documented many ofthese multi -pleconstraints.Ratherthancatalogthemhere,wewilluseafewexamples fromlanguage toillustratethefactthattheconstraintstendtobereciprocal:Theexampleshowsthattheydonotrunonlyfromsyntax tosemantics- they also runthe otherway.Itis clear,ofcourse,thatsyntax constrains theassignment ofmean-ing.WithoutthesyntacticrulesofEnglishtoguideus,wecannotcorrectlyunderstand whohas done whattowhominthefollowingsen-tence:Theboy theman chased kissed the girl .Butconsider these examples (Rumelhart ,1977;Schank,1973) :Isaw the grand canyon flyingtoNew York .these sentences ,then,is determined inpart by the semanticrelationsthattheconstituents ofthesentence mightplausibly bear tooneanother.Thus, the influencesappeartorun both ways, from the syn-tax to the semanticsand from the semanticsto the syntax.Inthese examples,wesee howsyntactic considerationsinfluencesemantic ones and how semantic ones influence syntactic ones.Wecannotsay that one kind of constraintis primary.Mutual constraintsoperate, not only betweensyntacticand semanticprocessing ,butalso withineach ofthese domains as well.Here weconsideran examplefrom syntacticprocessing , namely, the assignmentof wordsto syntacticcategories .Considerthe sentences :like the joke.like the drive.like to joke.like to drive.Inthiscase itlooksas thoughthewordsthe and to serve todeterminewhetherthefollowingwordwillbe read as a nounora verb.This,ofcourse,isaverystrongconstraintinEnglishand can serve toforceaverbinterpretationofa wordthatis notordinarilyused thisway:I like to mud.On the other hand,ifthe information specifyingwhether the functionword precedingthe final word is to or the is ambiguous, then the typicalreadingofthe word that follows itwilldetermine which way the func-tionword isheard.Thiswas shown inanexperiment byIsenberg,Walker, Ryder, and Schweikert(1980) .They presentedsoundshalfwaybetweento (actuallyItAI )and the (actuallyIdAI )and found that wordslike joke, which we tend to thinkoffirstas nouns, made subjectshearthe marginal stimulias the,while words likedrive,which we tend tothinkoffirstas verbs,made subjectshear the marginal stimulias to.Generally, then, itwould appearthat each word can help constrainthesyntacticrole, and even the identity, of every other word.Simultaneousmutualconstraintsinwordrecognition.Justas thesyntactic roleofone wordcan influencetheroleassigned toanotherinanalyzing sentences, so theidentityofone lettercan influencetheiden-tityassigned toanotherinreading.Afamousexampleofthis,fromSelfridge,is showninFigure2.Alongwiththisis a second example inwhichnoneoftheletters,consideredseparately,canbeidentifiedunambiguously,butinwhichthepossibilitiesthatthevisual71.THEAPPEALOFPDP~~~~I8THEPDPPERSPECTIVEFIGURE 2.Some ambiguousdisplays.Thefirstone isfromSelfridge,1955.Thesecondline showsthat three ambiguouscharacterscan eachconstrainthe identity of theothers.The third, fourth, and fifthlines show that thesecharactersare indeedambigu-ous inthat they assumeother identities inother contexts.(The ink-blot techniqueofmakinglettersambiguousis due to Lindsayand Norman, 1972).informationleaves open foreach so constrainthepossible identitiesofthe others thatwe are capable ofidentifyingall ofthem.Atfirstglance,thesituationheremustseem paradoxical:Theiden-tityofeach letteris constrained bytheidentitiesofeach oftheothers.Butsince ingeneral we cannot knowtheidentitiesofany oftheletters9 1.THEAPPEALOFPOPuntilwehaveestablishedtheidentitiesoftheothers ,howcanwegettheprocessstarted ?Theresolutionoftheparadox ,ofcourse ,issimple .Oneofthedif -ferentpossiblelettersineachpositionfitstogetherwiththeothers .Itappearsthenthatourperceptualsystemiscapableofexploringallthesepossibilitieswithoutcommittingitselftooneuntilalloftheconstraintsaretakenintoaccount .Understandingthroughtheinterplayofmultiplesourcesofknowledge.Itis clear thatweknowa good deal abouta large numberofdifferentstandardsituations .Severaltheoristshavesuggestedthatwestorethisknowledgeintermsofstructurescalledvariously :scripts(Schank ,1976) ,frames(Minsky ,1975) ,orschemata(Norman&Bobrow ,1976;Rumelhart ,1975) .Suchknowledgestructuresareassumedtobethebasisofcomprehension .Agreatdealofprogresshasbeenmadewithinthecontextofthisview .However ,itisimportanttobearinmindthatmosteverydaysitua -tionscannotberigidlyassignedtojustasinglescript .Theygenerallyinvolveaninterplaybetweenanumberofdifferentsourcesofinforma -tion .Consider ,forexample ,achild ' sbirthdaypartyatarestaurant .Weknowthingsaboutbirthdayparties ,andweknowthingsaboutres-taurants ,butwewouldnotwanttoassumethatwehaveexplicitknowledge(atleast ,notinadvanceofourfirstrestaurantbirthdayparty )abouttheconjunctionofthetwo .Yetwecanimaginewhatsuchapartymightbelike .Thefactthatthepartywasbeingheldinares-taurantwouldmodifycertainaspectsofourexpectationsforbirthdayparties(wewouldnotexpectagameofPin -the - Tail -on-the -Donkey ,forexample ) ,whilethefactthattheeventwasabirthdaypartywouldinformourexpectationsforwhatwouldbeorderedandwhowouldpaythebill .Representationslikescripts ,frames ,andschemataareusefulstruc -turesforencodingknowledge ,althoughwebelievetheyonlyapproxi -matetheunderlyingstructureofknowledgerepresentationthatemergesfromtheclassofmodelsweconsiderinthisbook ,asexplainedinChapter14.Ourmainpointhereisthatanytheorythattriestoaccountforhumanknowledgeusingscript -likeknowledgestructureswillhavetoallowthemtointeractwitheachothertocapturethegen -erativecapacityofhumanunderstandinginnovelsituations .Achievingsuchinteractionshasbeenoneofthegreatestdifficultiesassociatedwithimplementingmodelsthatreallythinkgenerativelyusingscript -orframe -likerepresentations .One reason forthe appeal ofPOP models is theirobvious "physiolog-ical"flavor : Theyseem so muchmoreclosely tiedtothephysiology ofthebrainthanareotherkindsofinformation -processing models.Thebrainconsistsofalargenumberofhighlyinterconnectedelements(Figure3)whichapparently send verysimpleexcitatoryand inhibitorymessages toeachotherandupdatetheirexcitationsonthebasis ofthese simplemessages.Theproperties ofthe unitsinmany ofthePOPmodelswewillbeexploringwereinspiredbybasic propertiesoftheneuralhardware.Inalatersectionofthisbook,wewillexamineinsome detailtherelationbetween POP models and the brain.10 THEPOPPERSPECTIVEPARALLEL DISTRIBUTEDPROCESSINGIntheexamples we have considered,a numberofdifferentpieces ofinformationmustbekeptinmindatonce.Eachplaysapart,con-strainingothersandbeingconstrainedbythem.Whatkindsofmechanisms seem wellsuitedtothese task demands?Intuitively ,thesetasks seem torequiremechanisms inwhicheach aspect oftheinforma-tioninthesituationcan act onotheraspects,simultaneouslyinfluenc -ingotheraspects andbeinginfluencedbythem.Toarticulatetheseintuitions , we and others have turnedtoa class ofmodels we call Paral-lelDistributedProcessing (POP)models.Thesemodelsassume thatinformationprocessing takesplace throughtheinteractionsofalargenumberofsimpleprocessing elements called units, each sending excita-toryandinhibitorysignalstootherunits.Insomecases,theunitsstand forpossible hypotheses aboutsuchthingsas thelettersina par-ticulardisplayorthesyntacticrolesofthewordsinaparticularsen-tence.Inthesecases,theactivationsstandroughlyforthestrengthsassociated withthedifferentpossible hypotheses,and theinterconnec-tionsamongtheunitsstandfortheconstraintsthesystemknowstoexistbetween thehypotheses.Inothercases, theunitsstand forpossi-blegoals andactions,suchas thegoaloftypingaparticularletter ,ortheactionofmovingtheleftindexfinger ,andtheconnectionsrelategoalstosubgoals,subgoals toactions,andactionstomusclemove-ments.Instillothercases,unitsstand notforparticularhypotheses orgoals,butforaspects ofthesethings.Thusahypothesisabouttheidentityofa word, forexample, is itselfdistributedinthe activations ofa large numberofunits.POPModels:CognitiveScienceor Neuroscience ?12THEPOPPERSPECTIVEThe Microstructure of CognitionTheprocessofhumancognition ,examinedonatimescaleofseconds andminutes,has a distinctlysequentialcharacter toit .Ideascome,seem promising,and thenare rejected;leads inthesolutiontoaproblemaretakenup,thenabandoned andreplaced withnewideas.Thoughtheprocess maynotbediscrete,ithas a decidedly sequentialcharacter,withtransitionsfromstate-to-stateoccurring,say,twoorthreetimesasecond.Clearly,anyusefuldescriptionoftheoverallorganizationofthissequential flowofthoughtwillnecessarily describea sequence ofstates.Butwhatistheinternalstructureofeachofthestatesinthesequence,andhowdotheycomeabout?Serious attemptstomodeleventhesimplestmacrosteps ofcognition - say,recognitionofsinglewords- requirevastnumbersofmicrostepsiftheyareimplementedsequentially.AsFeldmanandBallard( 1982)havepointedout ,thebiologicalhardwareisjusttoosluggishforsequentialmodelsofthemicrostructuretoprovideaplausibleaccount,atleastofthemicrostructureofhumanthought .Andthetimelimitationonlygetsworse,notbetter,whensequentialmechanismstrytotakelargenumbersofconstraintsintoaccount.Eachadditionalconstraintrequiresmoretimeina sequential machine,and,iftheconstraints areimprecise,theconstraintscanlead toacomputationalexplosion.Yetpeople getfaster,notslower,whentheyare able toexploitadditionalconstraints.Paralleldistributedprocessingmodelsofferalternativestoserialmodels ofthemicrostructureofcognition .Theydo notdeny thatthereisamacrostructure,justas thestudyofsubatomicparticles does notdenytheexistence ofinteractionsbetween atoms.WhatPDPmodelsdoisdescribetheinternalstructureofthelargerunits,justassubatomicphysicsdescribestheinternalstructureoftheatomsthatformthe constituentsoflarger unitsofchemical structure.Weshallshow as weproceed throughthisbookthattheanalysis ofthemicrostructureofcognitionhas importantimplicationsformostofthecentralissues incognitivescience.Ingeneral,fromthePDPpointofview,theobjects referredtoinmacrostructuralmodelsofcognitiveprocessing are seen as approximate descriptionsofemergent propertiesofthemicrostructure .Sometimes these approximate descriptions maybe sufficientlyaccurate tocapture a process ormechanism wellenough~butmanytimes,wewillargue,theyfailtoprovidesufficientlyelegantortractableaccountsthatcapturetheveryflexibilityandopen-endedness ofcognitionthattheirinventorshadoriginallyintendedtocapture.Wehopethatouranalysis ofPDPmodelswillshow howanexaminationofthemicrostructureofcognitioncan lead us closer toanadequate descriptionofthereal extentofhumanprocessing and learn-.. .Ing capacities.ThedevelopmentofPOP models is stillinitsinfancy.Thusfarthemodelswhichhavebeenproposed capturesimplifiedversionsofthekindsofphenomena wehavebeen describingratherthanthefullela-borationthatthesephenomena displayinrealsettings.Butwethinktherehave been enough steps forwardinrecent years towarrant a con-certedeffortatdescribingwhere theapproach has gottenandwhereitis going now, and topointoutsome directionsforthe future .Thefirstsectionofthebookrepresents anintroductorycourseinparalleldistributedprocessing.Therestofthischapterattemptstodescribe ininformaltermsanumberofthemodelswhichhavebeenproposed inpreviousworkandtoshowthattheapproach isindeedafruitfulone.Italso contains a briefdescriptionofthemajor sources oftheinspirationwehaveobtainedfromtheworkofotherresearchers.Thischapter is followed ,inChapter 2,bya descriptionofthequantita-tiveframeworkwithinwhichthese models can be described and exam-ined.Chapter 3 explicates one ofthecentral concepts ofthebook: dis-tributedrepresentation.Thefinalchapterinthissection,Chapter4,returnstothequestionofdemonstratingtheappealofparalleldistributedprocessing models and gives an overviewofourexplorationsinthemicrostructureofcognitionas theyare laidoutintheremainderofthisbook.1. THEAPPEALOFPDP 13EXAMPLESOFPDPMODELSInwhatfollows,wereviewa numberofrecentapplications ofPOPmodelstoproblemsinmotorcontrol ,perception,memory,andlanguage.Inmany cases, as we shall see,parallel distributedprocessingmechanisms are used toprovidenaturalaccounts oftheexploitationofmultiple ,simultaneous,and oftenmutualconstraints.We willalso seethatthese same mechanisms exhibitemergentproperties whichlead tonovelinterpretationsofphenomena whichhave traditionallybeen inter -preted inotherways.MotorControlHavingstartedwithanexampleofhowmultipleconstraintsappearto.....operateInmotorprogramming ,Itseemsappropriatetomentiontwo14THEPOPPERSPECTIVEmodelsinthisdomain .Thesemodelshavenotdevelopedfarenoughtocapturethefulldetailsofobstacleavoidanceandmultipleconstraintsonreachingandgrasping ,buttherehavebeenapplicationstotwoprob -lemswithsomeofthesecharacteristics .Fingermovementsinskilledtyping.Onemightimagine,atfirstglance ,thattypistscarryoutkeystrokessuccessively ,firstprogrammingonestrokeandthen,whenitiscompleted,programmingthenext .However ,thisisnotthecase.Forskilledtypists,thefingersare con-tinuallyanticipatingupcomingkeystrokes .Considerthewordvacuum.Inthisword ,thev,a,andcarealltypedwiththelefthand ,leavingtherighthandnothingtodountilitistimetotypethefirstu.However ,ahighspeed filmofa good typistshows thattherighthand moves uptoanticipatethetypingoftheu,evenas thelefthandis justbeginningtotypethev.Bythetimethecistypedtherightindexfingerisinposi -tionovertheuandreadytostrikeit .Whentwosuccessivekeystrokesaretobetypedwiththefingersofthesamehand ,concurrentpreparationtotypebothcanresultinsimilarorconflictinginstructionstothefingersand / orthehand .Consider ,inthislight ,thedifferencebetweenthesequenceevandthesequenceereThefirstsequencerequiresthetypisttomoveupfromhomerowtotypetheeandtomovedownfromthehomerowtotypethev,whileinthesecondsequence ,boththeeandthefareabovethehomerow .Thehandstakeverydifferentpositionsinthesetwocases.Inthefirstcase,thehandasawholestaysfairlystationaryoverthehomerow .Themiddlefingermovesuptotypethee,andtheindexfingermovesdowntotypethev.Inthesecondcase ,thehandasawholemovesup,bringingthemiddlefingerovertheeandtheindexfingeroverthef .Thus ,wecanseethatseveralletterscansimultaneouslyinfluencethepositioningofthefingersandthehands .Fromthepointofviewofoptimizingtheefficiencyofthetypingmotion ,thesedifferentpatternsseemverysensible .Inthefirstcase ,thehandasawholeismaintainedinagoodcompromisepositiontoallowthetypisttostrikebothlettersreasonablyefficientlybyextendingthefingersupordown .Inthesecondcase,theneedtoextendthefingersisreducedbymovingthewholehandup,puttingitinanear -optimalpositiontostrikeeitherkey .RumelhartandNorman( 1982)havesimulatedtheseeffectsusingPOPmechanisms .Figure4illustratesaspectsofthemodelas theyareillustratedintypingthewordvery.Inbrief ,RumelhartandNormanassumedthatthedecisiontotypeawordcausedactivationofaunitforthatword .Thatunit ,inturn ,activatedunitscorrespondingtoeachofthelettersintheword .Theunitforthefirstlettertobetypedwasmadetoinhibittheunitsforthesecondandfollowingletters ,theunitforthesecond toinhibitthethirdand followingletters,and so on.Asa resultoftheinterplayofactivationand inhibitionamong these units,theunitforthefirstletterwas at firstthemoststronglyactive, and theunitsforthe otherletters were partiallyactivated.Each letterunitexerts influencesonthehand and fingerinvolvedintypingtheletter .Thevunit ,forexample,tendstocause theindexfingertomovedownand tocause thewholehandtomovedownwithit .Thee unit ,ontheotherhand,tends tocause themiddlefingeronthelefthand tomoveup and tocause thewholehand tomoveup also.Therunitalsocauses theleftindexfingertomoveupandthelefthand tomoveup withit .Theextentoftheinfluencesofeach letteronthehand and fingeritdirects depends onthe extentoftheactivationofthe letter ." Therefore,atfirst ,intypingthewordvery,thevexertsthegreatestcontrol .wW"' X"' " ow~ zoo- - ~ z... a: ~ -upward...tw8rd + inwardupward;ftWid + OUIw ...downwardwW~ 5 ~ -loQz ~z ~ - -- ~ ( I : - lPALMPALMRESPONSESYSTEMLI (- 1,+0 .5)LM (+1, - 0 .3)LI (+1, - 0 .3)RI (+1,+1.3).targetfingerposition.. currentfingerpositionFIGURE 4.The interaction ofactivationsintyping thewordvery.Thevery unitisactivatedfromoutside the model .Itinturnactivatesthe units foreach ofthe com-ponent letters.Each letter unit specifiesthe target finger positions,specifiedina key-boardcoordinatesystem.LandR standfor the left and right hands, and I and M for theindex and middle fingers.The letter units receiveinformation about the current fingerposition fromthe responsesystem.Each letter unitinhibits the activation ofallletterunits that follow itinthe word:inhibitory connectionsare indicatedby the lines withsolid dots at their terminations.(From "Simulatinga Skilled Typist:A Study of SkilledMotor Pe~formance" by D. E. Rumelhartand D. A. Norman,1982, CognitiveScience ,6,p. 12.Copyright 1982by Ablex Publishing.Reprintedby permission.)15 1. THEAPPEALOFPOPTHUMBResponseSystemKeypressSchemataWordSchemaBecause the e and r are simultaneouslypullingthe hand up, though, thevistypedprimarilybymovingtheindexfinger ,andthereislittlemovementon thewhole hand.Onceafingeriswithinacertainstrikingdistance ofthekeytobetyped,theactualpressingmovementistriggered,andthekeypressoccurs.Thekeypress itselfcauses a stronginhibitorysignal tobe senttothe unitforthe letter justtyped, therebyremovingthisunitfromthepictureand allowingtheunitforthenextletterinthewordtobecomethemost stronglyactivated.Thismechanism provides a simpleway forallofthelettersto jointlydeterminethesuccessive configurationsthehandwillenterintointheprocess oftypinga word.Thismodelhas shownconsiderable successpredictingthetimebetween successive keystrokes as a functionofthedifferentkeys involved .Givena littlenoise inthe activationprocess,itcanalsoaccountforsomeofthedifferentkindsoferrorsthathavebeen observed intranscriptiontyping.Thetypingmodelrepresents anillustrationofthefactthatserialbehavior - asuccession ofkeystrokes- isnotnecessarily theresultofaninherentlyserialprocessing mechanism.Inthismodel ,thesequen-tialstructureoftypingemerges fromtheinteractionoftheexcitatoryand inhibitoryinfluences among the processing units.Reachingforanobjectwithoutfallingover.Similarmechanismscan be used tomodeltheprocess ofreaching foran object withoutlos-ingone' s balance whilestanding, as Hinton( 1984)has shown.He con-sidered a simpleversionofthistaskusinga two-dimensional"person"withafoot ,alowerleg,anupperleg,atrunk ,anupperarm,andalowerarm.Each ofthese limbsis joinedtothenextata jointwhichhas a singledegree ofrotationalfreedom.Thetaskposed tothisper-son istoreach a targetplaced somewhere infrontofit ,withouttakingany steps and withoutfallingdown.Thisis a simplifiedversionofthesituationinwhicha realperson has toreach outinfrontforan objectplaced somewhereintheplanethatverticallybisects thebody.Thetaskisnotas simpleas itlooks,since ifwe justswinganarmoutinfrontofourselves,itmay shiftourcenterofgravityso farforwardthatwewilllose ourbalance.Theproblem,then,istofinda setofjointangles thatsimultaneously solves thetwo constraints on the task.First ,thetipoftheforearmmusttouchtheobject.Second,tokeepfromfallingdown, theperson mustkeep its center ofgravityoverthe foot .Todo this,Hintonassigned a single processor toeach joint .On eachcomputationalcycle,each processor received informationabout how farthetipofthehand was fromthetarget and where thecenter ofgravitywas withrespect tothefoot .Usingthesetwopieces ofinformation ,each jointadjusted itsangle so as toapproach thegoals ofmaintaining16THEPOPPERSPECTIVEbalance andbringingthetipclosertothetarget.Afteranumberofiterations,thestick-person settledonpostures thatsatisfied thegoal ofreachingthetargetandthegoalofmaintainingthecenterofgravityover the "feet."Thoughthesimulationwas able toperformthetask, eventuallysatis-fyingbothgoals atonce,ithadanumberofinadequacies stemmingfromthefactthateach jointprocessor attemptedtoachieve a solutioninignorance ofwhat the other jointswere attemptingtodo.Thisprob-lemwas overcome by using additionalprocessors responsible forsettingcombinationsofjointangles.Thus,aprocessor forflexionand exten-sionofthelegwouldadjust theknee,hip,and ankle jointssynergisti-cally,whileaprocessor forflexionandextensionofthearmwouldadjusttheshoulderandelbowtogether.Withtheadditionofproces-sors ofthisform,thenumberofiterationsrequiredtoreach a solutionwasgreatlyreduced,andtheformoftheapproachtothesolutionlookedverynatural .Thesequence ofconfigurationsattainedinoneprocessing runis shown inFigure5.Explicitattemptstoprogramarobottocopewiththeproblemofmaintainingbalance as itreaches fora desired target have revealed thedifficultyofderivingexplicitlytherightcombinationsofactionsforeach possible startingstate and goal state.Thissimplemodelillustratesthatwemaybewrongtoseek such anexplicitsolution .Wesee herethata solutiontotheproblemcan emerge fromtheactionofa numberofsimpleprocessorseachattemptingtohonortheconstraintsindependently.I. THEAPPEALOFPDP 17++ + +FIGURE 5.Asequenceof configurationsassumedby the stick "person"performingthereachingtask describedin the text, from Hinton (1984).The small circle representsthecenter ofgravity ofthewhole stick-figure,and thecross representsthegoal tobereached.The configurationis shownon every seconditeration.Perception18 THEPOPPERSPECTIVEStereoscopicvision .Oneearlymodel usingparallel distributedpro -cessingwasthemodel ofstereoscopicdepthperceptionproposedbyMarrandPoggio( 1976 ) .Theirtheory proposedtoexplainthepercep -tionofdepthinrandom - dotstereograms ( Julesz ,1971 ;seeFigure6 )intermsofasimpledistributedprocessingmechanism .Julesz ' s random - dotstereograms presentinterestingchallengestomechanismsofdepthpercepti on .A stereogramconsistsoftworandom - dotpatterns .Inasimplestereogramsuchas theoneshownhere ,onepatternis anexactcopy oftheotherexceptthatthepatternofdotsinaregionofoneofthepatternsisshiftedhorizontallywithrespecttotherestofthepattern .Eachofthetwopatterns -correspondingtotworetinal images - consistsentirelyofapatternofrandomdots ,sothereisnoinformationineitherofthetwoviewscon -sideredalonethatcanindicatethepresenceofdifferentsurfaces ,letalonedepthrelations amongthosesurfaces .Yet ,whenoneofthesedotpatternsisprojectedtothelefteyeandtheothertotherighteye ,anobserversees eachregionasasurface ,withtheshiftedregionhoveringinfrontoforbehindtheother ,dependingonthedirectionoftheshift .FIGURE 6 .Random - dotstereograms .Thetwopatterns areidentical exceptthatthepatternofdotsinthecentral regionoftheleftpatternareshiftedoverwithrespecttothoseintheright .Whenviewedstereoscopicallysuchthattheleftpatternprojectstothelefteyeandtherightpatterntotherighteye ,theshiftedareaappearstohoverabovethepage .Somereaders may beabletoachievethisbyconvergingtoadistantpoint( e . g . ,afarwall )andtheninterposingthefigureintothelineofsight .( FromFoundationsofCyclopeanPerception ,p .21 ,byB .Julesz ,1971 ,Chicago :UniversityofChicagoPress .Copyright1971by Bell TelephoneLaboratories ,Inc .Reprintedbypermission . )1. THEAPPEALOFPDP 19What kindofa mechanismmight we proposetoaccount forthesefacts?Marr and Poggio(1976)beganby explicitly representingthe twoviews intwo arrays, as human observersmight in two different retinalimages.Theynoted thatcorrespondingblack dots atdifferentper-ceived distancesfromthe observer willbe offset fromeach other bygifferent amounts inthe two views.The jobof the model is to deter-mine which points correspond.This task is,ofcourse,made difficultbythefactthattherewillbeaverylargenumberofspuriouscorrespondencesofindividual dots.The goal of the mechanism, then,is to findthose correspondencesthat representreal correspondencesindepth and suppressthosethat representspuriouscorrespondences .To carry out this task, Marr and Poggioassigneda processingunit toeach possible,conjunction ofa point inone image and a point intheother.Since the eyesare offset horizontally, the possible conjunctionsoccur atvarious offsets ordisparitiesalong the horizontal dimension.Thus,foreach point inone eye,there was a set ofprocessingunitswith one unit assignedto the conjunctionof that point and the point ateachhorizontal offset from itin the other eye.Each processingunit receivedactivation wheneverboth of the pointsthe unit stood for containeddots.So far, then, units for both real andspurious correspondenceswouldbeequally activated.Toallowthemechanismtofindtherightcorrespondences ,theypointed outtwogeneralprinciplesabout the visual world:(a)Each point ineach viewgenerallycorrespondsto one and only one point in the other view, and(b)neighboringpoints in spacetend to be at nearly the samedepth andtherefore at about the same disparity inthe two images.While therearediscontinuities attheedges ofthings,overmostofatwo-dimensionalview ofthe world there willbe continuity.These princi-ples are calledthe uniquenessand continuityconstraints, respectively .Marr and Poggio incorporatedthese principles intothe interconnec-tions betweenthe processingunits.The uniquenessconstraintwascap-tured by inhibitory connectionsamong the units that stand for alterna-tivecorrespondencesofthesame dot.Thecontinuityprinciple wascapturedby excitatoryconnectionsamongthe units that stand for simi-lar offsets of adjacentdots.These additional connectionsallow theMarrand Poggio model to"solve"stereogramslike the one shown in the figure.Atfirst, when apair ofpatternsis presented, the units forall possiblecorrespondencesofa dotinone eye witha dotintheother willbe equally excited.However,theexcitatory connectionscause theunitsforthecorrectconjunctionsto receivemore excitation than units for spuriousconjunc-tions,and theinhibitoryconnectionsallow theunitsforthecorrectconjunctionsto turn offthe units forthe spuriousconnections.Thus,THEPOPPERSPECTIVEthemodeltendstosettledownintoastablestateinwhichonlythecorrect correspondence ofeach dotremains active.Thereare anumberofreasons whyMarrandPoggio( 1979)modi -fiedthismodel(seeMarr ,1982,foradiscussion) ,butthebasicmechanisms ofmutualexcitationbetween unitsthatare mutuallycon-sistentand mutualinhibitionbetween unitsthatare mutuallyincompa-tibleprovidea naturalmechanism forsettlingontherightconjunctionsofpointsandrejectingspuriousones.Themodelalsoillustrateshowgeneral principles orrules such as theuniqueness and continuityprinci -ples may be embodied inthe connections between processing units, andhowbehaviorinaccordance withthese principlescan emerge fromtheinteractions determinedby the pattern ofthese interconnections.20Perceptualcompletionof familiarpatterns.Perception,ofcourse,isinfluencedbyfamiliarity .Itis a well-knownfactthatwe oftenmisper-ceiveunfamiliarobjects as morefamiliarones andthatwe can getbywithless timeorwithlower-qualityinformationinperceivingfamiliaritemsthanweneedforperceivingunfamiliaritems.Notonlydoesfamiliarityhelpus determinewhat thehigher-levelstructures are whenthelower-levelinformationisambiguous~ italsoallowsustofillinmissinglower-levelinformationwithinfamiliarhigher-orderpatterns.Thewell -knownphonemic restoration effect isacase inpoint .Inthisphenomenon,perceivers hear sounds thathavebeen cutoutofwordsasiftheyhadactuallybeenpresent.Forexample,Warren(1970)presented /egi# /ature tosubjects, witha clickinthelocationmarked bythe# .Notonlydidsubjects correctlyidentifythewordlegislature~theyalsoheardthemissing / s/justas thoughithadbeen presented.Theyhad great difficultylocalizing theclick, whichtheytended tohearasadisembodiedsound.Similarphenomenahavebeenobservedinvisual perception ofwords since the workofPillsbury( 1897) .Twoofus have proposed a model describing theroleoffamiliarityinperceptionbased onexcitatoryandinhibitoryinteractionsamongunitsstandingforvarioushypotheses abouttheinputatdifferentlevelsofabstraction(McClelland&Rumelhart ,1981~ Rumelhart&McClelland,1982) .Themodelhas been appliedindetailtotheroleoffamiliarityintheperception oflettersinvisuallypresented words,and has provedtoprovideaverycloseaccountoftheresultsofalargenumberofex peri ments.Themodelassumes thatthereare unitsthatact as detectors forthevisualfeatures whichdistinguishletters,withoneset ofunitsassignedtodetectthefeaturesineachofthedifferentletter -positionsintheword.Forfour -letterwords, then, there are foursuch sets ofdetectors.Thereare also foursets ofdetectors fortheletters themselves and a setofdetectors forthe words.Inthemodel ,eachunithasanactivationvalue,correspondingroughlytothestrengthofthehypothesis thatwhatthatuni tstands forispresentintheperceptualinput .Themodelhonorsthefollowingimportantrelationswhichholdbetweenthese "hypotheses"oractiva-tions:First ,totheextentthattwohypotheses are mutuallyconsistent,theyshouldsupporteachother .Thus,unitsthataremutuallycon-sistent,inthewaythattheletterTinthefirstpositionisconsistentwiththewordTAKE,tendtoexciteeach other .Second,totheextentthattwohypotheses are mutuallyinconsistent, they should weaken eachother .Actually ,wecandistinguishtwokindsofinconsistency:Thefirstkindmightbecalledbetween-levelinconsistency.Forexample,thehypothesisthatawordbeginswithaTisinconsistentwiththehypothesis thatthewordis MO VE.Thesecond mightbe called mutualexclusion.Forexample,thehypothesisthatawordbeginswithTexcludesthehypothesisthatitbeginswithRsinceawordcanonlybegin withone letter .Bothkinds ofinconsistencies operate inthe wordperceptionmodeltoreducetheactivationsofunits.Thus,theletterunitsineach positioncompetewithallotherletterunitsinthesameposition,andthewordunitscompetewitheach other .Thistypeofinhibitoryinteractionisoftencalledcompetitiveinhibition.Inaddition,there are inhibitoryinteractionsbetween incompatibleunitson differentlevels.Thistypeofinhibitoryinteractionissimplycalledbetween-level inhibition.Theset ofexcitatoryand inhibitoryinteractionsbetween unitscan bediagrammedbydrawingexcitatoryandinhibitorylinksbetween them.Thewholepictureistoocomplextodraw,so weillustrateonlywithafragment :Some oftheinteractionsbetween someoftheunitsinthismodel are illustratedinFigure7.Letusconsiderwhathappens inasystem likethiswhenafamiliarstimulusispresentedunderdegraded conditions.Forexample,con-sider thedisplay shown inFigure8.Thisdisplay consists ofthelettersW,0,and R,completelyvisible,and enoughofa fourthlettertoruleoutalllettersotherthanRandK .Beforeonsetofthedisplay,theactivationsoftheunitsaresetatorbelowo.Whenthedisplayispresented,detectorsforthefeaturespresentineach positionbecomeactive(i .e.,theiractivations grow above 0) . Atthispoint , theybegin toexciteandinhibitthecorrespondingdetectorsforletters.Inthefirstthreepositions,W,0,and Rareunambiguouslyactivated,sowewillfocusourattentiononthefourthpositionwhereRandKarebothequally consistent withtheactive features.Here,theactivations ofthedetectors forRandKstartoutgrowingtogether,as thefeaturedetec-torsbelowthembecome activated.Asthese detectors become active,theyandtheactiveletterdetectors forW,0,and Rintheotherposi-tionsstarttoactivatedetectorsforwordswhichhavetheselettersinI. THEAPPEALOFPOP 21FIGURE8.Apossibledisplaywhichmightbepresentedtotheinteractiveactivationmodelofwordrecognition, and theresultingactivations ofselected letterand wordunits.Theletterunitsare fortheletters indicated inthe fourthpositionofa four -letterdisplay.However,theperceptual intelligenceabilitytorecognize familiarpat-WealsoshowfacilitationintheI. THEAPPEALOFPOP 23WordLevelLetter LevelKdetectorallowsittodominatethepatternofactivation,suppressingthe Rdetector completely.Thisexample illustrateshowPOP models can allowknowledge aboutwhatlettersgotogethertoformwordstoworktogetherwithnaturalconstraints onthetask(i .e.,thatthereshouldonlybe one letterinoneplace atonetime) ,toproduceperceptual completioninasimpleanddi rect way.Completionofnovel patterns.ofhumanperceiversfarexceedstheternsandfillinmissingportions.24THEPDPPERSPECTIVEperceptionoflettersinunfamiliarletterstringswhichareword -likebutnotthemselvesactuallyfamiliar .Onewayofaccountingforsuchperformancesistoimaginethattheperceiverpossesses,inadditiontodetectorsforfamiliarwords ,setsofdetectorsforregularsubwordunitssuchasfamiliarletterclusters ,orthattheyuseabstractrules ,specifyingwhichclassesofletterscangowithwhichothersindifferentcontexts .Itturnsout ,however ,thatthemodelwehavealreadydescribedneedsnosuchadditionalstructuretoproduceperceptualfacilitationforword -likeletterstrings ~ tothisextentitactsasifit" knows "theorthographicstructureofEnglish .Weillus -tratethisfeatureofthemodelwiththeexampleshowninFigure9,wherethenonwordrEADisshownindegradedformsothatthesecondletterisincompletelyvisible .Giventheinformationaboutthisletter ,consideredalone ,eitherEorFwouldbepossibleinthesecondposition .Yetourmodelwilltendtocompletethisletteras anE.Thereasonforthisbehavioristhat ,whenYEADisshown ,anumberofwordsarepartiallyactivated .ThereisnowordconsistentwithY,EorF,A,andD,buttherearewordswhichmatchYEA-( YEAR ,forexample )andotherswhichmatch_EAD(BEAD ,DEAD ,HEAD ,andREAD ,forexample ) .Theseandothernearmissesarepartiallyactivatedasaresultofthepatternofactivationattheletterlevel .Whiletheycompetewitheachother ,noneofthesewordsgetsstronglyenoughactivatedtocompletelysuppressalltheothers .Instead ,theseunitsactas agrouptoreinforceparticularlythelettersEandA .TherearenoclosepartialmatcheswhichincludetheletterFinthesecondposition ,sothisletterreceivesnofeedbacksupport .Asaresult ,Ecomestodominate ,andeventuallysuppress ,theFinthesecondposition .Thefactthatthewordperceptionmodelexhibitsperceptualfacilita -tiontopronounceablenonwordsaswellaswordsillustratesonceagainhowbehaviorinaccordancewithgeneralprinciplesorrulescanemergefromtheinteractionsofsimpleprocessingelements .Ofcourse ,thebehaviorofthewordperceptionmodeldoesnotimplementexactlyanyofthesystemsoforthographicrulesthathavebeenproposedbylinguists(Chomsky&Halle ,1968;Venesky ,1970)orpsychologists(Spoehr&Smith ,1975) .Inthisregard ,itonlyapproximatessuchrule -baseddescriptionsofperceptualprocessing .However ,rulesys-temssuchasChomskyandHalle ' sorVenesky ' sappeartobeonlyapproximatelyhonoredinhumanperformanceaswell(Smith&Baker ,1976) .Indeed ,someofthediscrepanciesbetweenhumanperformancedataandrulesystemsoccurinexactlythewaysthatwewouldpredictfromthewordperceptionmodel( Rumelhart&McClelland ,1982) .ThisillustratesthepossibilitythatPOPmodelsmayprovidemoreaccurateaccountsofthedetailsofhumanperformancethanmodels1.THEAPPEALOFPOP25Jlbased onasetsome domains.WordLevel~1:wI~ ~LetterLevelE~1:W~ ~ JC&FLTimeofrulesrepresentinghumancompetence - atleastinFIGURE9.Anexample ofa nonworddisplay thatmightbe presented totheinteractiveactivationmodelofwordrecognitionandtheresponse ofselected unitsattheletterandwordlevels.Theletterunitsillustratedaredetectorsforlettersinthesecondinputposition.RetrievingInformationFromMemoryContentaddressabi/ity.Oneveryprominentfeatureofhumanmemoryisthatitiscontentaddressable.Itseems fairlyclear thatweTHEPOPPERSPECTIVEcan accessinformationinmemorybased onnearlyany attributeofthe...representation we are trYing toretrieve.Ofcourse,somecuesaremuchbetterthanothers.Anattributewhichis shared bya verylarge numberofthingswe knowabout is nota very effectiveretrievalcue, since itdoes notaccurately pick outa par-ticularmemoryrepresentation.But ,several such cues,inconjunction ,candothejob .Thus,ifweaskafriendwhogoes outwithseveralwomen,"Whowas thatwomanIsaw youwith ?" ,hemaynotknowwhichonewemean- butifwespecifysomethingelse abouther- saythecolorofherhair ,what she was wearing(inso faras he remembersthisatall) ,wherewesaw himwithher - hewilllikelybeabletohitupon the rightone.Itis,ofcourse,possibletoimplementsomekindofcontentaddressability ofmemoryonastandard computerinavarietyofdif -ferentways.Onewayistosearchsequentially,examiningeachmemoryinthesystemtofindthememoryorthesetofmemorieswhichhas theparticularcontentspecifiedinthecue.Analternative,somewhatmoreefficient ,schemeinvolvessomeformofindexing -keepingalist ,foreverycontentamemorymighthave,ofwhichmemories have thatcontent .Suchanindexingschemecanbemadetoworkwitherror-freeprobes,butitwillbreak downifthereis an errorinthespecification oftheretrievalcue.Therearepossiblewaysofrecoveringfromsucherrors,buttheyleadtothekindofcombinatorialexplosionswhichplague thiskindofcomputerimplementation .Butsuppose thatweimaginethateach memoryisrepresented byaunitwhichhas mutuallyexcitatoryinteractionswithunitsstanding foreach ofitsproperties.Then,wheneveranypropertyofthememorybecame active,thememorywouldtendtobe activated,andwheneverthememorywas activated,allofitscontentswouldtendtobecomeactivated.Suchaschemewouldautomaticallyproducecontentaddressabilityforus.Thoughitwouldnotbeimmunetoerrors,itwouldnotbedevastatedbyanerrorintheprobeiftheremainingproperties specified the correct memory.As described thus far , whenever a propertythatis a part ofa numberofdifferentmemoriesisactivated,itwilltendtoactivateallofthememoriesitisin.Tokeepthese otheractivitiesfromswampingthe"correct"memoryunit ,we simplyneed toadd initialinhibitoryconnec-tionsamongthememoryunits.Anadditionaldesirable featurewouldbemutuallyinhibitoryinteractionsamongmutuallyincompatiblepropertyunits.Forexample,aperson cannotbothbe single and mar-riedat thesame time ,so theunitsfordifferentmaritalstates wouldbemutuallyinhibitory .26McClelland( 1981)developed a simulationmodelthatillustrateshowasystemwiththesepropertieswouldactasacontentaddressablememory. Themodelis obviouslyoversimplified ,butitillustratesmanyofthecharacteristicsofthemorecomplexmodelsthatwillbecon-sidered inlater chapters.ConsidertheinformationrepresentedinFigure10,whichlistsanumberofpeoplewemightmeetifwewenttoliveinanunsavoryneighborhood,and some oftheirhypotheticalcharacteristics.Asubset1. THEAPPEALOFPOP 27TheJetsandTheSharksNameGangAgeEduMarOccupation40 ' s30 ' s20 ' s40 ' s30 ' s20 ' s20 ' s20 ' s30 ' s20 ' s20 ' s20 ' s20 ' s20 ' s30 ' sPusherBookiePusherBurglarBookieBookieBurglarBurglarBurglarPusherBookiePusherCI) CI) CI) CI} CI} CI} CI} CI} CI} CI} CI} CI}. . . . . . . . . . . .============~~~~~~N~~~~~FIGURE 10.Characteristicsof a number of individualsbelongingto two gangs, the JetsandtheSharks.(From " Retrieving General andSpecific Knowledge FromStoredKnowledgeof SpecificS 'by J. L. McClelland,1981, Proceedingsof the ThirdAnnualConfer -enceof the CognitiveScienceSociety ,Berkeley, CA.Copyright 1981by J. L. McClelland.Reprintedby permission.)PusherBurglarBookieBookieBookieBurglarPusherBurglarBookieBurglarBurglarBookiePusherPusherPusherSing .Mar .Sing .Sing .Sing .Div .Mar .Mar .Sing .Mar .Div .Sing .Sing .Sing .Sing .Mar .Sing .Sing .Mar .Mar .Mar .Sing .Mar .Div .Mar .Sing .Div .oftheunitsneeded torepresent thisinformationis shown inFigure11.Inthisnetwork ,thereisan "instanceunit "foreach ofthe charactersdescribed inFigure10,andthatunitislinkedbymutuallyexcitatoryconnections toalloftheunitsforthefellow' s properties.Notethatwehave includedpropertyunitsforthenames ofthecharacters,as wellasunitsfortheirotherproperties.Now,suppose we wishtoretrievetheproperties ofa particularindi -vidual ,say Lance.Andsuppose thatwe knowLance' s name.Thenwecan probe thenetworkby activatingLance' s name unit ,and we can seewhatpattern ofactivationarises as a result .Assumingthatwe knowofnooneelse namedLance,wecan expecttheLancenameunittobehookeduponlytotheinstanceunitforLance.ThiswillinturnactivatethepropertyunitsforLance,therebycreatingthepatternofFIGURE 11.Someof the units and interconnectionsneededto representthe individualsshownin Figure 10.The units connectedwith double-headedarrowsare mutually excita-tory.Allthe units withinthe same cloud are mutually inhibitory.(From" RetrievingGeneral andSpecific ..KnowledgeFromStoredKnowledge ofSpecifics "byJ.L.McClelland,1981,Proceedingsofthe Third Annual Conferenceofthe CognitiveScienceSociety , Berkeley, CA.Copyright1981by J. L. McClelland.Reprintedby permission.)28THEPOPPERSPECTIVE29 1. THEAPPEALOFPOPactivationcorrespondingtoLance .Ineffect ,wehaveretrievedarepresentationofLance .Morewillhappenthanjustwhatwehavedescribedsofar ,butforthemomentletusstophere .Ofcourse ,sometimeswemaywishtoretrieveaname ,givenotherinformation .Inthiscase ,wemightstartwithsomeofLance ' sproperties ,effectivelyaskingthesystem ,say" WhodoyouknowwhoisaSharkandinhis20s ?"byactivatingtheSharkand20sunits .Inthiscaseitturnsoutthatthereisasingleindividual ,Ken ,whofitsthedescription .So ,whenweactivatethesetwoproperties ,wewillactivate.theinstanceunitforKen ,andthisinturnwillactivatehisnameunit ,andfillinhisotherpropertiesaswell .Gracefuldegradation .Afewofthedesirablepropertiesofthiskindofmodelarevisiblefromconsideringwhathappensaswevarythesetoffeaturesweusetoprobethememoryinanattempttoretrieveapar -ticularindividual ' sname .Anysetoffeatureswhichissufficienttouniquelycharacterizeaparticularitemwillactivatetheinstancenodeforthatitemmorestronglythananyotherinstancenode .Aprobewhichcontainsmisleadingfeatureswillmoststronglyactivatethenodethatitmatchesbest .Thiswillclearlybeapoorercuethanonewhichcontainsnomisleadinginformation - butitwillstillbesufficienttoactivatethe" rightanswer "morestronglythananyother ,aslongastheintroductionofmisleadinginformationdoesnotmaketheprobeclosertosomeotheritem .Ingeneral ,thoughthedegreeofactivationofaparticularinstancenodeandofthecorrespondingnamenodesvariesinthismodelasafunctionoftheexactcontentoftheprobe ,errorsintheprobewillnotbefatalunlesstheymaketheprobepointtothewrongmemory .Thiskindofmodel ' shandlingofincompleteorpartialprobesalsorequiresnospecialerror - recoveryschemetowork - itisanaturalby - productofthenatureoftheretrievalmechanismthatitiscapableofgracefuldegradation .TheseaspectsofthebehavioroftheJetsandSharksmodeldeservemoredetailedconsiderationthanthepresentspaceallows .Onereasonwedonotgointothemisthatweviewthismodelasasteppingstoneinthedevelopmentofothermodels ,suchasthemodelsusingmoredistributedrepresentations ,thatoccurinotherpartsofthisbook .Wedo ,however ,havemoretosayaboutthissimplemodel ,forlikesomeoftheothermodelswehavealreadyexamined ,thismodelexhibitssomeusefulpropertieswhichemergefromtheinteractionsofthepro -..cess1ngun1ts .Defaultassignment .Itprobablywillhaveoccurredtothereaderthatinmanyofthesituationswehavebeenexamining ,therewillbeotherTHEPOPPERSPECTIVEactivationsoccurringwhichmayinfluencethepatternofactivationwhichis retrieved.So,inthecase where we retrievedtheproperties ofLance,those properties,once theybecome active,can begin toactivatetheunitsforotherindividualswiththosesameproperties.ThememoryunitforLance willbe incompetitionwiththese unitsand willtendtokeeptheiractivationdown,buttotheextentthattheydobecome active,theywilltend toactivate theirownproperties and there-forefillthemin.Inthisway,themodelcanfillinpropertiesofindividualsbased on what itknows about other , similarinstances.Toillustratehowthismightworkwehavesimulatedthecase inwhichwedonotknowthatLance is a Burglaras opposed toa Bookieora Pusher.Itturnsoutthatthere are a groupofindividualsinthe setwhoareverysimilartoLanceinmanyrespects.WhenLance' spropertiesbecomeactivated,theseotherunitsbecomepartiallyactivated,and theystart activatingtheirproperties.Since theyallsharethesame "occupation,"theyworktogethertofillinthatpropertyforLance.Ofcourse, thereis noreason why thisshould necessarily be therightanswer,butgenerally speaking,themoresimilartwothingsare inrespects thatweknowabout,themorelikelytheyare tobe similarinrespects thatwe do not , and themodel implementsthisheuristic.30Spontaneousgeneralization .Themodelwehavebeendescribinghas anothervaluable propertyas well - ittends toretrievewhat is com-montothose memorieswhichmatcharetrievalcue whichistoogen-eraltocaptureanyonememory.Thus,forexample,we couldprobethesystem byactivatingtheunitcorrespondingtomembershipintheJets.ThisunitwillpartiallyactivatealltheinstancesoftheJets,therebycausing each tosend activationstoitsproperties.Inthiswaythemodelcan retrievethetypicalvalues thatthemembers oftheJetshaveoneachdimension- eventhoughthereisnooneJetthathasthese typicalvalues.Inthe example, 9 of15 Jets are single, 9 of15 areintheir20s,and9of15 haveonlyaJuniorHighSchooleducation~whenweprobebyactivatingtheJetunit ,allthreeofthese propertiesdominate.TheJets are evenlydividedbetween thethreeoccupations,so each ofthese unitsbecomes partiallyactivated.Each has a differentname, so thateach name unitis very weakly activated, nearly cancellingeach otherout .Intheexample justgivenofspontaneous generalization,itwouldnotbe unreasonable tosuppose thatsomeone mighthave explicitlystored ageneralizationaboutthemembersofagang.Theaccount justgivenwouldbeanalternativeto"explicitstorage"ofthegeneralization.Ithas twoadvantages,though,oversuchanaccount.First ,itdoes notrequireany special generalization formationmechanism.Second,itcanprovideuswithgeneralizationsonunanticipatedlines,ondemand.REPRESENTATION AND LEARNING IN PDP MODELS1. THEAPPEALOFPDP 31Thus,ifwe want toknow, forexample,whatpeople intheir20s withajuniorhighschooleducationarelike ,wecanprobethemodelbyactivatingthese twounits.Since allsuch people are Jets and Burglars,these twounitsare stronglyactivated by themodelinthiscase;twoofthemare divorcedand twoare married,so bothofthese unitsare par-tiallyactivated. 1Thesortofmodelweareconsidering,then,isconsiderablymorethanacontentaddressable memory.Inaddition,itperformsdefaultassignment,anditcan spontaneously retrievea general concept oftheindividualsthatmatchany specifiable probe.These properties mustbeexplicitlyimplementedascomplicatedcomputationalextensionsofothermodelsofknowledgeretrieval ,butinPOPmodelstheyarenatural by-products oftheretrievalprocess itself .IntheJetsandSharks model ,wecanspeak ofthemodel ' sactiverepresentationata particulartime,and associate thiswiththepatternofactivationovertheunitsinthesystem.Wecan also ask:Whatisthestoredknowledgethatgivesrise tothatpatternofactivation?Incon-sideringthisquestion,weseeimmediatelyanimportantdifferencebetween PDP models and othermodels ofcognitiveprocesses.Inmostmodels,knowledgeisstoredas astaticcopyofapattern.Retrievalamounts tofindingthepattern inlong-termmemoryand copying itintoabufferorworkingmemory.Thereisnoreal differencebetween thestoredrepresentationinlong-termmemoryandtheactiverepresenta-tioninworkingmemory.InPOP models,though,thisis notthecase.Inthese models,thepatterns themselves are notstored.Rather,whatisstoredistheconnectionstrengths between unitsthatallowthese pat-ternstobere-created.IntheJetsandSharksmodel ,thereisaninstance unitassigned toeach individual ,butthatunitdoes notcontaina copy oftherepresentation ofthatindividual .Instead,itis simplythecase thattheconnectionsbetween itandtheotherunitsinthesystemaresuchthatactivationoftheunitwillcausethepatternfortheindividualtobe reinstated onthe propertyunits.IInthisandallothercases ,thereisatendencyforthepatternofactivationtobeinnu -encedbypartiallyactivated ,nearneighbors ,whichdonotquitematchtheprobe .Thus ,inthiscase ,thereisaJetAI ,whoisaMarriedBurglar .TheunitforAIgetsslightlyactivated ,givingMarriedaslightedgeoverDivorcedinthesimulation .PERSPECTIVEThisdifferencebetweenPDPmodelsandconventionalmodelshasenormousimplications,bothforprocessing and forlearning.Wehavealreadyseensomeoftheimplicationsforprocessing .Therepresenta -tionoftheknowledgeissetupinsucha waythattheknowledgeneces-sarilyinfluencesthecourseofprocessing .Usingknowledgeinprocess-ingisnolongeramatteroffindingtherelevantinformationinmemoryandbringingittobear ~itispartandparceloftheprocessingitself .Forlearning ,theimplicationsareequallyprofound .Foriftheknowledgeisthestrengthsoftheconnections ,learningmustbeamatteroffindingtherightconnectionstrengthssothattherightpat -ternsofactivationwillbeproducedundertherightcircumstances .Thisisanextremelyimportantpropertyofthisclassofmodels ,foritopensupthepossibilitythataninformationprocessingmechanismcouldlearn ,asaresultoftuningitsconnections ,tocapturetheinterdependenciesbetweenactivationsthatitisexposedtointhecourseofprocessing .Inrecentyears,therehasbeenQuitealotofinterestinlearningincognitivescience .Computationalapproachestolearningfallpredom -inantlyintowhatmightbecalledthe" explicitruleformulation "tradi -tion ,as representedbytheworkofWinston( 1975) ,thesuggestionsofChomsky ,andtheACT .modelofJ.R.Anderson( 1983) .Allofthisworksharestheassumptionthatthegoaloflearningistoformulateexplicitrules(propositions ,productions ,etc .)whichcapturepowerfulgeneralizationsinasuccinctway .Fairlypowerfulmechanisms ,usuallywithconsiderable innateknowledge about a domain, and/ orsome start-ingsetofprimitivepropositionalrepresentations ,thenformulatehypotheticalgeneralrules ,e.g.,bycomparingparticularcases andfor -mulatingexplicitgeneralizations .TheapproachthatwetakeindevelopingPOPmodelsiscompletelydifferent .First ,wedonotassumethatthegoaloflearningisthefor -mulationofexplicitrules .Rather ,weassumeitistheacquisitionofconnectionstrengthswhichallowanetworkofsimpleunitstoactasthoughitknewtherules .Second ,wedonotattributepowerfulcompu -tationalcapabilitiestothelearningmechanism .Rather ,weassumeverysimpleconnectionstrengthmodulationmechanismswhichadjustthestrengthofconnectionsbetweenunitsbasedoninformationlocallyavailableattheconnection .Theseissueswillbeaddressedatlengthinlatersectionsofthisbook .Fornow ,ourpurposeistogiveasimple ,illustrativeexampleoftheconnectionstrengthmodulationprocess,andhowitcanproducenet -workswhichexhibitsomeinterestingbehavior .representation .Beforeweturntoanexplicitissue,weraiseabasicquestionabout32 THEPOPLocalvs.distributedconsiderationofthisrepresentation.Once we have achieved theinsightthattheknowledgeis stored inthestrengths ofthe interconnectionsbetween units, a ques-tionarises.Isthereanyreason toassign oneunittoeach patternthatwe wish tolearn?Anotherpossibility - one thatwe exploreextensivelyinthisbook- is thepossibilitythattheknowledge about any individualpatternisnotstoredintheconnectionsofaspecial unitreserved forthatpattern,butisdistributedovertheconnectionsamongalargenumberofprocessing units.Onthisview,theJets andSharks modelrepresents a special case inwhichseparate unitsarereserved foreachinstance.Modelsinwhichconnectioninformationisexplicitlythoughtofasdistributedhavebeenproposedbyanumberofinvestigators.Theunitsinthesecollectionsmaythemselvescorrespondtoconceptualprimitives ,ortheymayhavenoparticularmeaningas individuals .Ineithercase,thefocusshiftstopatternsofactivationoverthese unitsand tomechanisms whose explicitpurpose is tolearntherightconnec-tionstrengthstoallowtherightpatternsofactivationtobecomeactivated under therightcircumstances.Intherestofthissection,wewillgivea simpleexampleofaPOPmodelinwhichtheknowledge is distributed .Wewillfirstexplainhowthemodel wouldwork, givenpre-existingconnections, and we willthendescribe howitcouldcometoacquiretherightconnectionstrengthsthrougha verysimple learningmechanism.Anumberofmodels whichhave takenthisdistributedapproach have been discussed inthisbook' spredecessor,HintonandJ.A.Anderson' s( 1981)ParallelModelsofAssociative Memory.Wewillconsiderasimpleversionofacommontype ofdistributedmodel , a pattern associator .Patternassociators aremodelsinwhichapatternofactivationoveronesetofunitscan cause apatternofactivationoveranothersetofunitswithoutanyinterveningunitstostandforeitherpatternasawhole.Patternassociatorswould,forexample,becapableofassociating apatternofactivationononeset ofunitscorresponding totheappearanceofan object witha pattern on another set correspondingtothearomaoftheobject,sothat ,whenanobjectispresented visu-ally, causing its visualpattern tobecome active, themodelproduces thepattern corresponding toits aroma.33 I.THEAPPEALOFPOPHowapatternassociatorworks.Forpurposes ofillustration ,wepresent averysimplepatternassociator inFigure12.Inthismodel ,thereare fourunitsineach oftwopools.Thefirstpool ,theAunits,willbe thepoolinwhichpatterns corresponding tothesightofvariousobjects mightbe represented.Thesecond pool,theB units, willbe thepoolinwhichthepatterncorrespondingtothearomawillberepresented.Wecan pretendthatalternativepatterns ofactivationontheAunitsareproduceduponviewingaroseoragrilledsteak,andalternativepatterns ontheB unitsare produced upon sniffingthesameobjects.Figure13 shows twopairs ofpatterns,as wellas sets ofinter -connections necessary toallowtheAmemberofeach pair toreproducethe B member.Thedetailsofthebehavioroftheindividualunitsvaryamongdif -ferentversionsofpatternassociators.Forpresentpurposes,we' llassume thattheunitscan take onpositive ornegative activationvalues,with0 representing a kindofneutralintermediatevalue.Thestrengthsoftheinterconnectionsbetweentheunitscanbepositiveornegativereal numbers.Theeffectofan Aunitona B unitis determinedbymultiplyingtheactivationoftheAunittimesthestrengthofitssynapticconnectionwiththeBunit .Forexample,iftheconnectionfromaparticularAunittoaparticularBunithasapositivesign,whentheAunitis34 THEPOPPERSPECTIVEFromVisionA UnitsB UnitsFIGURE12.Asimplepatternassociator.Theexampleassumes thatpatterns ofactiva-tionintheAunitscan be produced bythevisual system andpatterns intheBunitscanbeproduced bytheolfactorysystem.Thesynaptic connections allowtheoutputsoftheAunitstoinfluencetheactivationsoftheBunits.Thesynaptic weightslinkingtheAunitstotheB unitswere selected so as toallowthepatternofactivationshown ontheAunitstoreproducethepatternofactivationshownontheBunitswithouttheneed forany olfactoryinput .FIGURE 13.Two simpleassociatorsrepresentedas matrices. The weightsin the first twomatricesallow the Apatternshown abovethe matrix to producethe B patternshown tothe right ofit.Note that the weightsin the first matrix are the sameas those shown inthe diagramin Figure 12.excited(activationgreaterthan0) ,itwillexcitetheBunit .Forthisexample,we' llsimplyassume thattheactivationofeach unitisset tothe sumoftheexcitatoryand inhibitoryeffects operating onit .Thisisone ofthe simplest possible cases.Suppose,now,thatwehavecreatedontheAunitsthepatterncorrespondingtothefirstvisualpatternshowninFigure13,therose.Howshouldwearrange thestrengthsoftheinterconnectionsbetweentheAunitsandtheBunitstoreproduce thepatterncorresponding tothearomaofarose?Wesimplyneedtoarrange foreach Aunittotendtoexciteeach B unitwhichhas a positiveactivationinthearomapattern and toinhibiteach B unitwhichhas a negative activationinthearomapattern.Itturnsoutthatthisgoalisachievedbysettingthestrengthoftheconnectionbetween a givenAunitand a givenBunittoa value proportionaltotheproductoftheactivationofthe twounits.InFigure12,theweights ontheconnections were chosen toallowtheApattern illustratedtheretoproduce theillustratedB pattern accordingtothisprinciple.Theactualstrengthsoftheconnectionsweresetto:::t: .25,ratherthan:t: 1, so thattheApattern willproduce therightmag-nitude,as wellas therightsign, fortheactivations ofthe unitsintheBpattern.Thesame connections arereproduced inmatrixforminFig-ure13A.Patternassociators liketheoneinFigure12 have a numberofniceproperties.Oneisthattheydonotrequirea perfectcopy oftheinputtoproduce the correct output , thoughits strengthwillbe weaker inthiscase. Forexample,suppose thatthe associator shown inFigure12 werepresented withan Apattern of( 1,- 1,0, 1) .Thisis theApattern showninthefigure ,withthe activationofone ofits elements set toO.TheBpatternproducedinresponse willhavetheactivationsofalloftheBunitsintherightdirection ;however,theywillbesomewhatweakerthantheywouldbe,hadthecompleteApatternbeen shown.Similar35 1.THEAPPEALOFPOP- 1+1- 1+1+1- 1- 1+1- 1- 1+ 1+ 1- 1+1+1- 1- .25+.25+.25- .25- .25+.25+.25- .25+.25- .25- .25+.25+.25- .25- .25+.25THE 36 POPPERSPECTIVEeffects are producedifan element of the pattern is distorted- or ifthemodel is damaged , either by removing whole units,or random sets ofconnections,etc.Thus,theirpattern retrievalperformance ofthemodeldegrades gracefullybothunderdegraded inputandunderdamage.Howa pattern associator learns.So far,we have seen how we asmodel builders can constructthe right set ofweights to allow one pat-tern to causeanother.The interestingthing, though, is that we do notneed tobuild these interconnectionstrengthsinby hand.Instead, thepattern associatorcanteach itselftherightsetofinterconnectionsthrough experienceprocessingthepatterns inconjunction witheachother.Anumber ofdifferent rules foradjustingconnectionstrengthshavebeen proposed.One of the first- and definitely the best known- is duetoD.O.Hebb (1949) .Hebb' sactual proposal was notsufficientlyquantitativeto build into an explicit model.However, a number of dif-ferent variantscan trace their ancestryback to Hebb.Perhapsthe sim-plest version is:When unitAand unitB are simultaneouslyexcited,increasethe strength of the connectionbetweenthem.Anatural extension ofthisruletocover thepositive and negativeactivationvaluesallowedin our exampleis:Adjust the strengthof the connectionbetweenunits A and B inproportion to the product of their simultaneousactivation.Inthisformulation,ifthe product ispositive,the changemakes theconnectionmore excitatory, and ifthe product is negative,the changemakesthe connectionmore inhibitory.For simplicity ofreference, wewillcall this the Hebb rule,although itis notexactly Hebb's originalformulation.Withthis simple learning rule,we could train a "blank copy"ofthepattern associatorshown in Figure 12 to producethe 8 pattern for rosewhen the Apattern is shown,simply by presentingthe Aand B pat-terns togetherand modulatingthe connectionstrengthsaccordingto theHebb rule.Thesize ofthechange made onevery trialwould,ofcourse, be a parameter .We generallyassumethat the changesmadeoneach instanceare rather small,and that connectionstrengthsbuild upgradually.The valuesshown in Figure 13A, then, would be acquiredasa result of a number of experienceswith the A and B pattern pair.Itisveryimportanttonotethattheinformationneeded touse theHebb ruletodeterminethevalue each connectionshould have is locallyavailable at the connection.Alla givenconnectionneeds toconsider istheactivationoftheunitsonbothsides ofit .Thus,itwouldbe possi-ble toactually implementsuch a connectionmodulationscheme locally,ineachconnection,withoutrequiringanyprogrammertoreachintoeach connection and set itto justtherightvalue.ItturnsoutthattheHebbruleas stated here has some serious limi -tations,and,toourknowledge,notheoristscontinuetouse itinthissimpleform.Moresophisticated connectionmodulation.schemeshavebeen proposed byotherworkers~mostimportantamong these are thedeltarule,discussed extensivelyinChapters 8 and11;thecompetitivelearningrule,discussed inChapter5~ and therulesforlearninginsto-chasticparallelmodels,describedinChapters6and7.Alloftheselearningruleshavethepropertythattheyadjustthestrengthsofcon-nections between unitsonthebasis ofinformationthatcan be assumedtobe locallyavailable totheunit .Learning,then,inallofthese cases,amountstoaverysimpleprocess thatcanbeimplementedlocallyateach connectionwithouttheneedforanyoverallsupervision.Thus,models whichincorporatethese learningrulestrainthemselves tohavetherightinterconnectionsinthecourse ofprocessing themembersofan ensemble ofpatterns.Learningmultiplepatternsinthesameset ofinterconnections.Uptonow,we have considered howwe mightteach ourpatternassociatortoassociate thevisualpatternforoneobjectwithapatternforthearoma ofthesame object.Obviously, differentpatterns ofinterconnec-tionsbetween theAandBunitsare appropriate forcausing thevisualpatternforadifferentobjecttogiverisetothepatternforitsaroma.Thesame principlesapply,however,andifwepresented ourpatterriassociator withtheAand Bpatterns forsteak,itwouldlearntherightset ofinterconnectionsforthatcase instead(these are shown inFigure13B) .Infact,itturnsoutthatwecan actuallyteach thesame patternassociator a numberofdifferentassociations.Thematrixrepresentingtheset ofinterconnectionsthatwouldbe learned ifwe taught thesamepatternassociator boththerose association and thesteak association isshowninFigure14.Thereadercanverifythisbyaddingthetwomatrices fortheindividualpatterns together.Thereader can also verifythatthisset ofconnections willallowtherose Apattern toproduce therose B pattern,and thesteak Apatterntoproduce thesteak B pattern:when eitherinputpattern is presented, the correct corresponding outputis produced.Theexamplesusedherehavethepropertythatthetwodifferentvisualpatterns are completelyuncorrelated witheach other .Thisbeing37 1. THEAPPEALOFPDPthecase,therose patternproduces noeffectwhen theinterconnectionsforthesteak have been established,and thesteak patternproduces noeffectwhentheinterconnectionsfortherose association are ineffect .Forthisreason,itispossible toadd togetherthepatternofintercon-nectionsfortherose association andthepatternforthesteak associa-tion ,and stillbe able toassociate thesightofthesteak withthesmellofa steak and thesight ofa rose withthe smellofa rose.Thetwosetsofinterconnectionsdo notinteract at all .OneofthelimitationsoftheHebbianlearningruleisthatitcanlearn theconnectionstrengths appropriate toan entireensemble ofpat-ternsonlywhenallthepatternsarecompletelyuncorrelated.Thisrestrictiondoesnot ,however,applytopatternassociators whichusemore sophisticated learning schemes.Attractivepropertiesof patternass0cia tormodels.Pattern associatormodels have theproperty thatuncorrelated patterns do notinteract witheach other ,butmoresimilarones do.Thus,totheextentthata newpatternofactivationontheAunitsis similartoone oftheoldones,itwilltendtohave similareffects.Furthermore ,ifwe assume thatlearn-ingtheinterconnectionsoccursinsmallincrements,similarpatternswillessentiallyreinforcethestrengthsofthelinkstheyshare incom-monwithotherpatterns.Thus,ifwe present thesame pair ofpatternsoverand over,buteach timewe add a littlerandomnoise toeach ele-mentofeach memberofthepair, thesystem willautomatically learn toassociate thecentraltendencyofthetwopatternsandwilllearntoignorethenoise.Whatwillbe storedwillbe an average ofthesimilarpatterns withtheslightvariationsremoved.Ontheotherhand,whenwepresent thesystem withcompletelyuncorrelatedpatterns,theywillnotinteractwitheach otherinthisway.Thus,thesame poolofunitscan extractthecentraltendency ofeach ofa numberofpairs ofunre-lated patterns. Thisaspect ofdistributedmodels is exploitedextensivelyinChapters 17 and 25 on distributedmemoryand amnesia.38THEPOPPERSPECTIVE--++--+++--+-++-1'.111_11--++++++-+ ++- --- -- -.~~~.I!III_IFIGURE 14.The weightsin the third matrix allow either Apatternshown in Figure 13torecreatethe corresponding8pattern. Eachweight in this caseis equal to the sum ofthe weightfor the A patternand the weightfor the B pattern, as illustrated.Extractingthestructureofanensemble ofpatterns.Thefactthatsimilarpatternstendtoproducesimilareffectsallowsdistributedmodelstoexhibitakindofspontaneousgeneralization,extendingbehaviorappropriateforonepatterntoothersimilarpatterns.Thispropertyisshared byotherPDPmodels,suchas thewordperceptionmodeland theJets and Sharks modeldescribed above~ themaindiffer -ence here is intheexistence ofsimple,local,learningmechanisms thatcan allowtheacquisitionoftheconnectionstrengths needed toproducethese generalizations throughexperiencewithmembersoftheensem-bleofpatterns.Distributedmodelshaveanotherinterestingpropertyas well :Ifthereareregularitiesinthecorrespondences between pairsofpatterns,themodelwillnaturallyextracttheseregularities.Thispropertyallowsdistributedmodelstoacquirepatternsofinterconnectionsthatlead themtobehave inways we ordinarilytake asevidence fortheuse oflinguisticrules.AdetailedexampleofsuchamodelisdescribedinChapter18.Here,wedescribe themodelverybriefly .Themodelisa mechanismthatlearnshowtoconstructthepast tenses ofwordsfromtheirrootformsthroughrepeated presentations ofexamples ofrootformspairedwiththecorrespondingpast-tenseform .Themodelconsistsoftwopools ofunits.Inone pool ,patterns ofactivationrepresenting thepho-nologicalstructureoftherootformoftheverbcanberepresented,and,intheother ,patternsrepresentingthephonologicalstructureofthepast tense can be represented.Thegoal ofthemodelissimplytolearntherightconnectionstrengthsbetweentherootunitsandthepast-tense units,so thatwhenever therootformofa verbis presentedthemodelwillconstructthe corresponding past-tense form .Themodelis trainedby presenting therootformoftheverbas a pattern ofactiva-tionovertherootunits,and thenusing a simple, local, learningruletoadjust theconnectionstrengths so thatthisrootformwilltendtopro-ducethecorrectpatternofactivationoverthepast-tenseunits.Themodelistestedbysimplypresentingtherootformasapatternofactivationovertherootunitsandexaminingthepatternofactivationproduced overthe past-tense units.Themodelistrainedinitiallywitha smallnumberofverbs childrenlearnearlyintheacquisitionprocess.Atthispointinlearning,itcanonlyproduce appropriateoutputsforinputsthatithas explicitlybeenshown.Butas itlearns moreand moreverbs,itexhibitstwointerest-ingbehaviors.First ,itproduces thestandard ed past tense when testedwithpseudo-verbsorverbsithas neverseen.Second,it"overregular-izes"thepast tense ofirregularwords itpreviouslycompleted correctly.Often,themodelwillblendtheirregularpast tense ofthewordwiththeregulared ending,andproduceerrorslikeCAMEDas thepast of1. THEAPPEALOFPOP 39COME .Thesephenomenamirrorthoseobservedintheearlyphasesofacquisitionofcontroloverpasttensesinyoungchildren .Thegenerativityofthechild ' sresponses - thecreationofregularpasttensesofnewverbsandtheoverregularizationoftheirregularverbs - hasbeentakenasstrongevidencethatthechildhasinducedtherulewhichstatesthattheregularcorrespondenceforthepasttenseinEnglishistoaddafinaled(Berko ,1958) .Ontheevidenceofitsper -formance ,then ,themodelcanbesaidtohaveacquiredtherule .How -ever ,nospecialrule -inductionmechanismisused,andnospeciallanguage -acquisitiondeviceisrequired .Themodellearnstobehaveinaccordancewiththerule ,notbyexplicitlynotingthatmostwordstakeedinthepasttenseinEnglishandstoringthisruleawayexplicitly ,butsimplybybuildingupasetofconnectionsinapatternassociatorthroughalongseriesofsimplelearningexperiences .Thesamemechanismsofparalleldistributedprocessingandconnectionmodi fica -.tionwhichareusedinanumberofdomainsserve ,inthiscase ,topro -duceimplicitknowledgetantamounttoalinguisticrule .Themodelalsoprovidesafairlydetailedaccountofanumberofthespecificaspectsoftheerrorpatternschildrenmakeinlearningtherule .Inthissense,itprovidesaricherandmoredetaileddescriptionoftheacquisi -tionprocessthananythatfallsoutnaturallyfromtheassumptionthatthechildisbuildingupa repertoireofexplicitbutinaccessiblerules .Thereisalotmoretobesaidaboutdistributedmodelsoflearning ,abouttheirstrengthsandtheirweaknesses ,thanwehavespaceforinthispreliminaryconsideration .Fornowwehopemainlytohavesug-gestedthattheyprovidedramaticallydifferentaccountsoflearningandacquisitionthanareofferedbytraditionalmodelsoftheseprocesses.Wesawinearliersectionsofthischapterthatperformanceinaccor -dancewithrulescanemergefromtheinteractionsofsimple ,intercon -nectedunits .Nowwecanseehowtheaquisitionofperformancethatconformstolinguisticrulescanemergefromasimple ,local ,connec -tionstrengthmodulationprocess .WehaveseenwhatthepropertiesofPOPmodelsareininformalterms ,andwehaveseenhowthesepropertiesoperatetomakethemodelsdomanyofthekindsofthingsthattheydo.Thebusinessofthenextchapteristolayoutthesepropertiesmoreformally ,andtointroducesomeformaltoolsfortheirdescriptionandanalysis .Beforeweturntothis ,however ,wewishtodescribesomeofthemajorsourcesofinspirationforthePOPapproach .40 THEPOPPERSPECTIVETheideasbehindthePDPapproachhaveahistorythatstretchesback indefinitely .Inthissection,we mentionbrieflysome ofthepeo-ple who have thoughtinthese terms,particularlythose whose workhashad an impact onourownthinking .Thissection shouldnotbeen seenas anauthoritativereviewofthehistory ,butonlyas adescriptionofourown sources ofinspiration.Some oftheearliest rootsofthePOP approach can be foundintheworkoftheuniqueneurologists,Jackson( 1869/ 1958)andLuria( 1966) .Jackson was aforcefulandpersuasive criticofthesimplisticlocalizationistdoctrinesoflatenine