Embodied Cognition and the Percept/Concept Distinction

Michael HaywardDepartment of Cognitive Science, UCSD

December, 1998

Disclaimer: This paper is the result of a brief research foray undertaken as part of a degree programrequirement. Although I believe it to be entirely accurate, it is neither complete nor authoritative.Please do not cite.

1

Embodied Cognition and the Percept/Concept Distinction

Michael HaywardDepartment of Cognitive Science, UCSD

mhayward@cogsci.ucsd.edu

Embodied cognition, a theoretical framework which extends the definition of cognitive activityto include bodily interactions with the world, stands as a challenge to the conventional view ofcognition as primarily abstract, rule-based symbol manipulation in the brain. Proponents ofembodied cognition downplay or altogether deny the existence of such an abstract conceptualsystem independent from perception. This paper integrates theory and evidence across manydomains in cognitive science, in an attempt to evaluate the claim of some embodied cognitionproponents that modal, perceptual activity can account for most or all of the cognitivephenomena traditionally viewed as "conceptual". I find that much of the evidence presented assupport for either side is misleading, as it is based on the assumption that existing theories arefar more complete than they really are. Nevertheless, I conclude that there is enough strongevidence to support several key claims, including the existence of perceptual components inconceptual thought, and the inability of some classes of propositional conceptual system toadequately explain the full range of human conceptual phenomena.

1. Introduction

1.1) Two Perspectives on Cognition

In the last few years, cognitive science has seen aresurgence of interest in placing cognition back in thecontext of a body acting in a social and materialenvironment. In Being There, Andy Clark stresses that"the biological mind is, first and foremost, an organ forcontrolling the biological body" (pg.1). Beyond simplyemphasizing the importance of body and environment,proponents of this view argue that cognitive behaviorcannot be understood without recognizing that oureveryday cognitive activity extends outside of the brain,and into our bodily interactions with the world (Hutchins,1995). Inherent in this "embodied cognition" view is anappreciation for the importance of the sensorimotorsystem as a locus of cognitive activity, rather than just amediator between the world and the mind.

The embodied cognition approach stands in contrastto many of the assumptions that have underpinnedcognitive science for decades. The process viewsupported by these (typically implicit) assumptions is asfollows: humans receive perceptual input, model it insome abstract "language" of conceptual thought, apply

rule-based symbol manipulation to process it, and finallyproduce motor output as a result. With the digitalcomputer as a metaphor, problem solving and planningare offered as the quintessential cognitive activities underthis "disembodied" view. Understanding therepresentations which underlie cognition then reduces tofinding the appropriate conceptual language to supportjust this kind of symbol manipulation.

In contrast, embodied cognition advocates seek todescribe cognition through representations formeddynamically across the interactions of the mind, thebody, and the world. Researchers downplay or, inextreme cases, entirely deny the existence of an abstractconceptual symbol system independent of perception.Instead, they posit perceptual symbol structures, ofteninstantiated in the same brain regions responsible forrepresenting external stimuli. The challenge is thereforeto redescribe conceptual phenomena in perceptual terms,and it is this "percept/concept debate" which is the focusof this paper. Just how far can a purely perceptual mindgo towards explaining cognition?

To give a sense for the appeal of each approach, letme briefly review some of the intuitive arguments whichsupport them. On the one hand, it seems self-evident thatthe brain did evolve primarily to control bodily behavior,and thus sensorimotor activity should form the

2

foundation of the mind. Furthermore, we certainlyexperience the conscious feeling of manipulating mentalimages, as if “pictures” truly were a medium of higherthought. On the other hand, it's not clear how abstractconcepts like "justice" could possibly be represented in amodal, perceptual form. Furthermore, classical,propositional artificial intelligence has produced somepowerful techniques which, in select problem domains,fit human performance data very well. These are thekinds of issues which have lead to the debate overconcept representation.

It is misleading, however, to suggest that thereexists a single dimension along which these views differ.By posing the problem as a "debate", it may seem tosuggest that there are only two opposing views. In fact,there is an enormous range of theoretical positionspresented in the literature. In order to make thisdiscussion tractable, I have tried to reduce the keydimension to that of the role of the sensorimotor systemin conceptual processing. Inevitably, however, such anapproach fails to entirely capture the depth andsignificance of all the issues involved.

1.2) Approach

The concept literature is filled with unsubstantiatedassertions and speculation. Debates are dominated byarguments over definitions and partitionings of thetheoretical space, rather than empirical evidence (e.g., the"situated cognition" special issue of Cognitive Sciencecentered around Vera & Simon, 1993).

My goal is to try to provide a somewhat moreobjective, evidence-oriented analysis of this issue,integrated across many domains in cognitive science. Iwill do so by testing the explanatory power of the majorframeworks against a common set of conceptualphenomena, each of which has been offered as evidencefor some view in the literature. I thus explore eachtheory's ability to explain a wide range of phenomena,including those that are most and least natural for it.

While attempting an objective evaluation of theevidence, the balance of my arguments must neverthelessfocus on confirming or denying the claims of theembodied camp. This imbalance is necessary primarilybecause the embodied approach is challenging theestablished view. As a result, most of the debate anddiscussion is focussed on supporting or refuting thatchallenge. Rarely is the disembodied approach argued fordirectly; instead, it is supported by rejecting the claims ofthe challengers.

1.3) Outline of the Paper

• In Section 2, "Differing views of the role ofperception...", I revisit the different sides of the debate,

giving a brief review of the background and importantdimensions along which the research varies.

• In Section 3, "A Functional View of the HumanConceptual System", I define "concept" by consideringthe types of phenomena theorists are trying to explain.The phenomena selected have each been presented asevidence for some view in the literature. This is the listof phenomena against which the major frameworks willlater be judged.

• In Section 4, "Domains and Theories", I outline thewide range of methods and domains of research that arerelevant to the issue. In each, I try to summarize thecontributions of a few influential individuals whose workI have chosen to focus on.

• In Section 5, "Analysis", I present each of thephenomena from Section 3, and consider how well theyare explained by each of the major frameworks fromSection 4. In the course of this analysis, I presentempirical evidence, and explore issues raised in section 4in greater depth.

• In Section 6, "Conclusions", I summarize the results ofthe analyses and present my own conclusions with regardto the debate, and the state of the research as a whole.

2. Differing views of the role ofperception, the body, and the

environment

In this section, I give a more detailed characterization ofthe views held on each side of the debate. Though theextreme versions of these positions differ greatly, none ofthe views described can be dismissed outright by theavailable evidence. Influential researchers in moderncognitive science represent all sides.

2.1) The Disembodied, Conceptual View

The disembodied view is an implicit assumptionunderlying much of modern cognitive science. Theadvent of the digital computer had a tremendous impacton our view of the human brain. The field of artificialintelligence was not just passively influenced bypsychological research; it actively influenced psychologythrough its emphasis on serial processing, input/outputmodels, theory of general computation through symbolmanipulation, etc. Computers became very powerful,very quickly through these techniques. With theadoption of the view of "cognition as computation",cognitive science looked to these same frameworks forguidance.

In computers, it was clear that the "real work" wasbeing done in the CPU; the "perceptual systems" were

3

simply modified versions of old technology: TV sets,typewriters, cassette players, etc. Artificial intelligenceadopted the same view, and set about trying to model the"real" intelligent behaviors of human beings: problemsolving, theorem proving, chess playing, and so on. Thegoal was to find the appropriate internal representation tomodel and manipulate the necessary symbols; input andoutput were secondary.

Newell and Simon’s influential "physical symbolsystem hypothesis" (1976) made the disembodied idealexplicit: intelligence could be completely described interms of the correct symbols moving in the correct ways,independent of their implementation. With this view, thebody and the world are pushed into the background. AImet with important successes with this approach,developing processes (e.g., production systems) andrepresentations (e.g. slot-and-filler structures, semanticnets, etc.) that are still among the most powerfulcomputing techniques applied today. It was in thisenvironment that cognitive science looked forpropositional representations to model humanknowledge, and rule-based symbol manipulators tomodel reasoning. Many of these approaches werefounded on the theory of formal logics, and themathematics of computers became the mathematics ofthe mind.

Today, after decades of such work, many of thesesame views still dominate artificial intelligence (e.g.,Lenat & Guha (1990)) and cognitive science (e.g. Fodor& Pylyshyn (1988)). I will be referring to this approachas "disembodied", although the term is rarely usedexplicitly.

The nature of human cognition under this view,then, is inherently nonperceptual. Concepts are typicallyrepresented as propositional structures abstracted fromthe sensory modalities by some process of filtering andtransduction. In this way, a computer-like serialprocessing sequence is enforced: perceptual input isreceived, translated for conceptual processing, and thentranslated into motor signals for output. The conceptualrepresentations form a sort of "language" underlying allcognitive activity. In their most traditional form, theserepresentations explicitly include dictionary-likenecessary and sufficient conditions (e.g. the conceptbachelor is explicitly defined as "unmarried male") orinvariant feature lists (e.g. bachelor: gender(male) +status(unmarried)). Other modern forms includeprobabilistic components and fuzzy set theory to accountfor graded category membership, and other suchtechniques. The common tie between all of them is thatconcepts are treated as abstract (non-perceptual), well-defined entities manipulated through a set of explicitrules.

2.2) The Embodied, Perceptual View

The embodied approach emphasizes a completelydifferent aspect of cognitive activity altogether. In thisview, the social and material environment are not onlythe driving forces behind cognition, but actually form anintegral part of most cognitive activity. Perceptual andmotor activity are in no way secondary to more abstractprocesses; perception and action in the real world formthe foundation of cognition. Cognitive activity is definedby a constant, dynamic relationship betweenrepresentations within the brain and outside the brain. Inmany cases, imagistic, analogistic, and dynamicalexplanations replace logical symbol manipulation as thefoundation for human reasoning. Importantly,representations often take the most "external" formpossible, in order to save the time and energy required totranslate information into "deeper", more abstract formsfor processing. When possible, this means thatrepresentations are left outside of the body entirely, in theform of "cognitive artifacts", as described by Hutchins(1995). Kirsh & Maglio (1992) present a clear exampleof pushing representation out into the world in theirresearch on the video game "Tetris". According to Kirsh& Maglio, when it becomes necessary to rotate a simplegeometric shape as a basis for a strategy judgement understrict time constraints, expert players prefer physical,rather than mental, rotation of the figures. This seems tosuggest that the sensorimotor loop is in some way moreefficient than "deep" conceptual processing.

In this view, the next best thing to externalrepresentations is to manipulate internal perceptual tracesdirectly, without having to translate to a deeper"conceptual" language. Conceptual symbols are: (a)systematically, rather than arbitrarily, related to theperceptual traces they represent, and (b) active in thesame modality-specific perceptual region of the brain asthe original perceptual trace. I will explore this view indepth in Section 4.

2.2.1. Why Embodiment?

Given the significant successes of the disembodiedapproach, what caused researchers to look to anembodied view as an alternative? Numerous reasonswere behind the move for change. Experimentalpsychologists were discovering that even the most robusthuman behaviors are highly context-dependent; relativelyminor changes to otherwise highly controlled laboratoryexperiments could change results significantly. Evenphenomena as "low-level" as color perception weresubject to context effects (Thompson, 1995). As a result,the validity of the laboratory setting came into question,as researchers formed a new appreciation for theimportance of the complicated and noisy environments inwhich humans operate naturally.

4

Along with a renewed appreciation for theimportance of context, it soon became clear thatorganisms, and particularly humans, create and takeadvantage of structure in their environments to offloadinternal cognitive work (Hutchins, 1995). Furthermore,these environmental interactions are so quintessentiallycognitive that it is hard to justify arbitrarily limiting theterm "cognitive" to refer to internal activity.

In AI, it was becoming clear that tasks such astheorem proving, originally assumed to be the mostdifficult, were turning out to be easier than the most basicphysical behaviors like object recognition, voicerecognition, and dynamic balance. For some, thissuggested that these tasks are "effortless" for humansprecisely because they were the first, and most important,abilities for early humans. To model human intelligence,one must take into account what the system wasoriginally designed to do. Furthermore, "toy" problemswere being solved efficiently using classic AIrepresentations, but these systems were rigid and brittle:in the face of noise and unpredictable changes (hallmarksof the real world), the systems would failcatastrophically.

Finally, some philosophers exploring the nature ofhuman meaning were asking difficult questions aboutmodels of understanding based entirely on symbolmanipulation. In short, if abstract symbols were givenmeaning only through relation to other abstract symbols,where did those other abstract symbols get theirmeaning? To avoid infinite regress, some argued,meaning must ultimately be grounded in the only"concrete" processes available: interaction with theworld.

2.2.2. Additional Themes in EmbodiedCognition Theory

Along with an increased appreciation for the role ofthe environmental context, the body, and the perceptualsystem, embodied theories regularly share some otherfeatures. None of these features is strictly necessary for,nor limited to, an embodied view, but tend neverthelessto be common to much of the work.

Dynamics. With an emphasis on real-worldinteraction, real-time behavior becomes an integral partof cognitive theory. More than simply acting "as fast aspossible", there are issues of synchrony between internaland external events, as well as physical limitations ofsignal propagation times to take into account (see Port &van Gelder, 1995, for a review). In many cases,researchers abandon classical representations in favor ofa lower-level, dynamical systems view of the mind andits interaction with the world. Note that this does notnecessarily entail anti-representationalism (Port and vanGelder remind us that "dynamical systems can berepresentational without having their rules of evolution

defined over representations", p.12), nor does itnecessarily preclude the possibility of a symbol-manipulation interpretation of the same mechanism atsome higher level of abstraction.

Emergence. Emergence is difficult to defineprecisely, but with the renewed emphasis onrepresentations distributed across the traditionalbrain/body/world boundaries, finding the locus ofcognitive phenomena becomes more difficult. Ratherthan trying to identify the hierarchically controlmechanism responsible for cognitive activity, theembodied view focuses on identifying dynamic systemsof interaction from which phenomena "emerge". RodneyBrooks’ "subsumption architecture", a robotic designmethodology in which behavior emerges from theinteraction of semi-autonomous sensory systems, is aclassic example I will review in Section 4.

3. A Functional View of the HumanConceptual System

In this section, I pursue the definition of "concept"in greater depth. Finding the available definitions highlyvariable and contentious, I move on to a functional viewof the types of conceptual phenomena we are trying toexplain. The resulting list of phenomena will later beused to judge the explanatory power of the varioustheories (in Section 5).

3.1) Defining "concepts"

A brief survey of the philosophical literature onconception reveals that there is no accepted definition atall; the available options are vague, highly variable, andcontentious. To give an impression of the range, thefollowing list is a small sample of definitions taken fromvarious dictionaries of philosophy:

• "psychologically or logically what we grasp inunderstanding an expression"

• "any abstract notion or idea by virtue of which we applygeneral terms to things"

• "a semantically evaluable, redeployable constituent ofthought, invoked to explain properties of intentionalphenomena such as productivity and systematicity"

• "that aspect of mentality which, ideally, refers to an object,property or relation and specifies some properties of thatitem"

• "an accumulation of related facts and/or processes towhich a symbol (representational or abstract) is assigned"

Any finer-grained definitions than these are evenmore controversial, and too tightly intertwined withparticular theories. It is abundantly clear that there is nodefinition precise enough for rigorous scientific analysis,

5

and any attempt on my part to construct such a definitionwould be misguided. The only reasonable way toproceed, then, is to ignore the intentional definitionsaltogether, and functionally identify which phenomenawe are aiming to describe. In the next section, I havecompiled some of the decisive phenomena that haveguided individual researchers in their exploration of theissue.

3.2) Conceptual Phenomena

The following list presents some of the humancognitive phenomena that are relevant to our debate. Ihave chosen to concentrate on those phenomena whichhave been presented in the literature as the strongestexamples of supporting evidence for either side of thedebate. I will then judge all of the frameworks againstthe entire collection of phenomena, seeing each at its(purportedly) best and worst. I use this approach toachieve some distance from the carefully constructedworld of supporting evidence that each author uses topresent his or her work.

As a result of the selection process, the followinglist does not capture a complete, cohesive view of theconceptual system (if such a thing even exists). Instead,it is a disparate sampling, biased somewhat towards the"fringe" of what might be considered conceptual.Nevertheless, any complete theory of the humanperceptual/conceptual system would need to be able toexplain each and every one.

• Mental simulation of spatial events(searching a room in your "mind’s eye", mentallyrotating an object, etc.)

• Using concepts that are heavily context-dependent

(e.g., "normal", "fair", or "large", as in "largemolecule" vs. "small planet")

• Reasoning with ad hoc categories(e.g., items in the room that are likely to get stolen)

• Understanding, reasoning with, and extendingwords without clear definitions

(e.g., making sense of the sentences "Greg openedthe store", but not "Greg opened the tree". Also,extending understanding of the phrase "open thecoconut" to understand "open the walnut")

• Reasoning with abstract concepts(e.g., comparing the concepts of "justice" and"fairness")

• Reasoning about impossible events

(e.g., reasoning that dragons weigh less on the moonthan on earth)

• Integrating conceptual and physical activity(e.g., finding the most "expensive", or "dangerous",item in a room)

• Rule-based categorization(e.g., using a rule to categorize 323,456,391 as anodd number, despite never having seen it.)

• Subtle perceptual categorization(e.g., distinguishing male and female faces, withoutnecessarily knowing how we do it.)

Also, we need to account for:

• Phylogenetic development: how we developedsmoothly from, presumably, purely perceptual("stimulus-response") creatures

• Ontogenetic development: how (and whether) wedevelop from concept-free or concept-limited infantsto highly conceptual adults

While it’s true that any complete theory ofconceptual activity would need to account for each itemon this diverse list, virtually no work in the field todayclaims to be such a complete theory. Nevertheless, it isworthwhile to bring compare the diverse frameworks inan effort to determine which are at least consistent withthe phenomena.

4. Domains and Theories

In this section, I explore the relevant domains ofresearch, outlining the key issues, individuals, andtheories in each. Needless to say, the domain boundariesare fuzzy; there is a lot of overlap. Due to spacelimitations, I have had to omit potentially relevantdomains of research and individuals. I have chosen oneor two influential researchers from each domain, basedon their importance and on satisfying the goal ofsurveying as wide a range of viewpoints as possible in alimited amount of space. For the most part, I will limitmy review to a description of the work, withoutintroducing evidence (which I will present in Section 5,"Analysis").

I begin with Table 1, which summarizes theinformation I’m about to present: the major domains, keyresearchers, and key terms I’ll explore. Table 1 willhopefully clarify the scope of the analysis, and willprovide a reference point for the later sections of thepaper.

6

Table 1: Key Researchers, Domains, and Theories

Researcher(s) Relevant Interest/Theory Notes

Barsalou Perceptual Symbol Systems “Simulators”. Modal, schematic, dynamic

Fodor Abstract Conceptual Systems Propositional, computational, atomistic concepts

Johnson Image-Schemas Amodal, perceptual, metaphorical, dynamic

Kosslyn Mental Imagery Picture-like, part of a dual system

Lakoff Theory-based Categorization Concepts defined in terms of naïve theories

Rosch Categorization Basic categories, objective categories

Neisser Categorical Perception Culturally-biased “direct perception”

Harnad Symbol Grounding Problem Abstract system, but meaning grounded in perception

Brooks Mobots/Subsumption Architecture Embodied mobots, anti-representational

Beer Dynamical Systems Real-time action, internal states allowed

Piaget Ontogenetic Origins of Concepts Separate conceptual system at age 1.5

Mandler Ontogenetic Origins of Concepts Separate conceptual system from birth

Clark & Grush “Emulators” as Origins of Symbols Phylogenetic origins

7

4.1) Perceptual Symbol Systems

This subsection is rather unique, in that it dealsspecifically with the work of one particular researcher:Larry Barsalou. I felt that it was appropriate to dedicate asection exclusively to Barsalou’s theory because it is oneof the most current, complete, and influential views inthis domain, and because it directly addresses many ofthe major issues in this paper. The remaining subsectionsidentify more general domains of research, spanningmultiple people and theories.

Barsalou’s (in press) "Perceptual Symbol System"framework is perhaps the most comprehensivedescription of a purely perceptual, modal implementationof a conceptual system. In his view, conceptualprocessing involves filtered, schematic perceptualsymbols which are manipulated in the same perceptualregions of the brain dedicated to processing externalstimuli. Specifically, a perceptual symbol is a subset ofthe unconscious portion of neural activity associated witha perceptual state. Thus, all such symbols are "modal",in the sense that they are tied to the specific sensorymodality in which they were originally active. Barsaloumakes the strong claim that these modal symbols aloneaccount for all conceptual activity; that the brain does notsupport any form of amodal conceptual content.Examples of the kinds of content captured by perceptualsymbols include: "tiger", "stripedness", "above","comparison", "anger", etc. He stresses that these modalsymbols are not like conscious mental images, but areabstracted perceptual schemata. Through a process ofselective attention that breaks down and abstractsperceptual traces into symbols, the schemata can come torepresent abstractions like shapes independent oforientation. Further details on the abstraction mechanismare not provided.

In his description of human conceptual processing,Barsalou invokes the concept of a "simulation": adynamic reactivation and recombination of perceptualsymbols in working memory. He stresses the active,productive nature of the perceptual system, in contrast tothe "passive recording device" view. Because theschemata are componential and feature-like, they can berecombined in an infinite number of ways, accounting forthe effectively infinite human conceptual capacity.

In order to include action and affect as elementswith which to ground meaning, Barsalou includesproprioception and introspection as "perceptual systems".Introspection provides the "perceptual" mediumnecessary for generating concepts referring to cognitiveoperations (e.g. "comparison") and emotional states.

Barsalou’s system is much like a perceptual re-castof a classical symbol-manipulation approach. The realdifference, though, is that the symbols remain modal;instead of transducing perceptual input into amodal formsfor processing in other parts of the brain, Barsalou

contends that traces merely need to be "filtered down"into a schematic form, and processed entirely within thesensorimotor system.

4.2) Philosophy

Not surprisingly, the oldest tradition in conceptresearch comes from the domain of philosophy. Muchwork is still at least partly philosophically-oriented,perhaps because the paucity of clear scientific evidenceleaves much to conjecture. In this section, however, I tryfocus only on "classically" philosophical arguments.Even so, it would be easy to get bogged down in theregress of thousands of years of philosophical debate;such a review could easily fill volumes. My goal here issimply to touch on two directly relevant issues: thephilosophy of propositional concepts as represented byJerry Fodor, and the philosophy of meaning, asrepresented by the work of Steven Harnad and MarkJohnson.

4.2.1) The Philosophy of Conceptsas Abstract Symbol Systems

I take this opportunity to briefly revisit the views ofadvocates of disembodied concepts, who, for reasonsdiscussed earlier, appear underrepresented in this paperas a whole. I use Jerry Fodor’s (1998) views as a modelbecause he represents one of the most outspoken andinfluential advocates of the propositional viewpoint.

In brief, Fodor argues adamantly for a view ofcognitive activity as computation. The style ofcomputation he advocates is classic, Turing-style symbolmanipulation. The goal of his work is an "atomistic"theory of concepts: one in which concepts are distinct,discrete entities with precise modes of interactiondefined. Importantly, Fodor stresses that the structure ofcomplex mental representations determines the kinds ofpossible interaction allowed. Promoting his view that"mental representation is a lot like language" (p. 25), heplaces perhaps more emphasis on the syntactic structuresthan on the atomic concepts that fill them. Propositional-style structured representations carry most of theinteresting content in this view; concepts themselves aresimply arbitrary atoms to be bound to the availablestructure slots.

Fodor is well known for seeking what he terms a’language of thought’ with combinatorial syntax andsemantics. The propositional content (i.e. meaning) of asentence in this language corresponds to the function thatdetermines, for any possible situation, whether thatsentence is true or false. As for the perceptual nature ofconcepts, he explains that his framework, and any like it,"demand mental representations that aren’t images" (p.9).

8

4.2.2) The Philosophy of Meaning

Ultimately, any complete theory of humanconceptual ability is necessarily also a theory of meaning;after all, concepts function primarily as units ofunderstanding. While the philosophy of meaning is anenormous topic, I am choosing to concentrate on onespecific, particularly relevant, aspect known as thesymbol grounding problem. The symbol groundingproblem is relevant to our debate because it represents anattempt to bridge the gap between conceptual structureand function. The central issue is how mentalrepresentations become endowed with meaning.

To see the significance of the problem, I turn toSteven Harnad’s (1990) analysis. Harnad describes aclassical, propositional style system of abstract symbolmanipulation, and asks: how is the meaning of arbitrarysymbols to be grounded in something other than justmore meaningless symbols? Harnad describes theproblem as tantamount to trying to learn to speak Chineseusing only a Chinese/Chinese dictionary: a "merry-go-round" of replacing symbols with symbols, none ofwhich hold any more meaning than the previous ones.

Harnad’s solution is to retain a very classical viewof conceptual symbol systems (consisting primarily ofsymbol strings), but to ground the symbolic system in anon-symbolic one. The non-symbolic system iscomposed of "iconic representations", which are sensoryprojections of objects and events, and "categoricalrepresentations", which are feature detectors that pick outthe "invariant features of objects and events" from theirsensory projections. The argument is one that is held invarious forms by most researchers in the field: thatultimately, only real objects, events, and actions in theworld can hold true meaning for humans. By proposing amediating system between the world and the abstractsymbol system, Harnad preserves many of the basicfeatures of the classic, disembodied view, whileproviding a mechanism for grounding meaning in such asystem.

For Mark Johnson, it is unnecessary to work topreserve the classical abstract symbol system. Johnsonagrees that meaning must be grounded in perception andaction, but argues that the conceptual system shouldfundamentally operate in this meaningful domain, ratherthan having meaning attached through associations to adistinct system. Thus, Johnson dismisses the abstract"symbol strings" view and replaces it with a system of"image schemas", amodal but still fundamentallypercept-like in nature. As Johnson’s work is primarilylanguage-oriented, I detail his influential views in thenext subsection.

4.3) Language

Language has played an important role in conceptresearch, because it is in language use that our conceptdefinitions show up most explicitly. The relationshipbetween language and the conceptual system is hotlydebated. On the one hand, we have seen that Fodorargues that mental representations are in many waysisomorphic to language, and tightly bound to it throughshared structural relations. On the other end of thespectrum, Barsalou confers no special status on language.He treats spoken and written language just like any otherperceptual traces; to be abstracted and stored along withother perceptual symbols.

Mark Johnson (1987) falls between these twoextremes, stressing that linguistic meaning is just onetype of meaning. He cautions that looking at meaningthrough sentences may mislead us to consider onlypropositional-style theories. Nevertheless, he looks tolanguage, or more specifically the use of language, as theprimary source of evidence regarding the nature of thehuman conceptual system. Johnson points to pervasiveuse of metaphor in language, and suggests that thisphenomenon is an indication of the fundamentallyanalogistic nature of concepts. This is an issue I willreturn to in the Analysis section, when I considerevidence.

For Johnson, there is more to understanding thanjust semantics: understanding is "historically andculturally embedded, humanly embodied, andimaginatively structured" (p. 175). Concepts are notobjective; they derive their meaning only in the contextof experience. Words, in turn, only get their meaningthrough their use in describing the world.

Johnson argues that propositional thought is onlypossible by virtue of a network of nonpropositionalschematic structures derived from bodily experience.These structures, which he terms "image schemas", arenot unlike Barsalou’s perceptual symbols. They are"dynamic patterns" abstracted from perceptual traces,representing concepts like "containment" and "force".Image schemas are not as rich as an image, nor asabstract as a proposition. Unlike Barsalou, however,Johnson’s schemas can be multimodal or amodal withoutbeing "nonperceptual". In his words, they represent "arecurrent pattern, shape, or regularity in ongoingactivities" (p. 48).

4.4) Mental Imagery

Alongside the perceptual schemas of Barsalou andJohnson, it is necessary also to briefly explore theconsiderable literature on "mental imagery". With thisterm, I’m referring to the conscious experience of fairlycomplete, percept-like (e.g. "picture-like") traces in theabsence of corresponding external stimuli. There is

9

considerable debate about the significance of the role ofsuch phenomena in conceptual processing. Perhaps thename most closely associated with mental imagery in thelast two decades is that of Steven Kosslyn (e.g., Kosslyn,1976). Kosslyn is a strong advocate of "offline"perceptual activity, but he does not claim that theprocessing of "pictures" can provide a foundation for allof human conceptual activity. Instead, he supports adual-system view, in which an imagistic componentinteracts with a more traditional propositional system.The difficulty, of course, comes in having to explainprecisely how these systems interact: how theycommunicate, how they "divide up" the cognitiveworkload, etc. This kind of dual-code system is not anuncommon view in the field, but there is, as yet, no clearconsensus on the issue of interaction. I explore Kosslyn’smental imagery evidence in greater depth in the Analysissection.

4.5) Categorization

Categorization research is as close as experimentalpsychology comes to the direct study of the nature ofconcepts. There is no consensus on the precise nature ofthe relationship between the two, however. Categorieshave been presented, in one view or another, as: a specialcase of concepts, equivalent to concepts, andoverlapping, but not subsumed by, concepts.Nevertheless, framing concept work in terms ofcategories allows researchers to pursue behavioralexperiments based on category extensions: similarityjudgements, category membership judgements, etc.

In one "realist" view, the world is divided intocategories based on objective features. Rosch’s (1978)foundational work on categorization took a realistapproach, defining an objective set of category levelsacross all humans. The most salient level of abstractionfor a category, for instance, is known as the "basic level",in which members of a category look alike and involvesimilar physical interactions. Such a system would haveimportant consequences for concept research, suggestingthat there is an objectively correct way to structurehuman knowledge. While this view embraces some ofthe major themes of embodiment (perceptual emphasis,and opportunistic use of structure in the world), it departsfrom the notion of context-dependence, and aconsiderable body of evidence has thrown the existenceof objective category levels into doubt.

George Lakoff (1987) suggests that such a staticview of categories, and concepts, is doomed to fail.Categories change according to context, and categorymembership is far from binary. As an example of gradedmembership, consider the seemingly well definedcategory "bachelor", taken to mean "unmarried male".Although it seems like any person will clearly fall insideor outside of this category, it’s nevertheless the case that

most people view James Bond as a "better" example of abachelor than the Pope. Lakoff’s solution to dynamiccategory definition is to embed all concepts in theories.Systematic relationships among concepts are explainedthrough the internal structure and coherence of people’s"naive theories" of how the world works. The naivetheories, of course, are ultimately derived fromexperience. Thus, Lakoff argues, category membershipjudgements come not only from attribute similarities, butthrough reasoning about the causal relations amongfeatures.

Finally, Neisser (1987) takes a more heavilyperceptual approach, invoking the idea of Gibson’s directperception in the context of a theory of categoricalperception. An entire literature on "categoricalperception" exists, concentrating most heavily on verylow level perceptual categories (like colors) (see Harnad,1987, for review), and mostly independent of research onthe kinds of high-level conceptual categorization whichare the focus of this paper. Neisser (and Clark, 1997, andJohnson, 1987) argues that simple "direct perception" isnot sufficient to explain categorization, but thatperception must be taken in the context of culturally-driven cognitive models. Neisser does not commithimself to a strict, perceptual view of concepts likeBarsalou’s, but he does make an important point aboutthe evolutionary genesis of the conceptual system: even ifwe started as perceptual creatures and "worked in"towards an increasingly abstract, and ultimately distinct,conceptual system, we would still expect to be able tolearn about the constraints on the conceptual system bystudying the perceptual system. This brings us to the nextimportant branch of the literature: development.

4.6) Development

Any complete theory of concepts would beconsiderably strengthened by an accompanyingdescription of how the system develops. There are twotimescales to conceptual development: the phylogenetic(evolutionary) scale and the ontogenetic (personaldevelopmental) scale. Once again, the discussion of suchmatters could easily fill volumes, but I will provide onlythe briefest review of the key issues.

Jean Piaget, one of the most influential figures inhuman ontology, adopted a classical, propositional viewof the adult conceptual system. He did not, however,believe in the innateness of concepts, instead treatinginfants at birth as "purely perceptual creatures". His task,therefore, was to explain the "leap" from perception toconception. Through extensive behavioral studies ofinfants, Piaget (1952) concluded that there was noactivity before (roughly) the age of 1.5 years that couldnot be accomplished directly in the perceptual system.After that age, however, infants performed recall taskswhich suggested they were able to represent objects and

10

events in their absence, and thus had accomplished aconceptual level of thought. Piaget thus concluded thatthe distinct conceptual system appeared around age 1.5.He suggested that the origin of the conceptual systemwas related to the internalization of sensorimotorschemas, but was unable to describe the mechanismsmore precisely.

More recently, Mandler (1992, 1997) revisitsPiaget’s views on the matter, suggesting that they havebeen largely unquestioned in the intervening decades.Based on considerable experimental evidence sincePiaget’s time, Mandler concludes that it is difficult toexplain why conceptual development is so long delayed.Although she retains the assumption that the perceptualand conceptual systems are indeed distinct, she proposesthat the two systems in fact develop in parallel frombirth. I will revisit Mandler’s evidence in the analysisportion of this paper.

Recently, Clark & Grush (in press) have attempteda further exploration of the kind of "internalizedsensorimotor schema" Piaget described, in the context oftrying to understand the phylogenetic origins of concepts.In principle, however, much of their description mightjust as easily apply to ontogeny. Based on evidence wewill consider later, Clark & Grush provide a compellingdescription of a developmental process which beginswith pure sensorimotor emulation of external events.Through small, but plausible stages, the emulationprocess gradually becomes increasingly detached fromthe external stimuli. In the first stage, the emulationsimply mirrors the activity in the world. Later, theemulation takes place in the presence of the stimuli, butmay run faster internally and thus "predict" the outcomeof the external events. In this stage, there is regularinteraction between the internal and the external. In laterstages, the emulation can occur in the absence of externalstimulation, a situation which Clark & Grush identify asthe first "true representation". While their explanationends there, I will return later to the intriguing possibilitythat such a mechanism might blend nicely into one of theperceptual schematic views we will analyze.

4.7) Robotics and Computer Simulation

Finally, I consider robotics and computer simulationto play an important role in understanding the nature ofconceptual representation, because they allow us toimplement and test otherwise heavily speculative theoriesin a concrete way. One of the obvious difficulties instudying concepts in human activity is lack of access;brain imaging techniques are not sophisticated enough togive us low-level descriptions of neural representation. Incomputers, however, representation is accessible, if notnecessarily easily interpretable.

An interesting question arises for embodimentproponents when computer simulation is introduced as a

tool. If real worlds, and real bodies, are essentialcomponents of human cognition, what level of detail (ifany) is appropriate for modeling? If our perceptualsystems are the locus of much cognitive activity, wouldresearch with robots be teaching us about generalcognition, or about the specifics of the sensory apparatuswe happen to endow them with? I can’t tackle theseissues in significant depth, given space restrictions.Nevertheless, it seems relatively uncontroversial toconclude that a proponent of disembodied symbolmanipulation would likely be more optimistic about thepossibility of capturing human-like cognitive activity incomputers than a diehard supporter of the embodiedview. This is not to suggest, however, that embodiedtheorists must completely dismiss the role of computersimulation, simply that they must necessarily acceptadditional limitations.

Rodney Brooks of the MIT Artificial IntelligenceLab has long been one of the most outspoken advocatesof a "situated" view of cognitive activity within therobotics community. In his view, computers "havepoisoned our view of what intelligence must be" (1991,p.). As an example of computation-biased design,Brooks points to what he refers to as the "sense-model-plan-act", or SMPA, design that dominates robotics.These systems employ the classic input/output viewdescribed above as one of the identifying features of thedisembodied approach. Brooks takes a dim view of theseapproaches that "over-model"; in his words, "the world isits own best model" (1991,p. 583). Brooks’ own roboticdesigns de-emphasize representation, instead opting toproduce motor commands directly from perceptual input(e.g. working with 2-D visual input, rather than trying tomodel the 3-D world first). When representation isrequired, Brooks prefers "deictic representations": insteadof modeling the world "objectively", his robots modelentities exclusively in terms of their relationship to them(a "self-centered" view).

Brooks’ design methodology, referred to as the"subsumption architecture", reflects most of the majorthemes of embodied cognition: situatedness, intelligencethrough interaction with the world, emergence, dynamics,etc. Brooks’ robots are conglomerations of semi-autonomous sensorimotor subsystems, arranged looselyin hierarchical layers. Layers can encourage, interrupt, oroverride one another, but cannot "communicate" in anyformal way. The complex, intelligent behaviors of therobot emerge from the interactions of sensorimotorsubsystems, without requiring a central ("conceptual")modeling component.

Randall Beer (1995) also explores the theory behindthe construction of situated agents, but focuses moreheavily on seeking explanations in dynamical systemstheory. Beer stresses that the primary goal for an agent is"appropriate behavior at the appropriate time". If real-time behavior is an essential component of embodied

11

action, Beer reasons, dynamical systems theory is themost suitable tool for mathematical description. Beer’swork is largely consistent with that of Brooks, though heis not as vehemently anti-representational. He arguesthat purely reactive systems (ones with no internal state)have no true autonomy; clearly internal states do exist inhumans and other autonomous creatures. The questionfor Beer is how to dynamically couple those internalstates to external states, and how to best makeopportunistic use of structure available in theenvironment.

5. Analysis

In this section, I review the list of conceptualphenomena presented in Section 3. For each, I examineevidence from the domains described in the previoussection, and consider the success of the majorframeworks in explaining the phenomena. Once again,for reasons already discussed, I will focus most of myefforts on examining the perceptually-orientedframeworks, to see if they stand up to the strong claimsmade about them. As we’ll see, in many cases theevidence (for either side) is not nearly as decisive as it isoften made out to be.

Each of the following analyses exposes one or morerelevant issues. Recall, the phenomena were chosenprimarily because they have each figured prominently inarguments surrounding the debate.

Mental simulation of spatial events

Examples: Searching a room in your "mind’s eye", mentally rotating an object, etc.

Review:The analysis of this, and later, phenomena demands

a review of the differences between the types ofperceptual representation involved in the keyframeworks. I take Barsalou’s (in press) perceptualsymbol systems, Johnson’s image schemas (1987), andKosslyn’s mental imagery work (1976) as examples ofthe most detailed efforts at describing concept-likeprocessing in perceptually-based systems.

In terms of the level of perceptual detail involved,Barsalou’s perceptual symbols fall approximatelyhalfway between Kosslyn’s "picture-like" mental imagesand Johnson’s highly abstract image schemas. Kosslyn’simages are like percepts, in the sense that they haveresolution, scale, etc. (recall, however, that Kosslyn doesnot claim that such a system can alone account for allconceptual activity). Barsalou’s perceptual symbols areinextricably tied to a particular perceptual system, but are

abstracted through a filtering process, and therefore notlike conscious perceptual experiences. Instead of"seeing" stripes on a mental image of a tiger, Barsalou’sframework instead describes the activation of a"stripedness" symbol in conjunction with a "tiger"symbol in the visual cortex. Johnson’s image schemasare amodal, and even further abstracted. He focusesmuch of his argument on a few very general schemata,like "containment", which, he believes, underlie mosthuman understanding. While he ultimately believes thatall conceptual activity must be grounded in suchsensorimotor representations, it’s not clear how detailedthe schemata can get, nor what additional mechanismsmight be needed to provide a complete account ofconceptual activity.

Discussion:The following is a sample of some of the most

compelling and influential empirical evidence relevant tomental imagery:

1. Kosslyn’s "image tracing task" (1976) askedsubjects to visualize a rabbit beside either an elephant ora fly. Subjects were then asked questions about detailedfeatures on the rabbit’s face. In the "fly" condition, inKosslyn’s interpretation, the rabbit "appeared" large inthe mental image, and thus subjects were able to answersuch questions quickly. In the "elephant" condition,subjects showed delays appropriate for having to "zoomin" on a picture frame in order to "see" the rabbit’sfeatures.

2. In mental rotation tasks with abstract figures(e.g., Shepard & Metzler, 1971), subjects show delaysproportional to angle of rotation, as if they are rotating a"real" image spatially.

3. The V1 area of primary visual cortex has beenshown to become active during mental imagery (Kosslynet al., 1995).

In a domain in which compelling evidence is hardto come by, this is certainly strong evidence for somenon-propositional, sensorimotor component to highercognition. Proponents of propositional views argue thatthe experience of a mental image is a constructed by-product of some types of conceptual activity. However,they have trouble explaining the differential delays in theimage tracing and mental rotation tasks: if the concept of"rabbit" has features associated propositionally, thosefeatures should be equally accessible, regardless of thecontext of the "scene".

Within the realm of perceptual explanations, theimage tracing and mental rotation tasks appear to beevidence for a full imagery component (à la Kosslyn)rather than perceptual schemata (à la Barsalou), whichabstract out issues of scale and resolution. Indeed, withrespect to this evidence, the schematic approaches have

12

no obvious advantage over the propositional ones. Thisleaves the perceptual approach with a dilemma.Abstraction is valuable in that it provides for "conceptcoherence", our experience of concepts as distinct,coherent entities. Also, it supports easier explanations ofobject identification (e.g., matching the concept of dog tothe wide variety of perceptual conditions under whichdogs are viewed) by abstracting away the details in whichmuch of the perceptual variability lurks. However, byabstracting out issues like scale and resolution, theschematic theories need to reintroduce a mechanism bywhich perceptually detailed scenes are reconstructed toexplain data like Kosslyn’s. This leaves them with thesame basic problem as the propositional theorists.

Furthermore, though this phenomenon of mentalsimulation does support a perceptual component ofcognition, it does not necessarily imply that the entireconceptual system is sensorimotor-based. Kosslyn’sdual-code (imagistic and propositional) system can alsoaccount for this evidence, though not without raisingadditional issues. The difficulty with such a system liesin giving an account of the apparently seamless interfacebetween the two. Consider, for example, a task thatcombines perceptual and (classically) conceptualinformation, such as mentally searching your office spacefor all items worth more than $50. There is inevitablyoverlap in the kinds of entities each system can represent;it is a hard task indeed to explain how these redundantrepresentations are kept consistent with each other, andno satisfactory account currently exists. I pursue thisissue in greater depth, later in this section.

Using concepts that are heavily context-dependent andReasoning with ad hoc categories

Examples: context-dependent concepts: "normal", or "large" ("large molecule" vs. "small planet") ad hoc categories: items in the room that are likely to get stolen

Discussion:These two phenomena raise many of the same

issues: both stress the ability of the conceptual system toconstruct meaning dynamically. The term "ad hoc"refers to categories which are defined on the spot, butnevertheless are subject to fluid reasoning just like anymore "permanent" category (e.g., "animal"). Barsalou,Johnson, and Clark & Grush all focus explicitly onconcepts as representations that are dynamicallyconstructed in working memory. This is in contrast to thetraditional propositional approach, which characterizesthe conceptual system as a (relatively) static structure inlong term memory.

Context dependence has proven to be a fundamentalcharacteristic of linguistic meaning. For all concepts,different aspects are highlighted under differentcircumstances. For example, the concept of "mammal" isoften presented as an example of a well-defined "naturalkind", and is thus presumably less subject to the effectsof context. Nevertheless, using a term like "mammal" todescribe my opponent in the context of a sporting eventwould likely to be taken to be derogatory. Would afixed, propositional concept of "mammal" thereforecontain every feature which might possibly be relevantunder any circumstance? To take the argument further,consider the ad hoc category given as an example above.I can reason about the items in the room that are likely toget stolen as a unitary concept, but do they reallyexplicitly share that feature in their long termrepresentation? In other words, does every item in mypossession carry a "likelihood- of-being-stolen" feature,as well as a "likelihood-of-being-destroyed-during-a-flood" feature, and so on? There would seem to be aninfinite number of such features, if we don’t restrict theirlevel of detail.

So clearly, feature lists cannot tell us everything weneed to know about concepts: concept meanings musthave a dynamic component, presumably related to thereasoning processes in the context of which we generateunderstanding. Barsalou (1987) proposes such a dynamicview:

Rather than being retrieved as static units from memoryto represent categories, concepts originate in a highlyflexible process that retrieves generic and episodicinformation from long-term memory to constructtemporary concepts in working memory. Because thisconcept construction process is highly constrained bygoals, context, and recent experience, the same conceptis rarely if ever constructed for a category. (p. 101)

Barsalou's “simulations”, Clark & Grush's“emulations”, and Lakoff's “theory-based concepts”represent three different versions of this dynamic,constructivist view. Barsalou differentiates the long-termand short-term components by denoting the former as"simulators", the latter as "simulations". Simulatorssimply retain the core schemata in memory, but ourexperience and use of concepts comes only when varioussimulators are activated and combined in dynamicperceptual simulations. In Lakoff's version, the longterm portion is the theory structure, and the active portionof a concept is a type of dynamic reasoning, based onthose theories (e.g., a theory about what kinds of thingsthieves are likely to take, etc.). Clark & Grush'semulators, though not presented in the context ofexplaining high-level cognitive activity, do suggest thatthe dynamic emulation of basic sensorimotor patternsmight be the core around which early symbols areconstructed. Such emulations would necessarily

13

incorporate context, as they originate undercircumstances in which they must be able to continuouslyinteract with the corresponding perceptual input.

While it is clear that the context sensitivity ofconcepts is commonly a component of embodied,perceptual theories, we need to consider whether they arereally any better at explaining it. Certainly, neitherperceptual nor propositional systems preclude thepossibility of taking context into account. The argumentmade by the embodied community is that the perceptualsystem can’t help but take context into account. This maybe true during the initial creation of a perceptual symbol(in fact, explaining how the "core" schemata areextracted from the context is problematic), but once asymbol system is in place, and symbols are beingreactivated to construct a conceptual scene, it’s not clearhow perceptual symbols have any more claim to context-dependence than abstract, conceptual symbols.Barsalou’s account of a perceptual symbol for a conceptlike "larger" is as atomic as any: it simply binds twoother perceptual symbols as arguments, as if it were apropositional system. Other perceptual theorists (e.g.,Johnson) hint at the increased flexibility that a perceptualrepresentation provides, but offer no detailed explanation.Context sensitivity may be impossible to describe in thebrittle, definitional style of propositional system, but thisdoesn’t necessarily hold for fuzzy or probabilistic formsof abstract concept systems. Basically, there is currentlyno account of graded concepts in either system thatclearly outperforms the other. I would suggest the claimsregarding the handling of dynamic concepts arise morefrom a bias which reflects the focus of the research ineach approach, rather than the actual capabilities of therepresentations.

Understanding, reasoning with, and extendingwords without clear definitions

Examples:We make sense of the sentences "Greg opened the

book" and "Greg opened the store", but not "Greg openedthe tree". How do we define "open"? What is anacceptable object to "open"? Furthermore, note that"opened the tree" could make sense in a very limitedcontext, like "tree surgery".

If I understand the phrase "open the coconut", I canalso extend my understanding of the term "open" tounderstand the phrase "open the walnut", even if I havenever seen a walnut opened.

Discussion:The first example emphasizes two facts: (1)

concepts are not always easy to explicitly define, and (2)meaningfulness is a continuum, a phenomenon whichrecalls the previously discussed issues of graded categorymembership and context-dependence. With regard to the

first point, Barsalou (1987) cites evidence that many verycommon terms are surprisingly difficult for people todefine. Again, these kinds of examples are a realproblem for propositional systems which suggest thatconcepts are defined by fixed sets of necessary andsufficient conditions, or invariant feature lists. Suchsystems seem too rigid to support the obviously fuzzy,unstable nature of concepts.

The perceptual-schematic explanation for thisexample contends that any object which can besuccessfully matched against a "container" or "closedform" schema can be understood as the object of "open".The "definition" of open itself is a dynamic schemawhich actually represents the opening of a closed formschema. The term never needs precise definition in anyother form. Presumably, "tree" is a poor object for "open"because it provides a poor perceptual match (unless the"knife-wielding tree surgeon" completes the picture, inwhich case it is marginally acceptable). While thisdescription is intuitively appealing, it relies on aperceptual matching process which has not beenadequately defined in any theory to date. It is not self-evident that extracting and/or comparing some coreperceptual "dogness" from the incredible range ofperceptual traces which invoke the idea "dog" is anyeasier than the extracting the corresponding abstractsymbol. Nevertheless, perceptual pattern matching isclearly something that the human brain is good at, soproponents of the perceptual view at least have someindirect evidence that they may yet find themechanism(s) they need.

Now let us reconsider this phenomenon of fuzzydefinitions from the perspective of a non-perceptualsymbol system. As established in the previous section, alist of discrete features complete enough to representevery possible characteristic (e.g. "book:can_be_construed_as_container") is implausible. Theremust be a dynamic, inferential step involved in theconstruction of meaning that establishes these kinds ofcharacteristics; it is the nature of this step that is thereforecentral to our debate.

One possible propositional explanation for such"fuzzy" language understanding would be exemplarbased: you don’t understand "opened the tree" becauseyou’ve never heard "open" used in that context before.Nevertheless, it’s clear that we can easily extend themeanings of words in novel, but systematically relatedways. The second example above is an instance of thisphenomenon: "open the walnut" makes sense eventhough I have no obvious definition to apply to this novelcircumstance.

Mark Johnson’s (1987) answer to this secondexample is that systematic meaning comes frommetaphor. Johnson believes that systematic, conventionalmetaphorical systems underlie nearly every domain ofhuman understanding. A classic example he explores in

14

detail is that of theories (an abstract concept) understoodin terms of buildings (a concrete one). Theories havesupport, foundations, form, strength, framework, etc.Thus, meaning is grounded in sensorimotor experience,and nearly all such grounding occurs in the context ofmetaphor. The strongest evidence for this view, Johnsonclaims, is linguistic. First, the metaphors show up in thelanguage we consistently use to describe concepts.Second, he argues that the different meanings ofpolysemous words can appear to be very different on thesurface, but can almost always be related by someabstract, metaphorically-extended image schema.Metaphor thus provides a high-level explanation for whysome meanings are better than others (better and worsematchings between the domains), how we can come toaccept new uses for a word, and how we can ground themeaning of more abstract concepts in more directlyphysical ones.

To relate this explanation back to our debate, weneed to consider whether this metaphorical system mustbe implemented in perceptual schemata, as opposed to apropositional format. Ostensibly, the answer is "no": thedetails of metaphorical matching are not provided in suchgreat depth as to preclude a probabilistic, propositionalimplementation. Nevertheless, Johnson argues the answeris "yes", and cites the symbol grounding problem as thereason. Ultimately, he argues, the meaning of abstractsymbols must be grounded in something other than otherabstract symbols; they must be grounded in experience.Since the sensorimotor system is most closely connectedto experience, an appeal to parsimony draws Johnson tothe conclusion that it must be perceptual schemata whichunderlie concepts. I consider this argument in greaterdetail with respect to the next phenomenon: abstractconcepts.

Reasoning with abstract concepts

Example:Comparing the concepts of "justice" and "fairness"

Discussion:The concept of "justice" has no obvious direct

physical instantiation which might serve as the core of aperceptual symbol. Barsalou, who gives the mostdetailed account of perceptual schema formation,describes it as a process of abstracting/filtering to extracta subset of the non-conscious neural activity associatedwith the related percept. The problem, in this case, is thatthere is no obvious percept to abstract from. Clearly,there are associated ideas like balance, which plausiblyhave perceptual instantiations. However, Barsalou’sframework demands that all aspects of the concept arerepresented in this format, which means being able todistinguish the subtle connotations differentiating, for

instance, "justice" and "fairness". Barsalou (in press)attempts to explain abstract concept handling through afew examples, but the examples are carefully chosen, andhe admits that the explanations are weak in their currentform.

A similar argument can be leveled against Johnson’s(even more abstract) image schemas, though he concedesamodal content and is less specific about whetheradditional mechanisms are needed to cover the wholerange of conceptual activity. It is intuitively possible thatthe metaphor-based system is able to create enoughassociations to properly capture the subtleties Barsalou’ssystem seems to lack, but there is no strong evidence tosupport this claim.

The argument from the propositional viewpoint isthat the abstract/concrete dimension is almost irrelevant.Neither is harder to generate or explain than the other.This claim, however, fails to acknowledge the issuessurrounding the symbol grounding problem. To say thatit is easy to bring a new abstract symbol into existenceand attach the label "justice" to it is one thing; to providean account for how the meaning is truly established isquite another. If associations to other abstract symbolsare not enough to account for meaning, and thus anexperiential, perceptual component of the conceptualsystem is actually necessary, then at best, thepropositional account is no better off than Johnson’smetaphor-based account. In fact, it may be worse, as itmay need a more sophisticated explanation of theperceptual-to-conceptual transduction process than theimage schema framework, as image schemas are alreadyinherently perceptual. Since some filtering ortransduction is needed in both cases, however, it’s notclear how strong an argument this really is.

In short, the propositional side of the debatetypically claims victory over this issue of abstractconcepts, but on closer inspection it appears to be adifficult phenomenon for anyone to explain.

Reasoning about impossible events

Example:Reasoning that dragons weigh less on the moon

than on earth.

Discussion:This kind of example is precisely what the

propositional systems used in traditional AI are good at.Although dragons are mythical, they are mythicalanimals, and thus inherit the default features of animals.In turn, animals inherit the feature of "mass" fromphysical objects, and the fact that objects with massweigh less on the moon can be conveniently captured in arule. Of course, in a disembodied system, the "reality"of the situation described is, in some sense, irrelevant.The world consists of the contents of the propositions,

15

and whether the propositions are labeled "dragon","huju", or "rock", their ontological status can be reducedto simply another recorded feature.

From the embodied view, ontological status isconsiderably more salient. Nevertheless, a perceptualunderstanding of the dragon sentence is not problematic,as one’s experience with pictures of dragons could easilysuffice to create a perceptual symbol like that of anyother animal. Barsalou stresses the productive nature ofthe perceptual system: symbols may be recombined in aninfinite number of ways. Inheritance is captured in theprocess of selecting what kinds of simulators are suitableto be included in what kinds of simulations: a "dragon"can participate in a simulation suitable for "animal", orindeed, "object with mass". There is no detailed accountof the process of storing or determining what constitutesan acceptable participant in a particular simulation,although it presumably is based in perceptual experience.

As a footnote, it is interesting to ponder apotentially deeper issue, concerning completely novelphysical entities, like "a new color, unlike any existingcolor", or a situation in 5 spatial dimensions. In theory, acompletely abstract propositional system with nogrounding in experience should have no more difficultydescribing a 5-dimensional event than a 2-dimensionalone. The former, though, seems distinctly more difficultfor human beings to conceive of.

Integrating conceptual activity andphysical activity

Example:Finding the most "expensive", or "dangerous",

item in a room

Discussion:There is considerable evidence that physical activity

interacts with many aspects of conceptual activity. Vanden Bergh, Vrana, and Eelen (1990) found that whensubjects were forced to indicate a preference betweensets of visually presented letter pairs, subjects with typingexperience showed a clear preference for pairs that wereeasier to type, whereas non-typists showed no clearpreference. Glenberg (1997) cites evidence that memoryfor commands that were acted out is better than memoryfor the verbal description of the commands. Finally,Klatzky, Pellegrino, McCloskey, and Doherty (1989)demonstrated that subjects were able to make semanticsensibility judgements of phrases describing meaningfuland nonsensical physical acts (e.g. "aim a dart" or "closea nail") faster when the subject was performingappropriate physical hand motions (e.g. pinching orclenching).

The embodied, situated view holds that interactionbetween the physical and conceptual is the defaultsituation in human cognition. It is argued that perceptual

symbol representations can be matched againstperceptual/motor experience more easily than performinga percept-to-concept transduction. However, the detailsof such a matching system (as argued above) are soseriously lacking, that it’s not clear that matchingperceptual schemata is necessarily easier than extractingand matching propositional representations. "Expensiveitem", for instance, is an abstract concept; it’s not obviouswhat an appropriate perceptual schema is, nor how itcould be perceptually matched against a visual scene.Once again, it seems the data that nobody is able toaccount for greatly outweighs the advantages any onerepresentation claims over another.

Before moving on, however, I would like toconsider two additional pieces of experimental workwhich relate sensorimotor experience to conceptualactivity, and provide some context for their possibleintegration. First, Rodney Brooks’ work on embodied,situated mobots (described in section 4.2.7) has shownthat, for some tasks at least, action in the world can beperformed more efficiently (in less time, and with lesscomputational power) if internal modeling and planningare minimized. As an example, Brooks (1991) describesa DARPA project aimed at building an autonomousvehicle that could drive down a road given visual inputfrom the driver’s viewpoint. The first attempts followedthe classic sense-model-plan-act methodology,attempting to build a 3-dimensional internal model of thescene which a planning system would then manipulate toproduce the appropriate output. This attempt failedbecause extracting the model proved too difficult, and thecomputation took too long. Two other teams, workingindependently, subsequently came up with successfulsolutions that worked directly from the 2-D picture inputto continuously update the servo output, withoutadditional modeling or planning. The system was thusable to run in real time, and the difficult problem of 3-Dmodeling was averted. One could argue that evolutionwould likely take the same shortcuts wherever possible,avoiding transduction to non-sensorimotorrepresentations.

The second relevant piece of experimental workcomes from Clark and Grush (in press), who argue thatemulation of sensorimotor activity is already necessary toexplain some motor phenomena, and offer it as a possibleexplanation of the evolutionary origin of conceptualphenomena as well. They present the case of smoothgrasping motion, the study of which has shown thatproprioceptive feedback is not fast enough to account forobserved performance. To explain how this "impossible"level of performance is achieved, they describe anemulation system which dynamically models the systemstate faster than the real world input is received, thus"predicting" the future state of the system and respondingappropriately. They then proceed to construct an elegantdescription of how such a system might become the

16

evolutionary foundation of mental representations. Theydescribe how sensorimotor emulation concurrent withactivity would be an easy phenomenon to explainneurally, and how such emulation may have graduallybecome desynchronized from the external stimuli inorder to provide these "predictions" which result ingreater performance. The next step beyonddesynchronization is complete detachment fromsensorimotor stimuli, in which the organism can simulatethe activity without performing it. This, too, isevolutionarily sound, in that it provides "practice" formaking smooth, delicate motions. Such offlinesensorimotor emulations could then form the foundationof perceptual symbols which could become (at least partof) a conceptual system.

What we get from these two examples is a sort of"proof of existence" that pure sensorimotor activity couldpossibly account for low level conceptual (or at leastconcept-like) behaviors. Unlike most of the otherarguments we have considered, these descriptions arefairly complete, and testable. Of course, neither attemptsto address the high-level concepts that have been theprimary focus of this paper. Nevertheless, they at leastsuggest an approach that may be able to establish quitedecisively whether the theory of embodied, perceptualcognition is constructed on a firm foundation.

Rule-based categorization

Example:Use of a rule to categorize 323,456,391 as an odd

number, despite never having seen it.

Discussion:Rule-based categorization could be described as a

case of quintessential propositional processing. Eventhough not all propositional theories rely exclusively onrule-based processing, all such systems (by definition) dopermit the existence of explicit rules.

Advocates of the perceptual view argue that there’sno need for "rule-like" behavior to be driven by explicitrules; countless examples of neural networks have shownthis to be true (Rumelhart & McClelland, 1986). Aperceptual simulator could easily capture the feature "lastdigit is 1,3,5,7 or 9" (all of the components of which haveperceptual interpretations).

The only problem for Barsalou’s framework, inparticular, is explaining the fact that the same rule mustbe redundantly stored in both the visual and auditorysystem. Barsalou’s insistence on placing all contentwithin particular modalities demands that the concept of"oddness" be separately represented for each. This seemsto be a very serious, and unnecessary, breach of the ruleof parsimony. Johnson’s acceptance of the multimodaland amodal (though percept-like) nature of someconceptual content avoids this implausible consequence.

Subtle perceptual categorization

Example:Distinguishing male and female faces, without

necessarily knowing how we do it.

Discussion:This issue is partially a question of what’s

accessible to consciousness. It is certainly possible thatthere is a very complex set of criteria that a propositionalsystem could apply probabilistically (and non-consciously) to differentiate male and female faces.However, the "hard work" of detecting and extractingsuch subtle features is clearly being done in theperceptual system (or in the percept-to-concepttransducer). Thus, even a propositional explanation mustadmit that the purely abstract part of the system is doingvery little of the work in this example.

As for the schemata-based systems, they savehaving to describe the additional transduction step, butare really no closer to describing the mechanism throughwhich such subtle categorization is performed. They lacka principled explanation of how some of the abstractedfeatures (like "stripedness") are accessible while others(like "maleness") are not, especially since Barsalouascribes the task of feature abstraction to the (largelyconscious) attentional system.

Full image-based systems like Kosslyn’s might havea better explanation through prototyping/exemplartheories in which examples of male and female facesultimately underlie the ability to differentiate. Such aprocess could be inaccessible to consciousness, and freeto derive unlimited complex feature combinations.Ultimately, however, the nature of a general, perceptualsimilarity comparator has proven so difficult tocharacterize that it’s unclear whether the exemplarapproach is actually any closer to an answer. Once again,we have a case where the data is apparently difficult forall sides to explain.

One additional piece of evidence relevant to theissue of categorization comes from Glaser’s (1992)review of some reaction time experiments whichmanipulated stimulus modalities. In particular, subjectsin one experiment were able to name the superordinatecategory of a stimulus faster if the stimulus waspresented as a picture rather than as a word. While thisexperimental design was plagued, in my opinion, by anumber of possible confounds (for instance, the verbalresponse might have been delayed by interference withthe stimulus in the word case), the results suggest thatimage processing is not considerably slower than wordprocessing, which is what might be expected in apropositional system.

17

Development

Examples:Phylogenetic development; how we developed

smoothly from, presumably, purely perceptual ("stimulus-response") creatures

Ontogenetic development; how (and whether) we develop from concept-free or concept-limited infants to highly conceptual adults

DiscussionIn section 4, I provided a very brief outline of the

theories of concept development proposed by Piaget andMandler. Piaget assumed that the perceptual systemprecedes a separate conceptual system in infantdevelopment. Mandler, however, cites considerableevidence of symbolic behaviors in infants less than 10months of age (e.g., Baillargeon, De Vos, & Graber,1989, shows recall by infants as early as 8 months ofage), and concludes that the conceptual activity co-evolves along with perceptual activity. This might beinterpreted as indirect support of close ties betweenperceptual and conceptual activity, though Mandlerherself nevertheless assumes (without directly arguingthe point) that conceptual activity is the responsibility ofa distinct system. She admits having no evidencesuitable for explaining the interface between perceptualand conceptual activity.

As described above, Clark & Grush (in press)provide a particularly elegant and plausible description ofthe phylogenetic origins of mental representations, whichmight just as easily apply to ontogeny as well. Whiletheir description places the first true mentalrepresentations in the sensorimotor system, this of coursedoes not preclude the possibility of the subsequentcreation of a distinct conceptual system. Any suchaccount, however, would need to explain whatevolutionary drive caused the creation of this separatesystem. As yet, no one has offered such an explanation.

With respect to development, then, I suggest thatearly symbolic thought seems likely to be based in, or atleast strongly influenced by, sensorimotor activity. Thisprovides some (albeit weak) evidence in support ofexperiential grounding, and makes the job of describing adistinct conceptual system somewhat harder.

6. Conclusions

The analysis above has, out of necessity, beenlimited in scope. Nevertheless, consideration of themajor arguments and evidence from each of thesedomains does lead to several conclusions. Perhaps themost obvious conclusion is that in every aspect of thisdebate, the number of unanswered questions so greatlyoutweighs the available evidence, that it is perhaps

premature to expect a more thorough analysis to producemore definite answers.

Despite these difficulties, I believe several reliableconclusions have come to light in the course of thisanalysis:

• Many of the phenomena, despite being presented asarguments for one side or the other in the literature, are infact difficult for any current framework to explain. Thesearguments often appeal to an intuitive sense of the"naturalness" of a representation, but their advantages arenot further elaborated, and they do not constitute decisiveevidence. For instance, rule-based categorization ispresented is support of propositional representation,when it presents no deeper challenge for perceptualframeworks. On the other hand, context sensitivity ofconcepts is presented as supporting evidence forperceptually-oriented systems, when some propositionalapproaches seem no farther from a solution.

• There is considerable evidence that perceptualrepresentations play some role in higher cognition. Atthe very least, it seems misguided to expect to describethe human conceptual system without taking the natureof the sensorimotor system into account. Even if theconceptual and perceptual systems are separable, onewould expect the nature of the perceptual system toprofoundly influence the structure of the conceptualsystem.

• The evidence from the image tracing task and mentalrotation tasks seem to support the conclusion thatconsiderable perceptual detail (e.g. realistic spatialorientation and scale) is potentially available in mentalimagery, a fact which may be troublesome even for thecurrent generation of perceptual schematic theories.Clearly, different levels of abstraction come withdifferent tradeoffs, and it appears that no single level iscapable of providing all of the answers.

• The argument for entirely modal systems seems tointroduce unnecessary restrictions and challenges that acombination modal/amodal system need not face.Explaining multimodal phenomena without unnecessaryduplication of representations, and without placing anyconceptual content outside of modality-specific brainregions, seems implausible and unnecessary. There islittle reason to believe that reduced versions of directperceptual input (a la Barsalou) can truly capture all thenuances of meaning, and represent abstract concepts thathave no obvious perceptual correlate. By allowingamodal conceptual content, percept-like representations(e.g. image schemas) can bridge modalities and engage inother forms of associations (i.e. analogistic) withoutgiving up the considerable advantages of keepingrepresentations "close to the sensorimotor surface".

18

• Propositional systems which describe concepts asstatic structures defined by necessary and sufficientconditions, or by invariant feature lists, are doomed tofail. Overwhelming evidence suggests that concepts aredynamic, fuzzy, and highly context-dependent.

• In light of the above, an atomistic view of conceptsas clearly individuated symbols may be in jeopardy aswell, although the evidence is not overwhelming. Itshould be noted, however, that perceptual accounts likeBarsalou’s are just as subject to this criticism as manypropositional systems.

• The symbol grounding problem constitutes aconsiderable obstacle for any propositional systemswhich reject a close relationship to perceptualrepresentations. There are simply no convincingarguments for how to endow a conceptual system withreal meaning without invoking a tight coupling betweenconcepts and the experience of interacting with theworld.

In light of these conclusions, how do we decidewhether it is useful and meaningful for cognitive scienceto distinguish between concepts and percepts? I wouldargue that both modal and amodal representationsdirectly play the role of conceptual content: there aresome things which seem too abstract to describecompactly in modal form, and some things which are soeasily described in modal form that it would be a breachof parsimony to translate the content to anotherrepresentation.

It may, however, be more appropriate to de-emphasize the impact of such a statement. "Amodalness"comes in degrees; there is a considerable range ofpossible "distances" from the sensory apparatus. Toexpect to be able to draw a hard line between "modal"and "amodal", let alone "perceptual" and "conceptual",might be an oversimplification of matters. To posit a"dual-representation" system suggests two distinctrepresentational formats with a definite transductionmechanism between them. This view demands anexplanation of the nature of the transduction, as well asthe developmental origins of the distinct conceptualsystem. Perhaps this is not the appropriate view. Theevidence suggests a wide range of representations, at awide range of abstraction. It seems plausible that thisrange might represent a continuum, in whichrepresentations retain as much of their perceptualcharacter as possible for the task at hand. Multiplerepresentations of the same entity may be activesimultaneously, tightly coupled but filtered to permitdifferent kinds of manipulation.

I have concluded that it is highly unlikely that thereis not a perceptual component to higher cognition. The

evidence is not as clear that it is necessary to have apropositional component that can be interpreted as therule-based manipulation of unambiguous, discretesymbols. I have suggested that even if such a systemexists, it would still need to be grounded in the meaningavailable in the perceptual/motor system. Together, thisevidence does seem to suggest a direction: perhaps anincreased appreciation for embodiment and the role of theperceptual system is appropriate, if only for pragmaticreasons. Such an approach could be conservative, in thesense of beginning with reliable, grounded perceptualproperties and positing more sophisticated machineryonly when it is needed.

Let me emphasize that I do not arrive at thisconclusion because I have been so impressed by theexplanatory successes of the existing, perceptually-oriented frameworks. Instead, it is the appeal of the"firm ground", the potential to actually approach acomplete description of the low-level foundations ofsymbolic thought through work like that of Brooks,Beers, and Clark & Grush. To describe the completehuman conceptual system from the ground up is, to put itmildly, a Herculean task. Nevertheless, the degree towhich high-level concept work is based in intuition andspeculation is disturbing. The advantage of theembodied, perceptual approach may be that it provides aconcrete target, a low-level context in which to situateand target the study of the high-level human conceptualsystem.

19

References

Baillargeon, R., DeVos, J., & Graber, M. (1989).Location memory in 8-month-old infants in a non-searchAB task: Further evidence. Cognitive Development, 4,345-367.

Barsalou, L.W. (in press). Perceptual symbol systems.Behavioral & Brain Sciences.

Barsalou, L.W. (1987). The instability of gradedstructure: Implications for the nature of concepts. InNeisser, U. (Ed.) Concepts and ConceptualDevelopment: Ecological and Intellectual Factors inCategorization. Cambridge: Cambridge UniversityPress.

Beer, R. (1995). A dynamical systems perspective onagent-environment interaction. Artificial Intelligence, 72,173-215.

Brooks, R. (1991). Intelligence without reason. InProceedings of the 12th International Joint Conferenceon Aritificial Intelligence. Morgan Kauffman.

Clark, A. & Grush, R. (in press). Towards a cognitiverobotics. Adaptive Behavior.

Clark, A. (1997). Being There: Putting Brain, Body, andWorld Together Again. Cambridge, MA: MIT Press.

Fodor, J.A. (1998). Concepts: Where Cognitive ScienceWent Wrong. Oxford: Oxford University Press.

Fodor, J.A. & Pylyshyn, Z.W. (1988). Connectionismand cognitive architecture: A critical analysis. Cognition,28, 3-71.

Glaser, W.R. (1992). Picture naming. Cognition, 42, 61-105.

Glenberg, A.M. (1997). What memory is for.Behavioral & Brain Sciences, Mar. 20(1), 1-55.

Harnad, S. (1990). The symbol grounding problem.Physica D, 42, 335-346.

Harnad, S. (1987). Categorical Perception: TheGroundwork of Cognition. Cambridge: CambridgeUniversity Press.

Hutchins, E. (1995). Cognition in the Wild. Cambridge,MA: MIT Press.

Johnson, M. (1987). The Body in the Mind: The BodilyBasis of Imagination, Reason and Meaning. Chicago:University of Chicago Press.

Kirsh, D., & Maglio, P. (1992). Some epistemic benefitsof action: Tetris, a case study. In Proceedings of theFourteenth Annual Conference of the Cognitive ScienceSociety. Hillsdale, NJ: Lawrence Erlbaum.

Klatzky, R.L., Pellegrino, J.W., McCloskey, B.P., &Doherty, S. (1989). Can you squeeze a tomato? The roleof motor representations in semantic sensibilityjudgements. Journal of Memory and Language, 28, 56-77.

Kosslyn, S.M., Thompson, W.L., Kim, I.J., & Alpert,N.M. (1995). Topographical representations of mentalimages in primary visual cortex. Nature, 378, 496-498.

Kosslyn, S.M. (1976). Can imagery be distinguishedfrom other forms of internal representation? Evidencefrom studies of information retrieval time. Memory &Cognition, 4, 291-297.

Lakoff, G. (1987). Cognitive models and prototypetheory. In Neisser, U. (Ed.) Concepts and ConceptualDevelopment: Ecological and Intellectual Factors inCategorization. Cambridge: Cambridge UniversityPress.

Lenat, D.B. & Guha, R.V. (1990). Building LargeKnowledge-Based Systems. Reading, MA: Addison-Welsley.

Mandler, J.M. (1997). Development ofcategorization: Perceptual and conceptual categories. InBremner, G., Slater, A., & Butterworth, G. (Eds.) Infant

development: Recent advances. Hillsdale, NJ: Erlbaum.

Mandler, J.M. (1992). How to build a baby: II.Conceptual primitives. Psychological Review, 99(4),587-604.

Neisser, U. (1987). From direct perception to conceptualstructure. In Concepts and Conceptual Development:Ecological and Intellectual Factors in Categorization,Neisser, U. (Ed.). Cambridge: Cambridge UniversityPress.

Newell, A. and Simon, H.A. (1976). Computer science asempirical inquiry: Symbols and search. Communicationsof the ACM, 19(3), 113-126.

20

Piaget, J. (1952). The Origins of Intelligence in Children.New York: International University Press.

Port, R. & van Gelder, T. (1995). Mind as Motion:Dynamical Perspectives on Behavior and Cognition.Cambridge, MA: MIT Press.

Rosch, E. (1978). Principles of categorization. In E.Rosch & B.B. Lloyd (Eds.) Cognition andCategorization. Hillsdale, NJ: Erlbaum.

Rumelhart, D.E., & McClelland, J.L., Eds. (1986).Parallel Distributed Processing: Explorations in theMicrostructure of Cognition: Vol. 1. Foundations.Cambridge, MA: MIT Press.

Shepard, R. & Metzler, J. (1971). Mental rotation ofthree-dimensional objects. Science, 171, 701-703.

Thompson, E. (1995). Colour vision : a study incognitive science and the philosophy of perception.London: Routledge.

Van den Bergh, O., Vrana, S., & Eelan, P. (1990).Letters from the heart: Affective categorization of lettercombinations in typists and nontypists. Journal ofExperimental Cognition: Learning, Memory, &Cognition, 16, 1153-1161.

Vera, A. & Simon, H. (1993). Situated action: Asymbolic interpretation. Cognitive Science, 17, 4-48.