The psycholinguistics of signed and spoken languages: how ... · The psycholinguistics of signed and spoken languages: how biology affects processing Karen Emmorey 43-Gaskell-Chap43

Linguistic research over the last few decadeshas revealed substantial similaritiesbetween the structure of signed and spo-

ken languages (for reviews see Emmorey, 2002;Sandler and Lillo-Martin, 2006). These similari-ties provide a strong basis for cross-modalitycomparisons, and also bring to light linguisticuniversals that hold for all human languages. Inaddition, however, biology-based distinctionsbetween sign and speech are important, and canbe exploited to discover how the input–outputsystems of language impact online language pro-cessing and affect the neurocognitive underpin-nings of language comprehension and production.For example, do the distinct perceptual and pro-ductive systems of signed and spoken languagesexert differing constraints on the nature of lin-guistic processing? Recent investigations havesuggested that the modality in which a languageis expressed can impact the psychological mech-anisms required to decode and produce the lin-guistic signal. This chapter explores what aspectsof language processing appear to be universal toall human languages and what aspects areaffected by the particular characteristics of audi-tion vs. vision, or by the differing constraints onmanual versus oral articulation.

Sign language processing is appropriatelycompared to speech processing, rather than toreading, because unlike written text, which can becharacterized as “visual language,” sign language

consists of dynamic and constantly changingforms rather than static symbols. Further, nei-ther sign language nor spoken language comespre-segmented into words and sentences for theperceiver. The production of writing, althoughperformed by the hand, differs substantiallyfrom sign language production because writingderives its structure from a separate system (theorthography of a spoken language). In contrastto written language, sign and speech are bothprimary language systems, acquired duringinfancy and early childhood without formalinstruction.

43.1 Sign perception andvisual processingAlthough non-signers may interpret the visualsigned signal simply as a collection of rapid handand arm motions, signers quickly extract com-plex meaning from the incoming visual signal.Similarly, speakers extract meaning from a rap-idly changing acoustic stream, if they know thelanguage. Listeners and viewers are able to auto-matically parse an incoming auditory or visuallinguistic signal by virtue of stored internal rep-resentations. Speech perception involves seg-mentation of speech sounds into phonemic units.For signed languages, a first question is whethersigns actually exhibit sublexical linguistic structure

CHAPTER 43

The psycholinguistics of signed and spokenlanguages: how biologyaffects processingKaren Emmorey

43-Gaskell-Chap43 3/11/07 10:53 AM Page 703

emmorey

Inserted Text

(2007) In G. Gaskell (Ed.), The Oxford Handbook of Psycholinguistics, pp. 703-721, Oxford University Press.

emmorey

Note

(2007) In G. Gaskell (Ed.), The Oxford Handbook of Psycholinguistics, pp. 703-721, Oxford University Press.

that could be used by a parser to segment visualsigned input. Is it possible to have a phonologythat is not based on sound?

43.1.1 Phonology in a languagewithout soundSeveral decades of linguistic research has shownthat signed languages, like spoken languages, havea level of structure in which meaningless elementsare combined in rule-governed ways to createmeaningful forms (e.g. Stokoe, 1960; Battison,1978; Sandler, 1986; Brentari, 1998). For spokenlanguages, these elements are oral gestures thatcreate sounds. For signed languages, manual andfacial gestural units are combined to create distinct signs. The discovery that sign languagesexhibit phonological structure was ground-breaking because it demonstrated that signs arenot holistic pantomimes lacking internal organ-ization. Furthermore, this discovery showed thathuman languages universally develop a level ofmeaningless linguistic structure and a systemthat organizes this structure.

Briefly, signs are composed of three basicphonological parameters: hand configuration,location (place of articulation), and movement.Orientation of the hand/arm is another con-trasting parameter, but many theories representorientation as a sub-component of hand config-uration or movement, rather than as a basicphonological element. Figure 43.1 providesillustrations of minimal pairs from LIS (LinguaItaliana dei Segni, Italian Sign Language). Thetop part of the figure illustrates two LIS signsthat differ only in hand configuration. Not allsign languages share the same hand configura-tion inventory. For example, the “t” hand config-uration in American Sign Language (the thumbis inserted between the index and middle fingersof a fist) is not found in European sign languages.Chinese Sign Language contains a hand config-uration formed with an open hand with all fin-gers extended except for the ring finger, which isbent—this hand configuration does not occurin American Sign Language (ASL). In addition,signs can differ according to where they aremade on the body or face. Figure 43.1B illus-trates two LIS signs that differ only their place ofarticulation, and these different locations do notadd meaning to the signs. Signs can also differminimally in orientation, as illustrated in Figure43.1C. Finally, movement is another contrastingcategory that distinguishes minimally betweensigns, as shown in Figure 43.1D.

In addition to segment-like units, syllables havealso been argued to exist in signed languages

(Brentari, 1998; Corina and Sandler, 1993; Wilbur,1993). The syllable is a unit of structure that isbelow the level of the word but above the level ofthe segment, and is required to explain phono-logical form and patterning within a word.Although sign phonologists disagree about pre-cisely how sign syllables should be characterized,there is general agreement that a sign syllablemust contain a movement of some type. In ASL,several phonological constraints have beenidentified that must refer to the syllable. Forexample, only certain movement sequences areallowed in bisyllabic (two-movement) signs:circle + straight movements are permitted, butstraight + circle movements are not (Uyechi,1994). Although a straight + circle movementsequence is ill-formed as a single sign, it is well-formed when it occurs in a phrase. Thus, the con-straint on movement sequences needs to refer toa level smaller than the word (the constraint doesnot hold across word boundaries), but largerthan the segment. Within Sandler’s (1986) HandTier model, signed segments consist of Movementsand Locations, somewhat akin to Vowels andConsonants.

However, syllables in signed language differfrom syllables in spoken language because thereis little evidence for internal structure within thesigned syllable. Spoken syllables can be dividedinto onsets (usually, the first consonant or con-sonant cluster) and rhymes (the vowel and finalconsonants). Such internal structure does notappear to be present for sign syllables, althoughsome linguists have argued for weight distinc-tions, i.e. “heavy” vs. “light” syllables, based ondifferences in movement types. Because of thelack of internal structure, there do not appear tobe processes such as resyllabification in sign lan-guages (e.g. a segment from one syllable becomespart of another syllable). These facts are impor-tant, given the emphasis that speech productionmodels place on syllabification as a separateprocessing stage (e.g. Levelt et al., 1999).

Syllabification processes and/or the use ofa syllabary may be specific to phonological encoding for speech production. The syllablelikely serves as an organizational unit for speech,providing a structural frame for multisegmentalwords. For example, MacNeilage (1998) arguesthat the oscillation of the mandible creates aframe around which syllable production can beorganized. Meier (2000; 2002) points out thatsigning differs dramatically from speakingbecause signing does not involve a single, pre-dominant oscillator, akin to the mandible. Rather,signs can have movement that is restricted to justabout any joint of the arm. Thus, sign production

704 · CHAPTER 43 The psycholinguistics of signed and spoken languages


Sign perception and visual processing · 705

CAMBIARE (“change”)BICICLETTA (“bicycle”)

STIPENDIO (“salary”)AFFITTO (“rent”)

PAPA (“father”) UOMO (“man”)

FORMAGGIO (“cheese”) SASSO (“stone”)

(a)

(b)

(c)

(d)

Figure 43.1 Examples of minimal pairs in Lingua Italiana dei Signi, LIS (Italian Sign Language) (A) signsthat contrast in hand configuration; (B) signs that contrast in place of articulation (location); (C) signsthat contrast in orientation; (D) signs that contrast in movement. Illustrations from V. Volterra (ed.), Lalingua italiana dei segni. Bologna: Il Mulino, 1987 (new edn 2004) Copyright © Virginia Volterra andElena Radutzky. Reprinted with permission.


may not be constrained to fit within a frameimposed by a single articulator. Further, multi-segmental signs (e.g. signs with more than threesegments) are relatively rare, regardless of howsigned segments are defined (see Brentari, 1998).In contrast, syllabification processes for speechproduction may serve a critical framing functionfor words, which can contain many segments.

In sum, the linguistic evidence indicates thatsign languages exhibit a level of sublexical struc-ture that is encoded during sign production andthat could be used to parse an incoming visuallinguistic signal. A next question is whether sign-ers make use of such internal representationswhen perceiving signs. Evidence suggesting thatthey do comes from studies of categorical per-ception in American Sign Language.

43.1.2 Categorical perception in sign languageJust as hearing speakers become auditorilytuned to perceive the sound contrasts of theirnative language, ASL signers appear to becomevisually tuned to perceive manual contrasts inAmerican Sign Language. Two studies have nowfound evidence of categorical perception forphonologically distinctive hand configurationsin ASL (Emmorey et al., 2003; Baker et al., 2005).“Categorical perception” refers to the findingthat stimuli are perceived categorically ratherthan continually, despite continuous variation inform. Evidence for categorical perception isfound (1) when perceivers partition continuousstimuli into relatively discrete categories and (2)when discrimination performance is betteracross a category boundary than within a cate-gory. For these categorical perception experi-ments, deaf signers and hearing non-signerswere presented with hand configuration con-tinua that consisted of two handshape endpointswith nine intermediate variants. These continuawere either generated via a computer morphingprogram (Emmorey et al., 2003) or from a livesigner (Baker et al., 2005). In addition, Emmoreyet al. (2003) investigated categorical perceptionfor place of articulation continua. For all experi-ments, participants performed a discriminationtask in which they made same/different judge-ments for pairs or triplets of images from a con-tinuum, and an identification task in which eachstimulus was categorized with respect to the end-points of the continuum (the discriminationtask always preceded the categorization task).

Deaf ASL signers and hearing English speak-ers (non-signers) demonstrated similar categoryboundaries for both hand configuration and

place of articulation (Emmorey et al., 2003; Bakeret al., 2005). This result is consistent with previ-ous studies which found that deaf and hearingparticipants exhibit similar perceptual groupingsand confusability matrices for hand configura-tion and for place of articulation (Lane et al.,1976; Poizner and Lane, 1978). Thus, these ASLcategories may have a perceptual as well as a linguistic basis. However, only deaf signers exhib-ited evidence of categorical perception, and onlyfor distinctive hand configurations. Only deafsigners were sensitive to hand configuration cat-egory boundaries in the discrimination task,performing significantly better across categoryboundaries than within a hand configurationcategory (Emmorey et al., 2003; Baker et al.,2005).

Interestingly, neither group exhibited categor-ical perception effects for place of articulation(Emmorey et al., 2003). Lack of a categoricalperception effect for place of articulation maybe due to more variable category boundaries. Inspeech, categorical perception is modulated bythe nature of the articulation of speech sounds.For example, categorical perception is often weakor not present for vowels, perhaps because ofthe more continuous nature of their articulationcompared to stop consonants (Fry et al., 1962).The same may be true for place of articulationin sign language. For example, the location ofsigns can be displaced within a major bodyregion in casual signing (Brentari, 1998) orcompletely displaced to the side during whis-pering. Category boundaries for place of articu-lation appear to be much less stable than forhand configuration. Categorical perception mayonly occur when articulations are relatively dis-crete for both sign and speech.

The fact that only deaf signers exhibited cate-gorical perception for ASL hand configurationsindicates that linguistic experience is what drivesthese effects. However, categorical perceptioneffects are weaker for sign than for speech. Deafsigners’ discrimination ability within hand configuration categories was better than thenear-chance discrimination ability reportedwithin stop consonant categories for speech(e.g. Liberman et al., 1957). Nonetheless, thesign language results resemble discriminationfunctions observed for categorical perception inother visual domains, such as faces or facialexpressions (e.g. Beale and Keil, 1995; de Gelderet al., 1997). Discrimination accuracy withinvisual categories tends to be relatively high; gen-erally, participants perform with about 70–85per cent mean accuracy rates within categories.The difference in the strength of categorical



perception effects between speech and sign mayarise from psychophysical differences betweenaudition and vision.

In sum, deaf signers appear to develop specialabilities for perceiving aspects of sign languagethat are similar to the abilities that speakersdevelop for perceiving speech. These findingssuggest that categorical perception emerges nat-urally as part of language processing, regardlessof language modality. In addition, the resultsindicate that phonological information is uti-lized during the perception of moving nonsensesigns (Baker et al., 2005) and when viewing stillimages of signs (Emmorey et al., 2003). Furtherresearch is needed to discover what parsing pro-cedures might be used to identify sign bound-aries, and whether categorical perception processesmight play a role in segmenting the signingstream.

43.2 Processing universalsand modality effects in themental lexiconMany models of spoken word recognition hypoth-esize that an acoustic-phonetic representation issequentially mapped onto lexical entries, andlexical candidates which match this initial repre-sentation are activated (e.g. Marslen-Wilson,1987; McClelland and Elman, 1986; Goldingeret al., 1989). As more of a word is heard, activa-tion levels of lexical entries which do not matchthe incoming acoustic signal decrease. Thesequential matching process continues untilonly one candidate remains which is consistentwith the sensory input. At this point, wordrecognition can occur. This process is clearlyconditioned by the serial nature of speech per-ception. Since signed languages are less depend-ent upon serial linguistic distinctions, visuallexical access and sign recognition may differfrom spoken language. To investigate this possi-bility, Grosjean (1981) and Emmorey and Corina(1990) used a gating technique to track theprocess of lexical access and sign identificationthrough time.

43.2.1 The time course of sign vs. word recognitionIn sign language gating tasks, a sign is presentedrepeatedly, and the length of each presentationis increased by a constant amount (e.g. onevideoframe or 33 msec). After each presenta-tion, participants report what they think thesign is and how confident they are. Results from

such studies show that ASL signers produce initial responses which share the place of articula-tion, orientation, and hand configuration of thetarget sign but differ in movement (Grosjean,1981; Emmorey and Corina, 1990). The move-ment of the sign is identified last, and coincideswith lexical recognition. This pattern of responsessuggests that, similarly to the speech signal, thevisual input for sign activates a cohort of poten-tial lexical candidates that share some initialphonological features. This set of candidatesnarrows as more visual information is presented—until a single sign candidate remains. Clark andGrosjean (1982) showed further that sententialcontext did not affect this basic pattern of lexicalrecognition, although it reduced the time toidentify a target sign by about 10 per cent.

However, unlike spoken word recognition,sign recognition appears to involve a two- stageprocess of recognition in which one group ofphonological features (hand configuration, ori-entation, and place of articulation) initiallyidentifies a lexical cohort, and then identifica-tion of phonological movement leads directly tosign identification. Such a direct correlationbetween identification of a phonological ele-ment and lexical identification does not occurwith English and may not occur for any spokenlanguage. That is, there seems to be no phono-logical feature or structure, the identification ofwhich leads directly to word recognition.Movement is the most temporally influencedphonological property of sign, and more time isrequired to resolve it. For speech, almost allphonological components have a strong tempo-ral component, and there does not appear to be asingle feature that listeners must wait to resolvein order to identify a word.

Furthermore, both Grosjean (1981) andEmmorey and Corina (1990) found that signswere identified surprisingly rapidly. Althoughsigns tend to be much longer than words, only35 per cent of a sign had to be seen before thesign was identified (Emmorey and Corina, 1990).This is significantly faster than word recognitionfor English. Grosjean (1980) found that approx-imately 83 per cent of a word had to be heardbefore the word could be identified. There are atleast two reasons why signs may be identifiedearlier than spoken words. First, the nature ofthe visual signal for sign provides a large amountof phonological information very early andsimultaneously. The early availability of thisphonological information can dramatically nar-row the set of lexical candidates for the incom-ing stimulus. Second, the phonotactics andmorphotactics of a visual language such as ASL

Processing universals and modality effects in the mental lexicon · 707


may be different from those of spoken languages.In English, many words begin with similarsequences, and listeners can be led down a gar-den path if a shorter word is embedded at theonset of a longer word—for example, “pan” in“pantomime.” This phenomenon does not com-monly occur in ASL. Furthermore, sign initialcohorts seem to be much more limited byphonotactic structure. Unlike English, in whichmany initial strings have large cohorts (e.g. thestrings [kan], [mæn], and [skr] are all shared bythirty or more words), ASL has few signs whichshare an initial phonological shape (i.e. the samehand configuration and place of articulation).This phonotactic structure limits the size of theinitial cohort in ASL. The more constrainedphonotactics and the early and simultaneousavailability of phonological information mayconspire to produce numerically and propor-tionally faster identification times for ASL signs.

In sum, lexical access and word recognitionare generally quite similar for spoken and signedlanguages. For both language types, lexical accessinvolves a sequential mapping process betweenan incoming linguistic signal and stored lexicalrepresentations. For signed languages, this appearsto be a two-stage process in which one set ofphonological elements are initially accessed andthen identification of movement leads to signrecognition. Finally, the phonotactics of ASL(and possibly other signed languages) leads toproportionally faster recognition times for signsthan for words.

43.2.2 The organization of a sign-based lexicon: evidence from tip-of-the-fingersA tip-of-the-tongue (TOT) experience refers tothe state in which a speaker is temporarilyunable to retrieve a word from memory, whilebeing sure that he or she knows the word (seeBrown, 1991; Schwartz, 1999 for reviews). Often,speakers are able to retrieve the first letter andsometimes the number of syllables, which pro-vides evidence for the organization of spokenlanguage lexicons. The existence and nature ofTOTs in spoken languages suggest that inde-pendent processing stages provide access to wordmeanings and word forms (e.g. Dell et al., 1997;Garrett, 1975; Levelt et al., 1999). However, forsigned languages, the division between semanticand phonological form has been questionedbecause they exhibit a high degree of iconicity.For example, Stokoe (1991) proposes a theory of“semantic phonology” in which representations ofa sign’s form can be derived from aspects of its

semantics. Semantic phonology eliminates form/meaning distinctions and rejects duality of pat-terning for signed languages (Armstrong et al.,1995). Under such a model, a “tip-of-the-fingers”experience should not occur for signers becausethere is no clear division between semantics andphonology.

Thompson et al. (2005) investigated whethera “tip-of-the-fingers” (TOFs) experience occurfor ASL signers and whether TOFs are similar toTOTs. Thonmpson et al. (2005) conducted a smalldiary study and also experimentally elicited TOFsby asking signers to translate English words (e.g.what is the sign for “Moscow”?). Thompson et al.(2005) found that ASL signers reported havingTOF experiences in which they could retrievedetailed semantic information, but had little orno access to the sign form. TOFs were similar toTOTs in that the majority involved proper names,and participants sometimes had partial access tophonological form (e.g. recalling the hand con-figuration and location of a sign, but not itsmovement). Although some TOF phonologicalparameters were iconic, there was no relation-ship between degree of iconicity for a particularparameter and access to it during a TOF. Thisseparation between semantic and phonologicalretrieval provides evidence against semanticphonology (Stokoe, 1991), and indicates thatindependent processing stages provide access tolexical meanings and lexical forms for both signand speech.

In addition, lexical access during sign produc-tion parallels access during sign perception.That is, during a TOF participants were equallylikely to recall hand configuration, location, andorientation, which constitute the onset of a sign,and least likely to recall movement, whichunfolds over time. These results parallel the gat-ing studies of sign perception, indicating thatphonological onsets are privileged for bothsigned and spoken languages.

Perhaps the most remarkable aspect of theTOFs for signs was participants’ frequent recallof as many as three out of the four phonologicalparameters. This qualitative aspect of TOFs hasfurther implications for models of signed lan-guage production, and distinguishes TOFs fromTOTs. TOFs appear to be qualitatively quite dif-ferent from TOTs with respect to the amount ofphonological information that is retrievedsimultaneously. However, recall of three phono-logical parameters did not result in more TOFresolutions compared to when fewer or noparameters were recalled. Thus, signs appear tobe stored as a set of phonological attributeswhere retrieval of one or more attributes does not



result in immediate access to the full phonologi-cal representation. TOFs therefore can occur whenany one parameter is insufficiently activated.

In sum, the existence of a tip-of-the-fingersphenomenon for sign language argues for atwo-stage model of lexical access and a divisionbetween semantic and phonological represen-tations. The nature of recall for TOFs was anal-ogous to TOTs in that partial phonologicalinformation (most frequently from word onsets)was sometimes retrieved. However, lexical TOFsdiffered from TOTs in the amount of informa-tion simultaneously available. The results of theThompson et al. (2005) study suggest that theASL mental lexicon is not organized by a singlephonological parameter (e.g. hand configura-tion) that guides retrieval. Rather, sign produc-tion appears to parallel sign perception, suchthat when in a TOF, signers are least likely to

retrieve the movement of the target sign. Moregenerally, the findings argue for a language uni-versal processing pattern in which onsets have aspecial status, regardless of language modality.

43.2.3 The preference for non-concatenative morphology: a processing explanationSigned languages differ from spoken languagesin the type of combinatorial processes that mostoften create morphologically complex words.Specifically, signed languages show a markedpreference for non-concatenative (simultaneous)morphological processes, in contrast to the pref-erence for linear affixation exhibited by spokenlanguages. Examples of typical simultaneousmorphological processes are given in Figure 43.2from British Sign Language (BSL). In Figure 43.2A,

Processing universals and modality effects in the mental lexicon · 709

(a)

(b)

(c)

Figure 43.2 Examples of non-concatenative (simultaneous) morphology in British Sign Language (BSL)From Kyle and Woll (1985). Reprinted with permission.


a second hand is added to indicate plural. InFigure 43.2B, the direction of movement of theverb indicates subject and object arguments,and in Figure 43.2C, aspectual information isindicated by the type of the movement superim-posed on the verb stem.

Linear affixation has been documented insigned languages. For example, ASL contains afew suffixes: the multiple suffix (Wilbur, 1987),the agentive suffix -ER, and a negative suffixZERO (Aronoff et al., 2005). Aronoff et al. (2005)also describe a derivational prefix in Israeli Signlanguage. However, the majority of morpholog-ical processes in signed languages appear to besimultaneous. In fact, Bergman (1982) claimsthat Swedish Sign Language has neither suffixa-tion nor prefixation, but exhibits several types of reduplication and other non-concatenativemorphological process. Similarly, Sutton-Spenceand Woll (1999) describe only non-concatena-tive morphological processes for British SignLanguage, with the exception of compounding.Thus far, the data from numerous signed lan-guages indicates that linear affixation is rare andthat simultaneous expression of a stem and itsmorphological markings is the preferred linguisticencoding.

In contrast, for spoken languages, simultane-ous affixation (e.g. template morphology, infix-ation, reduplication) is relatively rare, and linearaffixation is the preferred linguistic encoding formorphological processes. Cutler et al. (1985) arguethat processing constraints underlie the rarity ofmorphological processes which alter the phono-logical integrity of the base form (e.g. infixationwhich inserts an affix into the middle of aword). Languages avoid processes that disruptthe structural integrity of linguistic units. Hall(1992) also argues that the rarity of non-con-catenative (simultaneous) morphology is due tothe processing complexity associated with dis-continuous elements in general (e.g. centerembedding or verbs with particles). Concatenativemorphology requires much less computationalcomplexity because of the straightforward map-ping between the surface form of a word and itsunderlying representation (Anderson, 1992).Given these arguments, why do signed lan-guages prefer non-concatenative morphology,and does it pose the same processing challengesthat it does for spoken languages?

First, signed languages appear to favor non-concatenative morphology because the visualmodality affords parallel processing. Vision caneasily encode spatially distinct information inparallel (unlike audition), and as we have noted,the hand configuration, place of articulation,

and orientation of signs are perceived nearlysimultaneously. Second, the capacity for short-term memory is limited by articulation rate(Baddeley, 1986), and signs take longer thanwords to articulate (Bellugi and Fischer, 1972).Universal constraints on short-term memorycapacity and a slower articulation rate may inducesign languages to disfavor linear affixation. Third,unlike the non-concatenative processes of infix-ation and circumfixation, the morphologicalprocesses of signed languages do not interrupt thebase form and do not involve discontinuous affixes.As can be seen in the examples in Figure 43.2, inno case is the base form of the BSL sign actuallyinterrupted by the morphological marking.Discontinuous circumfixation is not the correctanalysis for these forms since the morphologicalmarking is superimposed onto the verb stem.Thus, the morphological parsing difficulties thatarise from non-concatenative processes in spo-ken languages do not arise for signed languages.

Finally, evidence for signed languages’ aver-sion to linear affixation comes from Supalla’s(1991) finding that when the linear morphologyof a spoken language is transferred to the visualmodality, deaf children exposed to this artificiallanguage do not acquire the system and in fact,alter it to create simultaneous (spatial) morpho-logical encoding. In the United States, ManuallyCoded English (MCE) is the cover term for signsystems developed in the 1970s to represent themorphology and syntax of English, such asSigning Exact English or SEE (Gustason et al.,1980). MCE was invented by educators (manyfluent in ASL) as a means to make English acces-sible to deaf children. The basic vocabulary ofMCE borrows heavily from the lexicon of ASL,but its inflectional morphology is strictly sequen-tial and based on English morphology. For exam-ple, to express “He asked her,” the MCE sign HEis made at the forehead with an “E” hand config-uration, the ASL sign REQUEST is produced(with no spatial inflection), then the MCE suffix-ED is produced, followed by the MCE signHER made at the chin with an “R” hand config-uration (see Figure 43.3). To express “He askedher” in ASL (or in BSL), the verb ASK is directedfrom the location in signing space associatedwith the subject (the referent of “he”) to thelocation of the object (the referent of “her”) (seeFigure 43.3B). Tense is not expressed morpho-logically in ASL. Further, pronouns in signedlanguages are generally directed toward loca-tions in signing space to convey co-reference.

Supalla (1991) found that children exposedonly to Manually Coded English modify theinflectional morphology of verbs and pronouns



to take advantage of the visual modality. That is,these children produce spatial non-linear modi-fications to base verbs in order to mark verb argu-ments, despite the fact that they were exposedonly to linguistic input that produced these dis-tinctions linearly. The children’s spatial mor-phological creations were idiosyncratic, but theywere systematic within a child and similar to thegrammatical morphology found in signed lan-guages of the world. Stack (1999) also found thatJamie, a young child exposed only to MCE,failed to acquire the non-spatial pronouns andlinear inflections of MCE; rather, she created apronominal system that utilized space and inno-vated non-linear morphology to express linguis-tic notions such as plurality (by reduplication),reciprocal aspect (the second hand mirrors thefirst), and lexical arguments (indicated by thebeginning and endpoints of a verb). These resultssuggest that not only does the visual modalityeasily afford non-linear affixation, but visualprocessing may actually demand it.

43.3 Comprehension anddiscourse: the unique role of space for signed languagesThe comprehension of sign language discoursedepends upon interpreting the meaning of

locations in signing space. For example, formany (if not most) signed languages, discoursereferents are associated with locations in signingspace, and pronominal signs and “agreeing”verbs (like ASK in Figure 43.3B) can be directedtoward those locations to refer back to the asso-ciated referents. In addition, signing space isused to convey information about spatial rela-tionships among objects. In this case, signingspace serves both a topographic and a referentialfunction (Emmorey et al., 1995). Several psy-cholinguistic studies have explored how ASLsigners understand and maintain the associa-tions between referents and spatial locations,and whether the same processing mechanismshold for spoken and signed languages. Thesestudies are briefly summarized below.

43.3.1 Understanding spatialcoreferenceTo understand a sentence containing a pronoun,a perceiver must correctly assign an antecedentto the pronoun, which for languages like ASLinvolves interpreting the direction of the pro-noun (where the pointing sign is directed) andrecalling the referent associated with the tar-geted location. Emmorey et al. (1991) used theprobe recognition methodology to investigatewhether ASL pronouns re-activate their antecedents,

Comprehension and discourse: the unique role of space for signed languages · 711

(a)

(b)

HE-ASK-HER

HE ASK -ED HER

Figure 43.3 Examples of Manually Coded English (MCE) and ASL. (A) the MCE sentence ‘He askedher’; (B) the equivalent sentence in ASL. The MCE signs are reprinted from Gustason et al. (1972).Reprinted with permission. ASL illustration copyright ©Ursula Bellugi, The Salk Institute.


as has been found with spoken language pronouns(e.g. Corbett and Chang, 1983; Gernsbacher,1989). In the Emmorey et al. (1991) study, deafparticipants viewed videotaped ASL sentenceswith and without pronouns and responded to probe signs presented after the pronoun.Participants had to decide whether the probesign occurred in the sentence, and probe signswere either antecedent or non-antecedent nouns.Response times to antecedent nouns were fasterthan to non-antecedent nouns, and responsetimes to antecedent nouns were faster when apronoun was present in the sentence. Furthermore,Emmorey (1997) found that ASL pronouns sup-press activation of non-antecedents, when theappropriate baseline condition is used. Theseresults indicate that ASL pronouns activate theirantecedents and suppress non-antecedents inmemory, just as has been found for spoken lan-guages (Gernsbacher, 1989).

In addition, ASL agreeing verbs license phono-logically null pronouns. In clauses with agreeingverbs, subjects and objects appear as null ele-ments that do not have an overt lexical form.Null pronouns are permitted to occur becauseof the morphological marking of agreeing verbs(see Figure 43.2B for an example of a BSL agree-ing verb and Figure 43.3B for an ASL example).Using the same probe recognition methodology,Emmorey and Lillo-Martin (1995) found thatnull pronouns that were licensed by ASL agree-ing verbs activate their antecedents to the sameextent as overt pronouns. Again, these resultsparallel what has been found for spoken lan-guages (e.g. Bever and McElree, 1988; Fodor, 1989)and suggest that the psycholinguistic mecha-nisms involved in anaphora resolution are universal and not dependent upon languagemodality.

Finally, Emmorey and Falgier (2004) investi-gated the unique case of “locus doubling,” in whicha single referent is associated with two distinctspatial locations (van Hoek, 1992). Emmoreyand Falgier (2004) asked whether an ASL pro-noun activates both its antecedent referent andthe location associated with that referent. In thisexperiment, participants were presented with anintroductory discourse that associated a referent(e.g. MOTHER) with two distinct locations (e.g.STOREleft, KITCHENright), and a continuationsentence followed that either contained a pro-noun referring to the referent in one location orcontained no anaphora (the control sentence).Deaf participants made lexical decisions toprobe signs presented during the continuationsentences, and the probe signs were either thereferent of the pronoun, the referent-location

determined by the pronoun, or the most recentlymentioned location (not referenced by the pro-noun). The results indicated that response timesto referent nouns were faster in the pronounthan in the no-pronoun control condition andthat response times to the location signs did notdiffer across conditions. Thus, the spatial natureof coreference in ASL does not alter the pro-cessing mechanism underlying the online inter-pretation of pronouns. Pronouns activate onlyreferent nouns, not spatial location nouns asso-ciated with the referent.

In sum, results from a series of sign languagecomprehension experiments indicate that theprocessing mechanisms used to resolve and inter-pret coreference relations do not differ cross-linguistically or cross-modally. Pronouns, whetherspatialized pointing signs, spoken words, or nullelements licensed by verb morphology, activateantecedents and suppress non-antecedents inmemory, thereby improving the accessibility of coreferent nominals within the discourse(Gernsbacher, 1989). Language modality does notappear to affect co-reference resolution processes,despite great differences in the surface form ofspoken and signed pronominal systems.

43.3.2 Understanding spatialdescriptionsMost spoken languages encode spatial relationswith prepositions or locative affixes. There is agrammatical element or phrase that denotes thespatial relation between a figure and groundobject, e.g. the English spatial preposition onindicates support and contact, as in The cup ison the table. The prepositional phrase on thetable defines a spatial region in terms of a groundobject (the table), and the figure (the cup) islocated in that region (Talmy, 2000). Spatialrelations can also be expressed by compoundphrases such as to the left or in back of. Both sim-ple and compound prepositions constitute aclosed class set of grammatical forms for English.In contrast, signed languages convey spatialinformation using classifier constructions inwhich spatial relations are expressed by wherethe hands are placed in signing space or withrespect to the body (e.g. Supalla, 1982; Engberg-Pedersen, 1993). For example to indicate ‘Thecup is on the table,’ an ASL signer would place aC classifier handshape (a curved handshapereferring to the cup) on top of a B classifierhandshape (a flat hand referring to the table).There is no grammatical element specifying thefigure–ground relation; rather, there is a schematicand isomorphic mapping between the location



of the hands in signing space and the location ofthe objects described (Emmorey and Herzig,2003). This spatialized form has important ram-ifications for the nature of addressee vs. speakerperspective within spatial descriptions and forhow these descriptions are understood.

Figure 43.4A provides a simple example of anASL spatial description. An English translationof this example would be “I entered the room.There was a table to the left.” In this type of nar-rative, the spatial description is from the pointof view of the speaker (for simplicity and clarity,“speaker” will be used to refer to the person whois signing.) The addressee, if facing the speaker,must perform a mental transformation of sign-ing space. For example, in Figure 43.4A, thespeaker indicates that the table is to the left byarticulating the appropriate classifier sign on hisleft in signing space. Because the addressee isfacing the speaker, the location of the classifierform representing the table is actually on theright for the addressee. There is a mismatchbetween the location of the table in the room

being described (the table is on the left as seenfrom the entrance) and what the addressee actually observes in signing space (the classifierhandshape referring to the table is produced to the addressee’s right). In this case, theaddressee must perform what amounts to a 180°mental rotation to correctly comprehend thedescription.

Although spatial scenes are most commonlydescribed from the speaker’s point of view (as inFigure 43.4A), it is possible to indicate a differ-ent viewpoint. ASL has a marked sign that canbe glossed as YOU-ENTER, which indicates thatthe scene should be understood as signed fromthe addressee’s viewpoint (see Figure 43.4B).When this sign is used, the signing space in whichthe room is described is, in effect, rotated 180°so that the addressee is “at the entrance” of theroom. In this case, the addressee does not need tomentally transform locations within signing space.However, ASL descriptions using YOU-ENTERare quite unusual and rarely found in naturaldiscourse. Furthermore, Emmorey et al. (1998)

Comprehension and discourse: the unique role of space for signed languages · 713

(c)

Speakers’s perspective

Addressee’s perspective

I-ENTER TABLE THERECL:C (2h)

YOU-ENTER TABLE THERECL:C (2h)

(a)

(b)

entrancePosition of the table described in (A) and (B)

Figure 43.4 Illustration of ASL descriptions of the location of a table within a room, described from (A)the speaker’s perspective or (B) the addressee’s perspective. Signers exhibit better comprehension forroom descriptions presented from the speaker’s perspective, despite the mental transformation thatthis description entails. Reprinted from Emmorey (2002), with permission.


found that ASL signers comprehended spatialdescriptions much better when they were pro-duced from the speaker’s point of view com-pared to the addressee’s viewpoint. In that study,signers viewed a videotape of a room and then a signed description and were asked to judgewhether the room and the description matched.When the room was described from the addressee’sperspective (using YOU-ENTER), the descrip-tion spatially matched the room layout shownon the videotape, but when signed from thespeaker’s perspective (using I-ENTER), thedescription was the reverse of the layout on thevideotape (a simplified example is shown inFigure 43.4). Emmorey et al. (1998) found thatASL signers were more accurate when presentedwith descriptions from the speaker’s perspec-tive, despite the mental transformation thatthese descriptions entailed.

One might consider this situation analogousto that for English speakers who must under-stand the terms left and right with respect to thespeaker’s point of view (as in on my left). Thecrucial difference, however, is that these rela-tions are encoded spatially in ASL, rather thanlexically. The distinction becomes particularlyclear in situations where the speaker and theaddressee are both in the environment, observingthe same scene. In this situation, English speakersmost often adopt their addressee’s point of view,for example giving directions such as, Pick the oneon your right, or It’s in front of you, rather thanPick the one on my left or It’s farthest from me(Schober, 1993; Mainwaring et al., 1996). However,when jointly viewing an environment, ASL sign-ers do not adopt their addressee’s point of viewbut use “shared space” (Emmorey and Tversky,2002). Signing space is shared in the sense that itmaps to the physically observed space and toboth the speaker’s and addressee’s view of thephysical space. In such situations, there is notrue speaker vs. addressee perspective and nomental transformation is required by theaddressee. Furthermore, spatial descriptions ofjointly viewed environments are not altered bythe location of an addressee. That is, in these sit-uations, ASL signers do not need to take intoaccount where their addressee is located, unlikeEnglish speakers, who tend to adopt theiraddressee’s viewpoint (Emmorey and Tversky,2002). These differences between languages derivefrom the fact that signers use the actual space infront of them to represent observed physicalspace.

In sum, the spatialization of linguistic expres-sion in ASL affects the nature of language com-prehension by requiring an addressee to perform

a mental transformation of the linguistic spaceunder certain conditions. Specifically, for descrip-tions of non-present environments, an addresseemust mentally transform the locations within aspeaker’s signing space in order to correctlyunderstand the left/right arrangements of objectswith respect to the speaker’s viewpoint. Forspeech, spatial information is encoded in anacoustic signal, which bears no resemblance tothe spatial scene described. An English speakerdescribing the room in Figure 43.4 might sayeither You enter the room, and a table is to yourleft or I enter the room, and a table is to my left.Neither description requires any sort of mentaltransformation on the part of the addresseebecause the relevant information is encoded inspeech rather than in space. However, whenEnglish speakers and addressees discuss a jointlyviewed scene, an addressee may need to performa type of mental transformation if the speakerdescribes a spatial location from his or her view-point. Again, this situation differs for ASL signersbecause the speaker’s signing space maps to theobserved physical space and to the addressee’sview of that space. Signing space is shared, and no mental transformation is required by theaddressee. When shared space is used, speakersand addressees can refer to the same locations insigning space, regardless of the position of theaddressee. Thus, the interface between languageand visual perception (how we talk about whatwe see) has an added dimension for signers (theyalso see what they talk about). That is, signers see(rather than hear) spatial descriptions, and thereis a schematic isomorphism between aspects of the linguistic signal (the location of the handsin signing space) and aspects of the spatial scene described (the location of objects in thedescribed space). Signers must integrate a visu-ally observed linguistic signal with a visuallyobserved environment or a visual image of thedescribed environment.

43.4 Speaking vs. signingCurrently, very little is known about the psy-cholinguistic mechanisms that translatethoughts into linguistic expression for signers,and it is unclear whether models of speech pro-duction can simply be appropriated for sign lan-guage. According to most production models,successful speech involves (1) the selection of aword that is semantically and syntactically appro-priate, (2) retrieval of the word’s phonologicalproperties, (3) rapid syllabification of the word incontext, and (4) articulatory preparation of theassociated gestures (see Levelt, 1999 for review).



Sign production is likely to involve these components as well, although evidence for syl-labification processes is weak (see above). In this section, I discuss aspects of sign productionthat are unique to the visual-manual modalityand review some provocative parallels and con-trasts between the nature of speech and signproduction.

43.4.1 Lexical selection andphonological encoding during sign productionEvidence for the time course of lexical selectionand phonological encoding for speech produc-tion comes from the picture–word interferencetask (Schriefers et al., 1990). In this task, subjectsare presented with a picture that they mustname, and a distractor name is presented eitherprior to, at the same time, or after the presenta-tion of the picture. Subjects are told to ignorethe distractor item, which they find difficult todo. If a semantically related word (e.g. goat for apicture of a sheep) is presented at the same timeor slightly (100–400 msec) before the presenta-tion of the picture, subjects are much slower toname the picture. That is, semantic inhibitionoccurs, and speakers are slow to produce thename of the object. In contrast, if a phonologi-cally related word (e.g. sheet for a picture of asheep) is presented at the same time or shortly(100–400 msec) after presentation of the picture,subjects are quicker to name the picture. That is,phonological facilitation occurs, and speakersare faster at producing the object name. Evidencefor early phonological facilitation has beenmixed (e.g. Starreveld, 2000; Schriefers et al.,1990), but the general pattern of results suggeststhat activation of semantic information occursearly in lexical retrieval, while phonologicalencoding occurs simultaneously with or subse-quent to lexical selection.

Recently, Corina and Knapp (forthcoming)investigated lexical selection processes for ASLusing a picture–sign interference task. Deafparticipants were asked to sign the name of apicture, and response time was measured fromthe time of picture presentation to when theparticipant’s hands moved from a rest position,breaking an infrared beam. Superimposed onthe picture was an image of a signer producing asign that was either phonologically or semanti-cally related to the picture, e.g. MATCH–cigarette(a semantically related sign–picture pair) orORANGE–ice cream (a phonologically relatedsign–picture pair—the signs ORANGE and ICE-CREAM are both made with an S handshape at

the mouth). Both the distractor sign and thepicture were clearly visible because the overlaidimage of the signer was semi-transparent (butstill quite recognizable). Corina and Knapp(forthcoming) found that, like speakers, signersexhibited semantic interference when they hadto name a picture that was preceded by a seman-tically related sign (-130 msec “SOA” – the timebetween the onset of the picture and the onsetof the superimposed distractor sign). Semanticinterference was not found with zero or laterSOAs. Thus, as for speech, sign productioninvolves the early retrieval of lexical semanticinformation, and sign production can be dis-rupted (slowed) by the prior activation of asemantically related sign.

However, the evidence for phonological facili-tation was not as clear-cut. No significant phono-logical effects were observed at any of the SOAsused in the experiment (-130, 0, +130 msec), andan examination of the number of shared phono-logical parameters also revealed no significantfacilitation effects from distractor signs thatshared one, two, or three phonological parame-ters with the target sign (the picture name).There was no evidence of increased facilitationwith increased phonological overlap. However,further post hoc analyses indicated that distrac-tor signs which shared movement and locationwith the target sign produced significant facili-tation effects at all SOAs, while signs whichshared handshape and location or handshapeand movement did not.

The fact that phonological facilitation wasonly observed when sign pairs shared movementand location supports phonological models thattreat hand configuration as an autonomous ele-ment and movement and location as segmentalunits that could frame sign production (e.g.Sandler, 1986; Corina, 1993). In addition, phono-logical facilitation based on shared movementand location for sign production is consistentwith results from a perceptual similarity judg-ment study. Hildebrandt and Corina (2002)found that native ASL signers rated non-signsthat shared movement and location as highlysimilar, and essentially ignored handshape simi-larity. It is possible that movement and locationform a syllabic unit that is perceptually salientand that can be primed during sign production.

In sum, the picture–sign interference resultsof Corina and Knapp (forthcoming) indicatethat semantic inhibition occurs only at an earlySOA (−130msec), while phonological facilita-tion (based on shared movement and location)occurs at both early and late SOAs. This patternof results mirrors what has been found for spoken

Speaking vs. signing · 715


languages, and suggests that lexical selectionprecedes phonological encoding for sign, as itdoes for speech (note, however, that the data donot rule out cascaded activation of phonologicalrepresentations during lexical selection).

43.4.2 Slips of the handIn her well-known 1971 paper “The non-anom-alous nature of anomalous utterances,” VictoriaFromkin demonstrated that slips of the tongue(speech errors) were not random mistakes inspeaking, but revealed something about speechplanning and the nature of the mental represen-tation of phonology. She argued that speecherrors provide evidence for the underlying unitsof speech production: “despite the semi-contin-uous nature of the speech signal, there are dis-crete units at some level of performance whichcan be substituted, omitted, transposed, oradded” (Fromkin, 1971: 217; emphasis in theoriginal). Errors of sign production provide sim-ilar evidence for the status of the major phono-logical parameters as discrete units involved insign production.

Sign error corpora collected for ASL by Newkirket al. (1980) and for Deutsche Gebärdensprache(DGS; German Sign Language) by Hohenbergeret al. (2002) document exchange, preservation,and anticipation errors which involve hand con-figuration, place of articulation, or movement(see also Leuniger et al., 2004). Figure 43.5 pro-vides an example of a hand configuration antic-ipation error in DGS. The signer planned to signSEINE ELTERN (‘his parents’), and incorrectlyproduced SEINE with a Y handshape instead ofa B handshape. The Y hand configuration of the

sign ELTERN was anticipated and substituted forthe intended B hand configuration of SEINE.

The existence of such errors suggests thatthese phonological parameters constitute unitsin the production of signed utterances. As notedabove, many models of sign language phonol-ogy treat hand configuration as a separateautosegment, much as tone is represented forspoken languages. The speech error data fromtone languages suggests that tones are inde-pendent units that can participate in exchange,anticipation, or perseveration errors (Gandour,1977)—just as we find for hand configuration insign language. Unlike tone, however, hand con-figuration errors are much more common thanerrors involving other phonological parameters(Newkirk et al., 1980; Hohenberger et al., 2002).Speech errors involving tone do not appear to bemore frequent than errors involving consonantsor vowels (Wen, 2000). One possible explanationfor the frequency of hand configuration errors isthat hand configuration is the most complexphonological parameter (Brentari, 1998; Sandlerand Lillo-Martin, 2006). The feature geometryrequired to represent hand configuration requiresseveral hierarchical nodes and more features thanare needed to specify the movement or locationof a sign. This complexity may render hand con-figuration more vulnerable to error during signproduction.

The fact that movement exchange errorsoccur (Klima and Bellugi, 1979) argues for aphonological representation in which movementis represented as a separate unit, rather thanderiving from articulation constraints, as wasproposed by Uyechi (1995). The data also sup-port the analysis of place of articulation as a


Intended Slip

SEINE ELTERN ERROR ELTERN

Figure 43.5 Illustration of the intended phrase SEINE ELTERN (‘his parents’) and a slip of the hand inDeutsche Gebärdensprache (German Sign Language) In the slip, the Y hand configuration of ELTERN isanticipated and substituted for the intended B hand configuration of SEINE. From Hohenberger et al.(2002). Reprinted with permission.


high-level unit within the representation of asign, rather than as a phonetic feature(s) associ-ated with a segment slot (as place of articulationis often represented for consonants). This argu-ment derives from evidence indicating that pho-netic features do not operate as units in speechproduction and rarely participate in speecherrors (see Roelofs, 1999). Thus, since place ofarticulation in sign participates in exchange andother types of errors, it suggests that this param-eter is a unit rather than a feature for the purposesof sign production.

Both the ASL and DGS error corpora con-tained very few sign exchange errors, e.g. LIKE,MAYBE TASTE instead of TASTE, MAYBE LIKE(‘Taste it, and maybe you’ll like it’; Klima andBellugi, 1979). Word exchanges are argued totake place at a separate stage of sentence plan-ning (Garrett, 1975; Levelt, 1989). Hohenbergeret al. (2002) report that only 1 per cent of errorswere sign exchanges, compared to 15 per centword exchange errors found in the Frankfurt corpus of spoken German errors. In addition,Hohenberger et al. (2002) did not find evidencefor morpheme stranding errors, and no strand-ing errors were reported by Newkirk et al. (1980).A morpheme stranding error in English would beThat’s why they sell the cheaps drink for theintended That’s why they sell the drinks cheap(Garrett, 1988). In this example, the -s suffix is“stranded” or left behind when the two wordsexchange. The fact that stranding errors do notoccur in sign languages is likely due to the rarityof sign exchange errors and to the fact that mor-phological processes are non-concatenativerather than affixal. Stranding errors may onlyoccur when morphemes are arranged linearly,rather than articulated simultaneously.

Finally, Hohenberger et al. (2002) found thatsign errors were repaired much faster than speecherrors. The locus of repairs for speakers is mostoften after the word (Levelt, 1983), but for DGSsigners the error was preferentially caught some-where within the sign, i.e. before the signer finishedarticulating the sign containing the error. ForDGS, 57 per cent of repairs were made within theword; in contrast, only 27 per cent of the errorrepairs of Dutch speakers occurred within theword (from Levelt, 1983). Hohenberger et al.hypothesize that the longer articulation time forsigns allows for earlier detection of sign errorscompared to speech errors. Early repair of errorsalso explains the lack of sign exchange errors,because the slip is detected before the secondexchanged sign is produced.

In sum, data from slips of the hand provideevidence for phonological encoding during sign

production. Signs are not produced as gesturalwholes without internal structure. As with speech,phonological elements in sign language can beanticipated, perseverated, and exchanged duringproduction. Sign and speech appear to differwith respect to the speed of error detection andthe nature of word and morpheme level errors.The slower rate of sign articulation leads to earliererror repairs for signers and to fewer exchangeerrors. The linear affixation processes found in most spoken languages lead to morphemestranding errors that are not observed for signlanguages.

43.4.3 Sign monitoringLevelt (1983, 1989) proposes that speakers moni-tor their internal speech and can intercept errorsbefore they are overtly uttered—he terms this“prearticulatory editing.” It is reasonable tohypothesize that signers also have such an inter-nal monitor. Working-memory experiments withASL provide evidence for a non-overt articula-tory-based system of sign rehearsal that is usedduring short-term memory tasks (Wilson andEmmorey, 1997; 1998). This rehearsal systemappears to be equivalent to subvocal rehearsal forspeech, and provides evidence for a type of innersigning. Like speakers, signers may be able moni-tor this internal signing, catching errors beforethey are actually articulated. In fact, Hohenbergeret al. (2002) report that a small proportion ofsign errors (8 per cent) are detected prior to artic-ulation of the intended (target) sign. For example,an incorrect hand configuration can be producedand corrected during the movement transition tothe target sign. In addition, signers produce thesigned equivalent of um (a 5 handshape withwiggling fingers), which indicates they are hav-ing production difficulty (Emmorey, 2002).Signers also sometimes stop signing and shaketheir head, suggesting that they have detected anerror prior to articulation. These data supportthe existence of an internal monitor for signproduction. Whether this monitor operates on aphonological or a phonetic (articulatory) repre-sentation is currently under investigation in mylaboratory.

Sometimes errors or inappropriate words dononetheless slip through, and speakers alsomonitor their overt speech and can catch errorsby listening to their own voice. Herein lies apotentially interesting difference between signand speech. Speakers hear their voices, but sign-ers do not look at their hands and cannot see theirown faces. Facial expressions convey criticalgrammatical information for signed languages

Speaking vs. signing · 717


(e.g. Zeshan, 2004). When speakers are pre-vented from hearing their own voices (e.g. bywearing headphones emitting loud white noise)or when speakers silently mouth words, they areless likely to detect speech errors compared towhen they can hear themselves speak (Lacknerand Tuller, 1979; Postma and Noordanus, 1996).These results suggest that speakers rely to someextent on auditory feedback to detect errors inproduction. Levelt (1989) proposes that the per-ceptual monitor for overt speech operates viathe language user’s speech-understanding sys-tem. However, this cannot be the entire story forsign language monitoring, because the sign-understanding system operates on the basis ofvisual input, which is unavailable or distortedfor self-signed input. Signers cannot see theirown grammatical facial expressions, the view oftheir own hands falls in the far periphery ofvision, and they have a “backward” view of theirhands. Thus, it is problematic to simply adoptthe same processing system which parses visualinput when comprehending another’s signing inorder also to parse the visual input from one’sown signing. It is possible that the nature of theperceptual loop for sign monitoring is quite dif-ferent from that for speech monitoring. My col-leagues and I are currently investigating thishypothesis by examining how differences in visualfeedback impact sign language production.

43.5 Summary andconclusionsPsycholinguistic studies of sign language haverevealed (and continue to reveal) significantinsights into the nature of human language pro-cessing. Linguistic and psycholinguistic evi-dence has established that all human languageshave a level of meaningless sublexical structurewhich must be assembled during language pro-duction and which is exploited during languageperception. Both signers and speakers exhibitcategorical perception effects for distinctivephonological categories in their language, andboth combine phonological units prior to artic-ulation, as evidenced by slips of the tongue andhand. Signs, like words, are not holistic gestureswithout internal structure. Furthermore, the factthat signs are generally more iconic than wordsdoes not lead to a lack of separation between therepresentation of meaning and the representa-tion of form. Signers experience a tip-of-the-fingers state (analogous to the tip-of-the- tonguestate) in which semantic information is retrieved,while access to the form of the sign is somehow

blocked. The study of signed languages has iden-tified universal properties of language process-ing and exposed aspects of language processingthat are impacted by the biology of the languageprocessing system.

Specifically, biology has an impact of the speedof linguistic articulation: the tongue is quickerthan the hands. Biology also affects languageperception: the auditory system is particularlyadept at processing rapid, temporal sequences,while the visual system can easily process shapeand location information presented simultane-ously within the visual field. These biological differences exert specific effects on language pro-cessing and linguistic structure.

With respect to lexical processing, languagemodality affects the time course of word recog-nition and the nature of morphological parsing.Words take proportionally longer to recognizethan signs. More of a word must be heard before itcan be recognized, probably owing to the phono-tactics of speech (many words share an initialcohort), the fact that words tend to contain moresegments than signs, and the fact that wordonsets may be less informative than sign onsets.In addition, it has been argued that spoken lan-guages avoid non-concatenative morphologicalprocesses such as reduplication and circumfixa-tion because such processes have a high process-ing cost (e.g. the stem is disrupted and difficultto identify). Thus, spoken languages prefer lin-ear affixation. In contrast, the biology of signedlanguages favors non-concatenative morphol-ogy because movement “affixes” can be super-imposed onto a sign stem without disruptinglexical recognition. Further, linear affixationresults in processing costs for signed languagesbecause affixes increase articulation time, thusincreasing demands on working memory. Infact, when the linear affixation processes aretransferred to the visual-gestural modality viaManually Coded English, children fail to acquirethe linear morphology and often create simulta-neous morphological processes that are notobserved in their input.

The modality of signed languages affords theuse of signing space to convey linguistic distinc-tions. Signed languages tend to use signingspace for co-reference functions and to conveytopographic information about spatial relation-ships. Despite large differences in the form ofsigned and spoken pronouns, psycholinguisticstudies indicate that the same mechanisms under-lie coreference processing, namely, activation ofpronominal antecedents and suppression of non-antecedents in memory. In contrast, the use ofspace to convey spatial information leads to



some modality-specific effects on languagecomprehension. For example, ASL signers makeuse of “shared space” when describing jointlyobserved scenes, which does not require a par-ticular viewer perspective. In contrast, Englishspeakers must adopt a spatial perspective (It’s onmy left or It’s on your right). When comprehend-ing descriptions of non-present spatial scenes,viewers of ASL must perform a mental transfor-mation of signing space. English speakers arenot faced with such spatial computations. Thus,although co-reference processing appears to belargely unaffected by the spatialization of linguis-tic form, the processing mechanisims required tocomprehend and produce spatial descriptions areclearly affected by the visual-spatial modality.

In conclusion, signed languages provide aunique tool for investigating the psycholinguis-tic mechanisms which underlie language pro-cessing. Their study reveals both universal andbiology-specific mechanisms, and clarifies thenature of constraints on spoken language pro-cessing. Future studies may reveal how the biol-ogy of language affects the nature of outputmonitoring, the nature of perceptual segmenta-tion, and the interplay between language andother cognitive systems.

AcknowledgementsPreparation of this chapter was supported bygrants from the National Institute on Child Healthand Human Development (R01 HD13249; R01HD047736). I thank Virginia Volterra for helpwith the LIS examples in Figure 43.1.

ReferencesAnderson, S. R. (1992) A-morphous Morphology.

Cambridge University Press, Cambridge.Armstrong, D. F., Stokoe, W. C., and Wilcox, S. E. (1995)

Gesture and the Nature of Language. CambridgeUniversity Press, Cambridge.

Aronoff, M., Meir, I., and Sandler, W. (2005) The paradoxof sign language morphology. Language, 81: 301–44.

Baddeley, A. (1986) Working Memory. Clarendon Press,Oxford.

Baker, S., Idsardi, W., Golinkoff, R. M., and Petitto, L. A.(2005) The perception of handshapes in American SignLanguage. Memory and Cognition, 33: 887–904.

Battison, R. (1978) Lexical Borrowing in American SignLanguage. Linstok Press, Silver Spring, Md.

Beale, J. M., and Keil, F. C. (1995) Categorical effects in theperception of faces. Cognition, 57: 217–239.

Bellugi, U., and Fischer, S. (1972) A comparison of signlanguage and spoken language, Cognition, 1: 173–200.

Bergman, B. (1982) Forskning om Teckenspråk [Research onsign language]. Stockholm University.

Bever, T., and McElree, B. (1988) Empty categories accesstheir antecedents during comprehension. LinguisticInquiry, 19, 35–43.

Brentari, D. (1998) A Prosodic Model of Sign LanguagePhonology. MIT Press, Cambridge, Mass.

Brown, A. (1991) A review of the tip-of-the-tongueexperience. Psychological Bulletin, 109: 204–223.

Clark, L., and Grosjean, F. (1982) Sign recognitionprocesses in American Sign Language: the effect ofcontext. Language and Speech, 25: 325–340.

Corbett, A., and Chang, F. (1983) Pronoun disambiguation:accessing potential antecedents. Memory and Cognition,11: 283–294.

Corina, D. P. (1993) To branch or not to branch:underspecifications in ASL handshape contours.In G. Coulter (ed.), Phonetics and Phonology: CurrentIssues in ASL Phonology, pp. 63–96. Academic Press,New York.

Corina, D. P. and Knapp, H. (forthcoming) Lexical retrievalin American Sign Language production. In L. M.Goldstein, D. H. Whalen, and C. T. Best (eds.), Papers inLaboratory Phonology 8: Varieties of PhonologicalCompetence. Mouton de Gruyter, Berlin.

Corina, D. P., and Sandler, W. (1993) On the nature ofphonological structure in sign language. Phonology,10: 165 – 207.

Cutler, A., Hawkins, J., and Gilligan, G. (1985) Thesuffixing preference: a processing explanation.Linguistics, 23: 723–758.

de Gelder, B., Teunisse, J-P., and Benson, P. J. (1997)Categorical perception of facial expressions: categoriesand their internal structure. Cognition and Emotion,11: 1–22.

Dell, G. S., Schwartz, M. F., Martin, N., Saffran, E. M.,and Gagnon, D. A. (1997) Lexical access in aphasic and nonaphasic speakers. Psychological Review,104: 801–838.

Emmorey, K. (1997) Non-antecedent suppression inAmerican Sign Language. Language and CognitiveProcesses, 12: 103–112.

Emmorey, K. (2002) Language, Cognition, and the Brain:Insights from Sign Language Research. Erlbaum,Mahwah, NJ.

Emmorey, K., and Corina, D. (1990) Lexical recognition insign language: effects of phonetic structure andmorphology. Perceptual and Motor Skills, 71: 1227–1252.

Emmorey, K., Corina, D., and Bellugi, U. (1995)Differential processing of topographic and referentialfunctions of space. In K. Emmorey and J. Reilly (eds)Language, Gesture, and Space, pp. 43–62. Erlbaum,Hillsdale, NJ.

Emmorey, K., and Falgier, B. (2004) Conceptual locationsand pronominal reference in American Sign Language.Journal of Psycholinguistic Research, 33: 321–331.

Emmorey, K., and Herzig, M. (2003) Categorical versusgradient properties of classifier constructions in ASL. InK. Emmorey (ed) Perspectives on Classifier Constructionsin Signed Languages, pp. 222–46, Erlbaum, Mahwah, NJ.

Emmorey, K., Klima, E. S., and Hickok, G. (1998) Mentalrotation within linguistic and nonlinguistic domains in users of American Sign Language. Cognition,68: 221–246.

References · 719


Emmorey, K. and Lillo-Martin, D. (1995) Processing spatialanaphora: referent reactivation with overt and nullpronouns in American Sign Language. Language andCognitive Processes, 10: 631–664.

Emmorey, K., McCullough, S., and Brentari, D. (2003)Categorical perception in American Sign Language.Language and Cognitive Processes, 18: 21–45.

Emmorey, K., Norman, F., and O’Grady, L. (1991) Theactivation of spatial antecedents from overt pronouns inAmerican Sign Language. Language and CognitiveProcesses, 6: 207–228.

Emmorey, K., and Tversky, B. (2002) Spatial perspectivechoice in ASL. Sign Language and Linguistics, 5: 3–25.

Engberg-Pedersen, E. (1993) Space in Danish SignLanguage: The Semantics and Morphosyntax of the Use ofSpace in a Visual Language. Signum, Hamburg.

Fodor, J. (1989) Empty categories in sentence processing.Language and Cognitive Processes, 4: 155–209.

Fromkin, V. A. (1971) The non-anomalous nature ofanomalous utterances. Language, 47: 27–52.

Fry, D. B., Abramson, A. S., Eimas, P. D., and Liberman, A. M.(1962) The identification and discrimination ofsynthetic vowels. Language and Speech, 5: 171–189.

Gandour, J. (1977) Counterfeit tones in the speech ofsouthern Thai bidialectals. Lingua, 41: 125–143.

Garrett, M. (1975) The analysis of sentence production.In G. Bower (ed.), Psychology of Learning and Motivation,vol. 9, pp. 133–177. Academic Press, New York.

Garrett, M. (1988) Processes in language production. In F. J. Newmeyer (ed.), Linguistics: The Cambridge Survey,vol. 3: Language: Psychological and Biological Aspects,pp. 69–96. Cambridge University Press, Cambridge.

Gernsbacher, M. (1989) Mechanisms that improvereferential access. Cognition, 32: 99–156.

Goldinger, S. D., Luce, P. A., and Pisoni, D. B. (1989)Priming lexical neighbors of spoken words: effects ofcompetition and inhibition. Journal of Memory andLanguage, 28: 501–518.

Grosjean, F. (1980) Spoken word recognition processes and the gating paradigm. Perception and Psychophysics,28: 267–283.

Grosjean, F. (1981) Sign and word recognition: a firstcomparison. Sign Language Studies, 32: 195–219.

Gustason, G., Pfetzing, D., and Zawolkow, E. (1980) Signing Exact English. Modern Signs Press,Los Alamitos, Calif.

Hall, C. J. (1992) Morphology and Mind: A UnifiedApproach to Explanation in Linguistics. Routledge,London.

Hildebrandt, U., and Corina D. (2002) Phonologicalsimilarity in American Sign Language. Language andCognitive Processes, 17: 593–612.

Hoek, K. van (1992) Conceptual spaces and pronominalreference in American Sign Language. Nordic Journal of Linguistics, 15: 183–199.

Hohenberger, A., Happ, D., and Leuniger, H. (2002)Modality-dependent aspects of sign languageproduction: evidence from slips of the hands and theirrepairs in German Sign Language. In R. P. Meier, K.Cormier, and D. Quinto-Pozos (eds) Modality andStructure in Signed and Spoken Languages, pp. 112–42,Cambridge University Press, Cambridge.

Klima, E., and Bellugi, U. (1979) The Signs of Language.Harvard University Press, Cambridge, Mass..

Kyle, J., and Woll, B. (1985) Sign Language: The Study ofDeaf People and Their Language. Cambridge UniversityPress, Cambridge.

Lackner, J. R., and Tuller, B. H. (1979) Role of efferencemonitoring in the detection of self-produced speecherrors. In W. E. Cooper and E. C. T. Walker (eds),Sentence Processing: Psycholinguistic Studies Presented to Merrill Garrett, pp. 281–294.Erlbaum, Hillsdale, NJ.

Lane, H., Boyes-Braem, P., and Bellugi, U. (1976)Preliminaries to a distinctive feature analysis ofAmerican Sign Language. Cognitive Psychology,8: 263–289.

Leuniger, H., Hohenberger, A., Waleschkowski, E., Menges,E., and Happ, D. (2004) The impact of modality onlanguage production: evidence from slips of the tongueand hand. In T. Pechmann and C. Habel (eds.),Multidisciplinary Approaches to Language Production,pp. 219–77. Mouton de Gruyter, Berlin.

Levelt, W. J. M. (1983) Monitoring and self-repair inspeech. Cognition, 14: 41 – 104.

Levelt, W. J. M. (1989) Speaking: From Intention toArticulation. MIT Press, Cambridge, Mass.

Levelt, W. J. M. (1999) Models of word production. Trendsin Cognitive Sciences, 3: 223–232.

Levelt, W. J. M., Roelofs, A., and Meyer, A. S. (1999) Atheory of lexical access in speech production. Behavioraland Brain Sciences, 22: 1–75.

Liberman, A. M., Harris, K. S., Hoffman, H. S., and Griffith,B. C. (1957) The discrimination of speech soundswithin and across phoneme boundaries. Journal ofExperimental Psychology, 54: 358–368.

MacNeilage, P. (1998) The frame/content theory ofevolution of speech production. Behavioral and BrainSciences, 21: 499–546.

Mainwaring, S., Tversky, B., and Schiano, D. (1996)Perspective choice in spatial descriptions. IRC Technical Report, 1996–06. Interval ResearchCorporation, Palo Alto, Calif.

Marslen-Wilson, W. (1987) Functional parallelism inspoken word recognition. Cognition, 25: 71–102.

McClelland, J. L., and Elman, J. (1986) The TRACE model of speech perception. Cognitive Psychology,18: 1–86.

Meier, R. P. (2000) Shared motoric factors in theacquisition of sign and speech. In K. Emmorey and H. Lane (eds.), The Signs of Language Revisited: AnAnthology to Honor Ursula Bellugi and Edward Klima,pp. 333–57, Erlbaum, Mahwah, NJ.

Meier, R. P. (2002) Why different, why the same?Explaining effects and non-effects of modality uponlinguistic structure in sign and speech. In R. P. Meier,K. Cormier, and D. Quinto-Pozos (eds.), Modality andStructure in Signed and Spoken Languages, pp. 1–26,Cambridge University Press, Cambridge.

Newkirk, D., Klima, E., Pedersen, C. C., and Bellugi, U.(1980) Linguistic evidence from slips of the hand. In V. Fromkin (ed.), Errors in Linguistic Performance: Slips of the Tongue, Ear, Pen, and Hand, pp. 165–198.Academic Press, New York.



Poizner, H., and Lane, H. (1978) Discrimination oflocation in American Sign Language. In P. Siple (e.d),Understanding Language through Sign LanguageResearch, pp. 271–87. Academic Press, New York.

Postma, A., and Noordanus, C. (1996) Production anddetection of speech errors in silent, mouthed, noise-masked, and normal auditory feedback speech.Language and Speech, 39: 375–392.

Roelofs, A. (1999) Phonological segments and features asplanning units in speech production. Language andCognitive Processes, 14: 173–200.

Sandler, W. (1986) The spreading hand autosegment ofAmerican Sign Language. Sign Language Studies,50: 1–28.

Sandler, W., and Lillo-Martin, D. (2006) Sign Language andLinguistic Universals. Cambridge University Press,Cambridge.

Schober, M. F. (1993) Spatial perspective-taking inconversation. Cognition, 47: 1–24.

Schriefers, H., Meyer, A., and Levelt, W. J. M. (1990)Exploring the time course of lexical access in speechproduction: picture-word interference studies. Journal ofMemory and Language, 29: 86–102.

Schwartz, B.L. (1999) Sparkling at the end of the tongue:the etiology of tip-of-the-tongue phenomenology.Psychonomic Bulletin and Review, 6: 379–393.

Stack, K. M. (1999) Innovation by a child acquiring SigningExact English II. Doctoral dissertation, University ofCalifornia at Los Angeles.

Starreveld, 2000) On the interpretation of onsets ofauditory context effects in word production. Journal ofMemory and Language, 42: 497–525.

Stokoe, W. (1960) Sign language structure: an outline ofthe visual communication systems of the AmericanDeaf. Studies in Linguistics, Occasional Papers 8,available from Linstok Press, Silver Spring, Md.

Stokoe, W. (1991) Semantic phonology. Sign LanguageStudies, 71: 107–114.

Supalla, S. (1991) Manually coded English: the modalityquestion in signed language development. In P. Sipleand S. D. Fischer (eds.), Theoretical Issues in SignLanguage Research, pp. 85–109, University ofChicago Press, Chicago.

Supalla, T. (1982) Structure and acquisition of verbs ofmotion and location in American Sign Language.Ph.D. dissertation, University of California,San Diego.

Sutton-Spence, R., and Woll, B.(1999) The Linguistics ofBritish Sign Language: An Introduction. CambridgeUniversity Press, Cambridge.

Talmy, L. (2000) Toward a Cognitive Semantics, vol. 1:Concept Structuring Systems. MIT press,Cambridge, Mass.

Thompson, R., Emmorey, K., and Gollan, T. (2005) Tip-of-the-fingers experiences by ASL signers: insightsinto the organization of a sign-based lexicon.Psychological Science, 16: 856–860.

Uyechi, L. (1995) The Geometry of Visual Phonology.CSLI Publications, Stanford, Calif.

Wen, I.P. (2000) Mandarin phonology: evidence fromspeech errors. Doctoral dissertation, State University ofNew York, Buffalo.

Wilbur, R. B. (1987) American Sign Language: Linguisticand Applied Dimensions, 2nd edn. College Hill,Boston, Mass.

Wilbur, R. B. (1993) Syllables and segments: hold themovements and move the holds! In G. Coulter (ed.),Phonetics and Phonology: Current Issues in ASLPhonology, pp. 135–68. Academic Press, San Diego, Calif.

Wilson, M., and Emmorey, K. (1997) A visual-spatial“phonological loop” in working memory: evidence from American Sign Language. Memory and Cognition,25: 313–320.

Wilson, M., and Emmorey, K. (1998) A “word lengtheffect” for sign language: further evidence on the role oflanguage in structuring working memory. Memory andCognition, 26(3), 584–590.

Zeshan, U. (2004) Head, hand, and face: negativeconstructions in sign languages. Linguistic Typology,8: 1–57.

References · 721



Documents

The psycholinguistics of signed and spoken languages: how ... · The psycholinguistics of signed and spoken languages: how biology affects processing Karen Emmorey 43-Gaskell-Chap43