‘‘Hold that thought’:Hold that thought’:Effects of held gestures on Effects of held gestures on

referent accessibilityreferent accessibility

Mischa Park-DoobMischa Park-Doob

UC Berkeley, Dept. of LinguisticsUC Berkeley, Dept. of Linguistics

UC Berkeley ICBS Faculty Retreat — December 7, 2007

BackgroundBackground• Goldin-Meadow et al. (2001) found that

gesturing while speaking improves performance on concurrent memorization tasks.

Why might this happen?Why might this happen?

• This would yield improved rehearsal for the concurrent memorization task.

BackgroundBackground• Goldin-Meadow et al. (2001) found that

gesturing while speaking improves performance on concurrent memorization tasks.

• Access to gestures may sidestep some of the need for internal rehearsal of material related to the speaking task.

• This would yield improved rehearsal for the concurrent memorization task.

Improved performance!Improved performance!

BackgroundBackground

• Internal rehearsal requires a certain amount of voluntarily directed attention, a central processing task with a unitary focus at a given moment in time (Miyake & Shah, 1999).

• Because of this “central processing bottleneck”, less rehearsal for one task should allow more rehearsal for a separate, simultaneous task (Pashler, 1997).

Does rehearsal really work this way?Does rehearsal really work this way?

BackgroundBackgroundWorking Memory architecture...Working Memory architecture...

Central Processor:– attention switching– manipulation / processing– rehearsal management...– retrieval from the following:

(re)writeable storage buffers:

(c.f. Baddeley 1986 and later work)

“Phonological Loop” – Verbal WM: sound and verbal articulation

Visuo-Spatial “sketch-pad”

Visual cache:– patterns & textures

“Inner scribe”– spatial relations, manual actions, paths, etc.

BackgroundBackground

• Gesture might result in information being “written” to Working Memory in a more robust way than it would be without gesture.

• Why? Direct visual and kinesthetic experience might create robust WM traces which require less rehearsal to maintain adequate activation, as compared to mental imagery generated without an embodied accompaniment.

How would gestures affect rehearsal?How would gestures affect rehearsal?

BackgroundBackground

• Since gestures are externalized they are available to the visual (and kinesthetic) facets of perceptual attention, and may also engage the automatic orienting response.

• Both provide a means by which information from gesture can “enter” Working Memory, without needing attention to be voluntarily directed at gesture:

In addition (and perhaps more importantly)In addition (and perhaps more importantly)

BackgroundBackgroundPossible “gestural loophole”Possible “gestural loophole”

Central Processor:– attention switching– manipulation / processing– rehearsal management...– retrieval from the following:

(re)writeable storage buffers:

Writing to storagebuffers withoutattention switching??

“Phonological Loop” – Verbal WM: sound and verbal articulation

Visuo-Spatial “sketch-pad”

Visual cache:– patterns & textures

“Inner scribe”– spatial relations, manual actions, paths, etc.

?

BackgroundBackground

• Gestures need not disappear instantly. Having continued access to a gesture allows more flexibility in any process that activates or reactivates the gesturally presented information.

• Instead of constant internal rehearsal, such information can be allowed to degrade temporarily as long as there is the possibility of accessing it again later, from a gesture that is still accessible to perception.

Such gestures can serve as retrieval cues.

On an entirely different note...On an entirely different note...

Retrieval CuesRetrieval Cues• A retrieval cue can be defined as something

which helps “reinstate the encoding context” (Nelson & Goodmon, 2003).

• What gestures can serve as retrieval cues?

• A recurring novel gesture (i.e., a “catchment” in McNeill’s terms), seems an obvious example of how a gesture could be involved in the recreation of an earlier “encoding context”.

• How about maintenance of a such a context, instead of just “reinstatement”? A new place for holds in a typology of gesture?

Speakers vs. AddresseesSpeakers vs. Addressees• A gesture held by a speaker:

provides visual and kinesthetic presence.

• A gesture hold observed by an addressee: provides visual presence only.*

• Difference especially in the case of distraction: Spr. suddenly enthralled by Goodyear BlimpSpr. suddenly enthralled by Goodyear Blimp Adr. suddenly enthralled by Goodyear BlimpAdr. suddenly enthralled by Goodyear Blimp

• Not clear (yet) whether this difference matters...

• Hold ?= a retrieval cue “free of charge”! Unlike a catchment, a hold is simply “still there”,

requiring no reenactment. Adaptation!?

Why I care...Why I care...• Can we find human adaptations that may

improve our ability to maintain coherence and/or efficiency of thought?

• Especially during collaborative (e.g. conversational) activity...

• In spite of the disruptions and distractions which are frequent in real-life situations...

The ExperimentThe Experiment• Looked at the effect of a speaker’s gesture hold on the

addressee, bridging across a moment of audio/visual distraction.

• Movie clips simulated the role of speaker.• Three gesture conditions: 1.

“Long” (held through distraction) 2. “Short” (dropped before end of distraction) 3. “Zero” (no gesture hold)

• Other parameters (e.g. the distraction itself) kept as identical as possible across conditions.

• Between-subjects design (independent groups for each gesture condition)...

• Approximately 60 participants yielded the data presented here (20 per condition).

The ExperimentThe Experiment• It was NOT obvious to participants that gesture was the

focus of the research.• Participants observed movie clips on a 17” monitor, using

headphones, and responded to an audio True-False question following each movie, by pressing a button.

• Experiment was run using E-Prime 2.0 (Beta), which collected response time and accuracy data.

• Audio distraction: a “meta distraction” (not intended to truly impede comprehension).

• Video distraction: temporarily replaced movie of speaker with movie of a bouncing object. Intent: to overwrite VSWM for gesture tracking.

• Stimulus clips categorized loosely into several types, based on relevance of the gesture to the material later quizzed. Two main types.

Type A:

Non-iconically related information tagged onto a “gestural anchor”

- - - - - - ‘Long’ vs. ‘Short’ condition: video

- - - - - -(videos not included in print version)

Long Short Zero

T/F: “The first place you pass is the pizza shop.” (True)

“... you have to walk down past the pizza place...”

“...but you have to walk down the little alley for a while...”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “The first place you pass is the pizza shop.” (True)

Long Short Zero

T/F: “The tree was an elm tree.” (True)

“I was out walking with my dog in the woods, and there was this huge elm tree, like huge...”

“... and looked at me like he expected me to entertain him...”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “The tree was an elm tree.” (True)

Long Short Zero

T/F: “The spider hid under my keyboard.” (False)

“... I tried to flick it off, but it scurried under my notebook...”

“... and suddenly it just comes zooming right out at me...”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “The spider hid under my keyboard.” (False)

Long Short Zero

T/F: “At the beginning, John went to Steve’s cubicle.” (True)

“...so like, John had walked over to Steve's desk to ask him...”

“... grabbed him up and pushed him so hard he slammed...”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “At the beginning, John went to Steve’s cubicle.” (True)

Results so far...

Nothing yet!

(no significant differences)

Note: Throughout, incorrect responses are not included in the reaction time calculations.

A slightly different type:

Referents again representednon-iconically, but in a

relevant spatial configuration

Long Short Zero

T/F: “The church sits next to the movie theater.” (True)

“... and you'll get off at the station just after passing a church on your right.”

“... out of the old abandoned movie theater nearby.”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “The church sits next to the movie theater.” (True)

Results:

Nearly significant between long and short condition...

This trend became more pronouncedwith the following type:

Type B:

Iconically distinctive information represented in the form

of the gesture itself

- - - - -(video examples)

Long Short Zero

T/F: “The thing that bounced off my forehead was a softball.”

“... a basketball conking me on the top of the head,a facefull of tennis balls, baseball mitts, and tennis rackets,and finally a softball that bounced right off my forehead...”

“... I just walked away and made myself a pizza.”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “The thing that bounced off my forehead was a softball.”

Long Short Zero

T/F: “I was eating a hamburger.” (False)

“... so I choked and kind of got even more food on myself...”

“I was sitting there in a booth all alone,trying to eat this hotdog...”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “I was eating a hamburger.” (False)

Long Short Zero

T/F: “To get through the crowd we charged through.” (False)

“... so that we had to just like, weave around people...”

“... see the speaker. But it ended up being really boring...”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “To get through the crowd we charged through.” (False)

Long Short Zero

T/F: “He showed us the mountains in a photo album.” (False)

“... the guide was telling us,‘take a look at those mountain peaks...’”

“... the tallest one is the most sacred mountain in the area.”

Mean Reaction Time by Gesture Type

0

100

200

300

400

500

600

700

800

900

1000

1100

1200

1300

Long Short Zero

Reaction Time (ms)

T/F: “He showed us the mountains in a photo album.” (False)

Results

Significant differences found:• Between ‘Long’ vs. ‘Short’ conditions

• Ocassionally between ‘Short’ vs. ‘Zero’

‘Short’ condition = slowest response

Summary & Discussion 1Summary & Discussion 1• Among those who observed a held gesture at the

beginning (before the distraction): They responded significantly faster if they were given renewed visual access to the hold... But only if the form of gesture yielded an iconic representation that was directly relevant to their choice of answer to the True-False question.

• Tentative conclusion: Held gestures are capable of benefiting addressees by serving as retrieval cues: Having renewed access to a still-held gesture appears to increase accessibility of info presented when the gesture was first enacted.

Summary & Discussion 2Summary & Discussion 2• Among those who did not have renewed access

to a held gesture (following the distraction): They appeared to respond faster if they never observed a held gesture in the first place!

• Can gesture be a liability??Can gesture be a liability?? Tracking a multi-modal message may make one more susceptible to distractions that intersect the same parts of Working Memory. But using more than one “storage buffer” provides “diversification”, a situation less likely to experience total collapse, in general? In this case, verbal WM was less disrupted by the distraction, but things could’ve been different.

Summary & Discussion 3Summary & Discussion 3• Is verbal WM encoded differently, or less

robustly, in a multi-modal message?

• Possibility 1: Yes: observing a multi-modal message results in less robust memory for the verbal portion of the message, if we are forced to consider it alone (is this a natural situation?).

• Possibility 2: It may not be a WM issue – instead, retrieval cues may depend on both presence and absence of relevant visual stimuli....

i.e., “change is bad”, “same is good”... (state-dependent learning?)

Thank You!Thank You!

• contact:

• Special Thanks:Keith JohnsonCarla Hudson KamEve Sweetsermembers of the Berkeley Gesture Project

References:Baddeley, A.D. (1986). Working Memory. Oxford: Oxford University Press.Goldin-Meadow, S., H. Nusbaum, S.D. Kelly, and S. Wagner (2001). Explaining math:

Gesturing lightens the load. Psychological Science 12(6): 516-522.McNeill, D. (2005). Gesture and Thought. Chicago: University of Chicago Press.Miyake, A. & P. Shah, eds. (1999). Models of Working Memory: Mechanisms of Active

Maintenance and Executive Control. Cambridge: Cambridge University Press.Nelson, D.L. & L.B. Goodmon (2003). Disrupting attention: The need for retrieval cues in

working memory theories. Memory & Cognition 31(1): 65-76.Pashler, H.E. (1997). The Psychology of Attention. Cambridge, MA: The MIT Press.