The EXBODIED MIND: Motion in Communication and Cognition Research RWTH A h U i it 8 9 A il 2011RWTH Aachen University, 8-9 April 2011

Gesture, Gesture, Conceptualization, andConceptualization, andConceptualization, and Conceptualization, and Distributed CognitionDistributed Cognition

Robert F. Williams

Appleton WisconsinAppleton, Wisconsin

ApproachApproach

Distributed Cognition Cognitive Semantics

Studying Human CognitionStudying Human CognitionClassical View

Studying Human CognitionStudying Human CognitionClassical View S H I F T Situated View

Cognitive Functional Systems:Everyday Examples

• Tracking attendance

• Determining order of service• Determining order of service

• Counting objects

• Telling time

Functional SystemFunctional System

??

? !? !

C di ti C t li tiCoordination Conceptualization

QuestionsQuestions

• How do people accomplish cognitive activities?How do people accomplish cognitive activities?– coordination of representational media => computation

• How do they construct relevant meanings?• How do they construct relevant meanings?– conceptualization

• How do they guide the meaning of others?– instructional discourse (multimodal)

• How do they reason collaboratively?– group discourse (multimodal)g ( )

MethodsMethods

• Data collectionData collection

– Ethnographic study (cognitive ethnography)

– Quasi-experimental situations

• Analysis

– Distributed cognition: how activities are accomplished

Cognitive semantics: conceptual structures/processes;– Cognitive semantics: conceptual structures/processes; guiding conceptualization

=> role of bodily actions (gestures, manipulations, enactments, etc.)

Example: Counting ObjectsExample: Counting Objects

“one, two, three…”o e, t o, t ee

#How ?

Coordinatingaction

…

#many? action

Functional Systems for CountingFunctional Systems for Countingone, two, three… one, two, three…

six, seven, eight… A, B, C…

Functional Systems for CountingFunctional Systems for CountingSEQUENTIAL TOUCHING(POINTING / LOOKING)

MOVING OBJECTS

“one” “two” “three” “two”“four”“six”

…

USING FINGER PROXIES

[a]

[c][b]

[a]

[c][b]

Coordination:Use of the body to coordinate spoken representations with material objects

Functional Systems for CountingFunctional Systems for CountingSEQUENTIAL TOUCHING(POINTING / LOOKING)

MOVING OBJECTS

SG?

“one” “two” “three”

TR

Already counted To be countedTo be counted

“two”“four”“six”

… Already counted

USING FINGER PROXIES

[a]

[c][b]Already counted

To be counted[a]

[c][b]

Already counted

To be counted

TRS

Conceptualization:Image-schematic structure (SOURCE-PATH-GOAL, PROXIMITY, CONTAINER)Material/spatial anchoring of conceptual categories

Instructional Discourse:Counting on the Clock

Selecting Episodes for AnalysisActivity Focus Duration

8:56

Selecting Episodes for Analysis

Presentation Review dividing a circle into halves and fourths on felt board 0:34

Presentation Equate one fourth to one quarter by analogy to money 0:30

Presentation Divide the clock face into quarters 0:33

P t ti R d ti “ t t” 0 41Presentation Read a time as “quarter past” 0:41

Group practice Read a quarter past eight with prompting 0:15

Group practice Read a quarter past ten and a quarter past three 0:15

Presentation Count on the clock face to read the time as “ fifteen” 0:26Presentation Count on the clock face to read the time as _ fifteen 0:26

Group practice Read five fifteen and a quarter past five 0:17

Group practice Read a quarter past seven and seven fifteen 0:13

Individual practice Read eight fifteen and a quarter past eight 0:29p g q p g

Individual practice Read a quarter past two and two fifteen 0:28

Presentation Write two fifteen as ‘2:15’ 0:14

Individual practice Read a quarter past four and four fifteen; write 4:15 0:46

Individual practice Read six fifteen and a quarter past six; write 6:15 1:23

Individual practice Read ten fifteen and a quarter past ten; write 10:15 1:52

Transcribing

Transcribing

Transcribing

AnalyzingAnalyzing

• Diagramming conceptual inputs and operationsDiagramming conceptual inputs and operations step-by-step in the unfolding discourse

A l i th l f b dil t th i• Analyzing the roles of bodily movements, their relation to speech, coupling with environment, etcetc.

Prompting for a New Conceptualization

12 12

11

6

9 3

2

457

8

10

6

now another way that we sa:y it

Activating a Cognitive Model

12 12

11“fi t fift ”

6

9 3

2

457

8

10“five, ten, fifteen…”

SG

6

Counting by Fives

…

is we count by fi:ves

Mapping #1

“fi t fift ”12 1

211

#1“five, ten, fifteen…”

SG

6

9 3

2

457

8

10#1

6…

12

9 3

12

48

1011

64

578

when we move this,

Mappings #2 and #3

12 12

11“fi t fift ”

6

9 3

2

457

8

10“five, ten, fifteen…”

SG

#2 6

12S

…2

#3

12

9 3

12

48

1011

64

578

G?

from number to number=

Mapping #4 (and #5)

12 111“fi t fift ”

days... ...

6

9 3

2

457

8

10“five, ten, fifteen…”hours

minutes

... ...

... ...S

G

#4

6 57

Time MeasurementS

… (#5)

12

9 3

1210

11

S

5 min

6

9 34

578

G?

there’s five minutes between each number

Transition

12 111“fi t fift ”

days... ...

6

9 3

2

457

8

10“five, ten, fifteen…”hours

minutes

... ...

... ...S

G

6 57

S

…

12

9 3

1210

11

S

5 min

6

9 34

578

G?

so if we were going to count by fives it would be:

Counting on the Clock

12 1210

11“five, ten, fifteen…”

days

hours

... ...

6

9 3

457

8

10 hours

minutes

... ...

... ...S

G

…

“five”S S S12

9 3

1210

11 “ten”12

9 3

1210

11five

12

9 3

1210

11

“fifteen”... ...5 5

55

5

5

G

64

578

64

578

64

578

G

(0.4) fiftee:n=(0.6) te:n(0.5) fi:ve

Conceptual Integration Network

12 111“fi t fift ”

days... ...

6

9 3

2

457

8

10“five, ten, fifteen…”hours

minutes

... ...

... ...S

G

6 57

Time MeasurementCounting by Fives

…

12 1210

11“five”5

S

6

9 3

457

8G?

Counting on the Clock

Group ReasoningGroup Reasoning

Explaining the phases of the moon

Group ReasoningGroup Reasoning

Reasoning about the seasons

Methodological IssuesMethodological Issues

• Capturing phenomenaCapturing phenomena

– Ethnography provides “real” data & warrants

access, time, equipment, when/what to record, missed data

– Quasi-experimental situations provide control & captureQuasi experimental situations provide control & capture

less natural, lack ethnographic warrants for interpretations

Methodological IssuesMethodological Issues

• Transcribing (distilling)Transcribing (distilling)

– Coding gesture highlights types & patterns

isolates gestures, may mask functions

– Annotated images highlight gesture functionsAnnotated images highlight gesture functions

no support for categorical analysis

Methodological IssuesMethodological Issues

• Interpreting / analyzingInterpreting / analyzing

– Distributed cognition & cognitive semantic analysis

can’t be automated (requires expert analyst), hard to generalize

– Quantitative / statistical analysisQuantitative / statistical analysis

bleaches out situated aspects of cognition & communication

What can be done?What can be done?

• For nowFor now

– Multiple avenues: match approach to question

• For the future: new digital tools

– Transcripts with embedded videos & metadata?

– Links to analyses?

Searchable databases or gesture corpora?– Searchable databases or gesture corpora?

• And then: …?

Selected ReferencesSelected ReferencesFauconnier, G. & Turner, M. (2002). The Way We Think: Conceptual Blending and the

Mind’s Hidden Complexities. New York, NY: Basic Books.Hutchins E (1995) Cognition in the Wild Cambridge MA: MIT PressHutchins, E. (1995). Cognition in the Wild. Cambridge, MA: MIT Press.Hutchins, E. (2005). Material anchors for conceptual blends. Journal of Pragmatics,

37(10): 1555-1577.Williams, R. F. (2006). Using cognitive ethnography to study instruction. In S. A. Barab,

K. E. Hay, and D. T. Hickey (Eds.), Proceedings of the 7th International Conference of the Learning Sciences (vol. 2, pp. 838-844). International Society of the Learning Sciences / Erlbaum.

Williams, R. F. (2007). Counting and conceptual blending. 10th International Cognitive ( ) g g gLinguistics Conference, Krakow, July 15-20.

Williams, R. F. (2008a). Gesture as a conceptual mapping tool. In A. Cienki & C. Müller(Eds.), Metaphor and Gesture [Gesture Studies 3] (pp. 55-92). Amsterdam: John Benjamins.Benjamins.

Williams, R. F. (2008b). Situating cognition through conceptual integration. 9th

Conference on Conceptual Structure, Discourse, and Language, Case Western Reserve University, Cleveland, OH, October 18-20

Williams R F (2010) Gesture in everyday scientific reasoning and explanation 4thWilliams, R. F. (2010). Gesture in everyday scientific reasoning and explanation. 4th

Conference of the International Society for Gesture Studies, Europa University Viadrina, Frankfurt/Oder, July 25-30.

Thank YouThank You

Natural Media & EngineeringHuman Technology Centre (HumTec)RWTH Aachen University

Interdisciplinary Center for Clinical ResearchUniversity Hospital AachenRWTH Aachen University